Algebraic Analysis for Nonidentifiable Learning Machines

This article clarifies the relation between the learning curve and the algebraic geometrical structure of an unidentifiable learning machine such as a multilayer neural network whose true parameter set is an analytic set with singular points. By using a concept in algebraic analysis, we rigorously prove that the Bayesian stochastic complexity or the free energy is asymptotically equal to lambda(1) log n - (m(1) - 1) log log n + constant, where n is the number of training samples and lambda(1) and m(1) are the rational number and the natural number, which are determined as the birational invariant values of the singularities in the parameter space. Also we show an algorithm to calculate lambda(1) and m(1) based on the resolution of singularities in algebraic geometry. In regular statistical models, 2lambda(1) is equal to the number of parameters and m(1) = 1, whereas in nonregular models, such as multilayer networks, 2lambda(1) is not larger than the number of parameters and m(1) > or = 1. Since the increase of the stochastic complexity is equal to the learning curve or the generalization error, the nonidentifiable learning machines are better models than the regular ones if Bayesian ensemble learning is applied.