The Differentiation of Pseudo-Inverses and Nonlinear Least Squares Problems Whose Variables Separate

For given data $(t_i ,y_i ),i = 1, \cdots ,m$, we consider the least squares fit of nonlinear models of the form \[ \eta ({\bf a},{\boldsymbol \alpha} ;t) = \sum _{j = 1}^n {a_j \varphi _j ({\boldsymbol \alpha} ;t),\qquad {\bf a} \in \mathcal{R}^n ,\qquad {\boldsymbol \alpha} \in \mathcal{R}^k .} \] For this purpose we study the minimization of the nonlinear functional \[ r({\bf a},{\boldsymbol \alpha} ) = \sum\limits_{i = 1}^m {\left( {y_i - \eta \left( {{\bf a},{\boldsymbol \alpha} ,t_i } \right)} \right)^2 } . \] It is shown that by defining the matrix $\{ {\bf \Phi} ({\boldsymbol \alpha} )\} _{i,j} = \varphi _j ({\boldsymbol \alpha} ;t_i )$, and the modified functional $r_2 ({\boldsymbol \alpha} ) = \| {\bf y} - {\bf \Phi} ({\boldsymbol \alpha} ){\bf \Phi} ^ + ({\boldsymbol \alpha} ){\bf y} \|_2^2 $, it is possible to optimize first with respect to the parameters ${\boldsymbol \alpha} $, and then to obtain, a posteriors, the optimal parameters $\bf {\hat a}$. The matrix ${\bf \Phi} ^ + ({\boldsymbol{\alpha}} )$ is the Moore–Penrose generalized inverse of ${\bf \Phi} ({\boldsymbol{\alpha}} )$. We develop formulas for the Frechet derivative of orthogonal projectors associated with ${\bf \Phi} ({\boldsymbol{\alpha}} )$ and also for ${\bf \Phi} ^ + ({\boldsymbol{\alpha}} )$, under the hypothesis that ${\bf \Phi} ({\boldsymbol{\alpha}} )$ is of constant (though not necessarily full) rank. Detailed algorithms are presented which make extensive use of well-known reliable linear least squares techniques, and numerical results and comparisons are given. These results are generalizations of those of H. D. Scolnik [20] and Guttman, Pereyra and Scolnik [9].