Existence of Optimal Stochastic Control Laws

We give an approach to optimal control of systems described by stochastic functional-differential equations of Itô’s type : $dx(t) = f(t,x,u(t,x))dt + dw(t))$, $0 \leqq t \leqq 1$, with a cost functional $k(u) = E\int_0^1 {c(t,x,u(t,x))dt} $ to be minimized. Here $w( \cdot )$ is Brownian motion, f and c are nonanticipative functionals describing system dynamics and cost rate respectively, and $u(t, \cdot )$ is a causal control law, to be chosen, taking values in a space $\Gamma $ of control points, and depending at t at most on given information about the past $\{ x(s),s \leqq t\} $. A key role is played by the information available for control. This is described by giving, for each t, a sub-$\sigma $-algebra $G_t $ of the $\sigma $-algebra $S_t $ over the continuous functions $C = C[0,1]$ generated by sets $\{ y:y(s) \in A\} $ with $0 \leqq s \leqq t$ and A Borel in $R^d $. $S_t $ is the $\sigma $-algebra corresponding to knowing the whole past of the trajectory prior to t. The set $\mathcal{U}$ of admissible control laws consists of functions $u:[0,1] \times C \to \Gamma $ which are Lebesgue in t, and $G_t $-measurable in y for each t. There is a $\sigma $-algebra G over $[0,1] \times C$ such that admissibility is equivalent to G-measurability. Our formulation is based on a result of Girsanov (Teoriya Veroyatnostei, 5 (1960), p. 285) : For $\varphi $ a nonanticipative functional of Brownian motion w, the transformed measure $d\tilde P = \exp \xi (\varphi )dP$ with \[\xi (\varphi ) = \int_0^1 {\varphi \, dw - \frac{1}{2}} \int_0^1 {\left| \varphi \right|^2 dt} \] makes the functions $w( \cdot ) - \int_0^ \cdot {\varphi \, dt} $ a Wiener process, provided $E\exp \xi (\varphi ) = 1$. This result suggests and justifies taking, for the solution $x( \cdot )$ of the system equations, the process determined by Girsanov’s device with $\varphi = f(t,w,u(t,w));$ in this case we say that u attains the density $\exp \xi (\varphi )$. The control problem is reformulated as a search for admissible u that achieve $\inf _{u \in \mathcal{U}} E\exp \xi (\varphi )\int_0^1 {cdt} $. That is, with each $u \in \mathcal{U}$ we associate, as the solution of the system equations to be considered for the purpose of our minimization of $k( \cdot )$, the functions $w( \cdot )$ under the measure $\exp \xi (\varphi )dP$, with the justification that under this measure \[w(t) - \int_0^t f(s,w,u(s,w))ds\] is a Wiener process. Novelty of the approach lies in these features : (i) control is closed loop; (ii) admissible controls need not be smooth; (iii) the Radon–Nikodym derivative used by Girsanov directly gives a measure corresponding to a solution of the system equations. As a principal result, we prove that if $\Gamma $ is compact metric, if $f(t,y,u)$ grows at most linearly with $y(t)$, and if $G_t = S_t $ (i.e., if the whole past is available for control), then the set of densities $\{ \exp \xi (\varphi ):\varphi = f(t,w,u(t,w)),u \in \mathcal{U}\} $ attainable by the admissible control laws is convex, and there exists an optimal control law $u^ * \in \mathcal{U}$ achieving $\inf _{u \in U} E\exp \xi (\varphi )\int_0^1 {cdt} $.