We describe several interconnections between the topics mentioned in the title. In particular, we show how some previously known formulas for inverting Toeplitz operators in both discrete- and continuous-time can be interpreted as versions of the Christoffel-Darboux formula for the biorthogonal Szegö and Krein polynomials on the circle and the line, respectively. The discrete-time inversion result is often known as Trench's formula, while the continuous-time result was apparently first deduced (in radiative transfer theory) by Sobolev. The concept of innovations is used to motivate the definitions of the Szegö and especially the Krein orthogonal functionals, and connections to work on the fitting of autoregressive models and inversion of the associated covariance matrices are also noted.