Modelling Dynamics

"We can't finish the gravity equation without data from inside the singularity. The universe hides its rules where we can't reach them."
— Paraphrased from the movie Interstellar (2014)

In finance, as in physics, the observable world often masks simpler underlying dynamics. Asset prices fluctuate, economic indicators rise and fall, but beneath these surface movements lie fundamental forces—latent states that drive market behavior. These hidden dynamics, whether they manifest as volatility regimes, credit cycles, or common risk factors, cannot be directly measured. Yet understanding them is essential for forecasting, risk management, and optimal decision-making.

This chapter explores state-space models and dynamic factor models—mathematical frameworks that allow us to infer these unobservable drivers from noisy, incomplete observations. Unlike black-box machine learning approaches, state-space models provide interpretable representations of latent dynamics, enabling us to understand not just what will happen, but why through the lens of underlying factors.

The challenge is fundamental: we observe market prices, economic indicators, and financial time series, but we need to extract the hidden states that govern their evolution. This is the "data from inside the singularity"—the latent dynamics that determine outcomes but remain invisible to direct measurement. State-space models provide the mathematical machinery to bridge this gap, transforming observable data into insights about unobservable drivers.

Why This Matters

State-space models have become indispensable in modern finance. Central banks use them for nowcasting GDP before official releases, combining monthly indicators with quarterly aggregates. Risk managers employ them to estimate volatility regimes that shift between calm and turbulent periods. Portfolio managers rely on them to extract common factors driving asset returns, enabling better diversification and risk allocation.

The power of state-space models lies in their ability to handle real-world complexities: missing data, mixed frequencies, measurement error, and temporal dependencies. They provide a unified framework that encompasses classical time-series models (ARIMA, GARCH) as special cases while extending to modern applications like nowcasting and factor extraction.

Chapter Structure

This chapter progresses from theoretical foundations to practical implementation:

Section 6-01: What Is State Space? introduces the mathematical framework of state-space models, explaining how latent states connect to observable data through transition and observation equations. We explore the historical development from control theory to econometrics, examine key properties like observability and stability, and illustrate applications in volatility modeling and nowcasting.

Section 6-02: Finding Latent State addresses the core estimation challenge: how do we extract unobservable states from noisy observations? We progress from the simplest method (Principal Component Analysis) through recursive filtering (Kalman filter), smoothing (Kalman smoother), to joint state-parameter estimation (EM algorithm). Each method builds on the previous, solving increasingly complex inference problems.

Section 6-03: Dynamic Factor Model provides a hands-on tutorial for building and using Dynamic Factor Models with the dfm-python package. We cover configuration, training, forecasting, and nowcasting applications, demonstrating how theoretical concepts translate into practical tools for financial analysis.

Section 6-04: Factors and Autoencoders explores the limitations of linear models and introduces nonlinear extensions. We establish the connection between PCA and autoencoders, showing how neural networks generalize classical dimension reduction while maintaining factor model structure. This section bridges classical econometrics and modern deep learning.

Section 6-05: Deep Dynamic Factor Model completes the journey with a practical tutorial on Deep Dynamic Factor Models (DDFMs). Using neural encoders to extract factors while maintaining interpretable dynamics, DDFMs capture nonlinear relationships and adapt to structural breaks—capabilities essential for modeling complex financial systems.

Connection to Previous Chapters

This chapter builds on several foundations established earlier. From Chapter 2, we use classical time-series concepts (ARIMA, GARCH) that appear as special cases of state-space models. From Chapter 3, we leverage Python tools for data manipulation and visualization. From Chapter 4, we connect forecasting evaluation frameworks to state-space predictions. And from Chapter 5, we see how extracted factors inform optimal decision-making under uncertainty.

The latent states we extract here become inputs to downstream applications: portfolio optimization uses factor exposures, risk management relies on volatility regime estimates, and forecasting systems incorporate nowcasted economic conditions. State-space models thus serve as a bridge between raw data and actionable insights.

What You Will Learn

By the end of this chapter, you will be able to:

• Understand the mathematical structure of state-space models and their relationship to observable data
• Extract latent factors from high-dimensional time series using PCA, Kalman filtering, and EM algorithms
• Build and train Dynamic Factor Models for nowcasting and forecasting applications
• Recognize when linear models fail and when nonlinear extensions (DDFMs) become necessary
• Implement practical solutions using the dfm-python package, from configuration to deployment

The journey from simple dimension reduction to sophisticated state-space inference may seem complex, but each step builds naturally on the previous. We start with static methods, add temporal structure, then introduce nonlinearity—always maintaining the interpretable factor model framework that makes these methods valuable for financial applications.

What Is State Space?

Financial markets are driven by forces we cannot directly observe. Volatility regimes shift between calm and turbulent periods, credit cycles move through expansion and contraction phases, and market sentiment fluctuates based on unobservable psychological factors. Yet these hidden drivers determine asset prices, economic indicators, and investment outcomes. The fundamental challenge in financial AI is extracting these latent states from noisy, incomplete observations.

State-space models provide the mathematical framework to bridge this gap. They describe systems where observable data $y_t$ depend on latent states $x_t$ that evolve through time. The transition equation captures how latent states evolve—how volatility regimes persist and transition, how credit cycles build and shift, how market sentiment changes. The observation equation links these hidden states to what we actually measure—stock prices, bond yields, economic indicators. This framework unifies many classical models (GARCH, ARIMA, structural time series) while extending naturally to modern applications (dynamic factor models, nowcasting, deep state-space models).

Motivation: Hidden States Matter in Finance

As we have seen in previous chapters, there have been consistent attempts to model latent dynamics, from early volatility models to modern factor extraction methods. ARCH and GARCH models [@engle1982arch; @bollerslev1986garch] represent one of the most well-known efforts, treating volatility as an unobservable process that evolves over time. The GARCH model captures volatility clustering—the tendency for high volatility periods to be followed by high volatility. This is a state-space structure: volatility is the latent state, returns are the observations, and the GARCH equations describe how volatility evolves. State-space models formalize these hidden processes and provide a unified framework that encompasses GARCH models as special cases while extending to more complex applications like dynamic factor models and nowcasting.

The Data Scarcity Challenge in Finance

Financial and economic data are fundamentally different from the large-scale datasets that power modern deep learning. Unlike image recognition or natural language processing, where millions of examples are readily available, financial time series are constrained by the nature of economic activity itself. GDP is published quarterly, employment data arrives monthly, and even daily stock prices provide only one observation per trading day. Over a decade, we might accumulate only 40 quarterly GDP observations, 120 monthly employment figures, or roughly 2,500 daily stock returns. This inherent data scarcity makes traditional "data-hungry" deep learning approaches challenging in finance.

The challenge is fundamental: financial and economic processes evolve continuously, but we observe them through sparse, noisy, and often delayed measurements. State-space models excel in this setting because they leverage structural assumptions about how latent states evolve, rather than relying solely on large datasets. They can work effectively with 50-200 time periods, making them suitable for macroeconomic applications where historical data is limited. This structural modeling approach is essential when data is scarce—by explicitly modeling the relationship between latent states and observations, state-space models extract maximum information from limited data.

State-space models provide interpretable parameters that teach us about latent dynamics: transition matrices reveal persistence and mean reversion, observation matrices show how indicators relate to underlying activity, and noise covariances quantify uncertainty. They naturally handle missing data, mixed frequencies, and measurement error through the Kalman filter framework, which we explore in detail in Section 6-02. This combination of structural modeling and efficient inference makes state-space models particularly valuable for financial applications where data is inherently limited but interpretability and uncertainty quantification are crucial.

What is a State? Intuition for Finance

Before diving into the mathematics, let's build intuition about what a "state" means in finance. Think of a state as a complete description of the financial system at a given moment—all the information needed to predict future asset prices and economic outcomes.

Consider a portfolio manager tracking their positions. The observable includes current portfolio value, daily returns, and transaction records. The latent state encompasses true risk exposure, hidden correlations, and regime-dependent betas. The portfolio's state includes not just current positions, but also the underlying risk factors that drive returns—market risk, sector exposure, style factors. Defining the latent state properly is crucial for good risk management and portfolio construction, as it reveals the true drivers of portfolio performance that are not directly visible in market prices.

This differs from simple feature engineering. While feature engineering creates new variables from existing data, state-space modeling treats certain quantities as fundamentally unobservable. The state is not just a transformation of observations—it represents the true underlying system that generates those observations. We cannot measure the state directly; we can only infer it from noisy, incomplete observations.

Similarly, the financial market has a state: observable stock prices, bond yields, and economic indicators reflect a latent state of true economic activity, volatility regime, credit cycle phase, and market sentiment. Just as a portfolio's state determines its future returns, the market's state determines how assets will evolve. The key insight: we can't directly observe the state, but we can infer it from observations using state-space methods.

States evolve over time following patterns: volatility regimes persist then transition during crises, credit cycles build pressure before shifting, and business cycles move through expansion, peak, contraction, and trough phases. In classical state-space modeling, we use the Markov property: today's state depends primarily on yesterday's state, not the entire history. This "memoryless" assumption, while seemingly restrictive, captures the essential dynamics while enabling tractable inference. In finance, this often holds approximately—volatility regimes depend primarily on yesterday's regime, credit conditions reflect recent developments, and common factors evolve based on current values with past information embedded in the current state.

Understanding latent states enables numerous financial AI applications. In risk management, we can identify regime shifts before they manifest in prices, allowing proactive risk adjustment. For portfolio construction, we allocate based on factor exposures revealed through state-space inference, optimizing risk-adjusted returns. Nowcasting allows us to estimate current economic conditions before official releases, providing timely information for investment decisions. Forecasting predicts future states and their impact on asset prices, enabling better investment strategies.

Historical Context and Origins

The mathematical foundation of state-space models came from Rudolf Kalman's work in the 1960s on optimal filtering for linear systems. Kalman's key insight—that optimal state estimation could be performed recursively, updating beliefs as new data arrives—proved powerful for financial applications. The recursive structure is computationally efficient, processing data in a single forward pass rather than requiring batch optimization.

Economists in the 1980s and 1990s recognized that many economic phenomena could be modeled as unobservable states driving observable outcomes. The state-space framework unified seemingly disparate models: ARIMA models, structural time-series models, and dynamic factor models all became special cases. Stock and Watson's work on dynamic factor models [@stock2002forecasting] demonstrated how state-space methods could extract common factors from large panels of economic data, laying the foundation for modern nowcasting systems used by central banks worldwide.

Financial applications expanded in the 1990s with stochastic volatility models, term structure models, and credit risk models. Recent developments since the 2000s have integrated state-space models with machine learning. Deep state-space models [@andreini2020deep; @rangapuram2018deep] combine the interpretability of classical state-space models with the flexibility of neural networks, enabling modeling of complex nonlinear relationships while maintaining factor structure. These developments address limitations of linear models during structural breaks, as evidenced by the COVID-19 experience when traditional models struggled with rapidly changing economic relationships.

Formal Definition

A general state-space model consists of two equations:

Transition equation (state dynamics):

\begin{aligned} x_t = F_t x_{t-1} + G_t u_t + w_t, \quad w_t \sim N(0, Q_t) \end{aligned}

Observation equation (measurement):

\begin{aligned} y_t = H_t x_t + v_t, \quad v_t \sim N(0, R_t) \end{aligned}

In these equations, $x_t\in {\mathbb{R}}^m$ represents the latent state vector at time $t$—the unobservable quantity we want to estimate. The observed data vector $y_t\in {\mathbb{R}}^n$ contains what we actually measure. The transition matrix $F_t\in {\mathbb{R}}^{m\times m}$ captures how the state persists and evolves from one period to the next, while the control matrix $G_t\in {\mathbb{R}}^{m\times p}$ maps any exogenous inputs $u_t$ into state changes. The observation matrix $H_t\in {\mathbb{R}}^{n\times m}$ links the hidden states to our measurements, revealing what aspects of the state we can observe. Process noise $w_t\in {\mathbb{R}}^m$ with covariance $Q_t\in {\mathbb{R}}^{m\times m}$ represents fundamental uncertainty in state evolution, while observation noise $v_t\in {\mathbb{R}}^n$ with covariance $R_t\in {\mathbb{R}}^{n\times n}$ captures measurement error.

The subscripts $t$ on the matrices indicate that these can vary over time, enabling time-varying dynamics and observation structures. When these matrices are constant, we have a time-invariant system, which is common in many financial applications. Time-varying matrices become essential for handling mixed-frequency data and regime-switching models where relationships change over time, such as during financial crises when factor loadings may shift dramatically.

The transition matrix $F_t$ determines how the latent state evolves, capturing persistence of volatility regimes, mean reversion of credit spreads, or momentum in factor returns. The eigenvalues of $F_t$ determine stability: for asymptotically stable systems, all eigenvalues must satisfy $|{\lambda }_i|<1$, ensuring state trajectories converge to a steady state. Eigenvalues near 1 indicate high persistence (near-nonstationarity), common in macroeconomics. The control matrix $G_t$ maps known interventions into state changes, such as central bank policy shocks. In many financial applications, control inputs are not used ($G_t=0$). The observation matrix $H_t$ defines what we can observe about the latent state. For factor models, this matrix contains factor loadings showing how each asset responds to common drivers, as we explore in Section 6-02.

The noise covariances $Q_t$ and $R_t$ encode uncertainty in two distinct forms. The process noise covariance $Q_t$ represents fundamental uncertainty in state evolution, while the observation noise covariance $R_t$ captures measurement error. The signal-to-noise ratio determines how much we trust observations versus predictions. Often $Q_t$ and $R_t$ are assumed diagonal, meaning uncorrelated innovations, which simplifies estimation and interpretation.

The Markov Property

Classical state-space models assume the Markov property: the future state depends only on the current state, not the entire history:

\begin{aligned} p(x_t | x_{1:t-1}) = p(x_t | x_{t-1}) \end{aligned}

This "memoryless" property formalizes that the current state contains all information needed to predict the future. The past matters only through its influence on the current state. The key insight is that the current state can encode information from the past. If volatility clustering is important, we can include past volatility in the state vector. If cumulative stress matters, we can include it as a state component. The Markov property doesn't mean the past is irrelevant—it means the past is summarized in the current state.

The Markov property enables recursive inference: we update beliefs about $x_t$ using only $x_{t-1}$ and $y_t$, maintaining a compact representation that summarizes all past information. In finance, the Markov property often holds approximately—volatility regimes depend primarily on yesterday's regime, credit conditions reflect recent developments, and common factors evolve based on current values with past information embedded in the current state. Some financial phenomena exhibit long memory or path dependence. In these cases, we can restore the Markov property by augmenting the state: include additional variables such as regime duration, cumulative stress, or moving averages so that the augmented state is Markovian.

The Markov property implies that the joint distribution of states factors as:

This factorization is the foundation for the Kalman filter, Kalman smoother, and EM algorithms which are fundamental to estimating dynamic state-space models. The Markov property enables efficient algorithms that process data sequentially rather than requiring batch optimization over all time periods. Instead of optimizing over the entire state sequence simultaneously, we can process data one time step at a time, updating our beliefs recursively as new information arrives.

Linear vs. Nonlinear Models

The advantage of linearity is interpretability. When transitions and observations are linear functions and noise is Gaussian, we obtain closed-form solutions for inference (the Kalman filter, which we explore in Section 6-02) that are both optimal and computationally efficient. Linear models are appropriate when relationships are approximately linear, Gaussian noise is reasonable, and computational efficiency is critical. The parameters have clear economic meaning: transition matrices show persistence and mean reversion, observation matrices reveal factor loadings, and noise covariances quantify uncertainty.

However, real-world financial dynamics are often nonlinear. Nonlinear models become necessary when regime switches occur, volatility clustering creates heteroskedasticity, or option surfaces exhibit complex nonlinearities. When regime shifts occur, previous parameter estimates become invalid, and simple demeaning or linear transformations don't capture the structural change. In this book, we introduce neural approximations for nonlinear state-space models. Deep Dynamic Factor Models (Section 6-04 and 6-05) use neural networks to capture nonlinear relationships while maintaining the interpretable factor structure. The linear-Gaussian framework provides the foundation that these methods extend, ensuring that our nonlinear models reduce to linear models when relationships are approximately linear.

Financial Application Examples

Volatility modeling provides a clear example of state-space thinking. GARCH models [@engle1982arch; @bollerslev1986garch] treat volatility as a latent state that evolves over time, with current volatility depending on past squared returns and past volatility. The GARCH(1,1) model can be written in state-space form where the latent state is the conditional variance, and the observation is the squared return. Stochastic volatility models extend this framework by representing log-volatility as a latent state: $\log {\sigma }_t=\varphi \log {\sigma }_{t-1}+w_t$ where $w_t\sim N(0,{\sigma }_w^2)$ and $r_t={\sigma }_t{\epsilon }_t$ where ${\epsilon }_t\sim N(0,1)$. The latent state $x_t=\log {\sigma }_t$ evolves as an AR(1) process with persistence $\varphi$ (typically 0.95-0.99), while observed returns $r_t$ depend on this hidden volatility through a multiplicative relationship $r_t=\exp (x_t){\epsilon }_t$. This framework underpins option pricing models and risk management systems, where accurate volatility estimation is crucial for pricing derivatives and calculating value-at-risk.

Hidden default intensity processes model credit cycles using state-space structure. The unobservable default intensity ${\lambda }_t$ evolves over time, driving observed defaults and credit spread movements: ${\lambda }_t=\varphi {\lambda }_{t-1}+w_t$ and $\text{Default}(t)\sim \text{Poisson}({\lambda }_t)$. When ${\lambda }_t$ is high, defaults are more frequent and credit spreads widen. When ${\lambda }_t$ is low, the credit environment is benign. This framework supports portfolio credit risk models and CDS pricing.

Key Properties of State-Space Systems

Three fundamental properties—observability, controllability, and stability—determine what we can learn from data, what we can control, and whether the system behaves well over time.

A system is observable if we can uniquely determine the initial state $x_0$ from a finite sequence of observations $y_{1:T}$. The observability matrix $\mathcal{O}=[H^T,(HF{)}^T,(H{F}^2{)}^T,\dots ,(H{F}^{n-1}{)}^T{]}^T$ must have full column rank (rank = $m$). In factor models, observability ensures that factor loadings are sufficiently diverse—if all assets load identically on factors, we cannot distinguish factor values from loadings.

A system is controllable if we can drive the state to any desired value using control inputs $u_t$ in finite time. The controllability matrix $\mathcal{C}=[G,FG,F^{2}G,\dots ,F^{m-1}G]$ must have full row rank (rank = $m$). Controllability matters for policy interventions, though many financial systems are observable but not fully controllable—we can estimate factors but cannot directly control them.

A system is stable if state trajectories remain bounded over time. For linear time-invariant systems, stability is determined by the eigenvalues of the transition matrix $F$: all eigenvalues must satisfy $|{\lambda }_i|<1$ for asymptotic stability. Eigenvalues near 1 indicate near-nonstationarity, common in macroeconomics where differencing may be required.

State-Space Models vs. Alternative Frameworks

Understanding when to use state-space models versus alternatives helps guide model selection in financial AI applications.

ARIMA models are special cases of state-space models, suitable for simple univariate forecasting with complete data. State-space models become preferable when handling missing data, building multivariate systems, or quantifying uncertainty in latent states—common requirements in financial applications.

VAR models [@sims1980macroeconomics] assume all variables are observable. State-space models allow latent states, enabling dimensionality reduction, handling missing data naturally, and working with mixed frequencies. Traditional machine learning focuses on prediction but provides limited interpretability. State-space models provide interpretable factors and uncertainty quantification, valuable for risk management and regulatory compliance. Hybrid approaches, such as the deep state-space models we explore in Section 6-04, combine both: neural networks capture complex relationships while maintaining interpretable factor structure [@andreini2020deep].

HMMs use discrete states for regime detection, while state-space models use continuous states, enabling factor extraction and nowcasting with richer dynamics. State-space models are particularly valuable when extracting latent factors, handling missing data, working with mixed frequencies, or requiring interpretable uncertainty quantification.

Finding Latent State

Now that we have expressed our system in state-space form (Section 6-01), we face the fundamental challenge: how do we extract the latent states from noisy, incomplete observations? In finance, we observe market prices, economic indicators, and financial time series, but we need to extract the hidden states that govern their evolution—volatility regimes, credit cycles, market sentiment, and common risk factors.

This section covers the fundamental methods for estimating state-space models, progressing systematically from the simplest approach (Principal Component Analysis) through recursive filtering (Kalman filter), smoothing (Kalman smoother), to joint state-parameter estimation (EM algorithm). Each method builds on the previous, solving increasingly complex inference problems. This progression from static to dynamic, from simple to sophisticated, mirrors the evolution of factor modeling in finance and sets the foundation for nonlinear extensions in Section 6-04.

The Estimation Challenge

Extracting latent states from financial data presents three fundamental challenges: observations are inherently noisy (market prices reflect trading noise and liquidity effects, economic indicators have measurement error), information is often incomplete (missing values, publication lags, mixed frequencies), and latent states are never directly measured—we only see their effects through observable variables. We cannot simply invert the observation equation because observations are noisy and the system is underdetermined.

We approach this through a hierarchy of methods. Principal Component Analysis (PCA) provides the foundation, treating each time period independently to extract static factors. The Kalman filter adds temporal structure, recursively estimating states as new data arrives. The Kalman smoother uses all observations to refine historical estimates. The EM algorithm jointly estimates states and parameters. Finally, variational inference and deep extensions (Section 6-04) handle nonlinear relationships. Each method builds on the previous, solving increasingly complex inference problems.

Principal Component Analysis: Linear Dimension Reduction

Principal Component Analysis (PCA) is the simplest method for extracting latent factors from observed data [@hotelling1933pca]. It finds linear combinations of observed variables that capture maximum variance, directly connecting to factor models. In finance, when assets move together during market-wide movements, PCA identifies these common directions as principal components. The first principal component often captures the "market factor"—the common driver affecting all assets—while remaining components capture progressively less important patterns. The computational efficiency of PCA (requiring only eigendecomposition) makes it ideal for initialization in more sophisticated methods.

Before applying PCA, we must preprocess the data. Given data $X\in {\mathbb{R}}^{T\times n}$ where each row is a time period and each column is a series, we first center the data: $X\leftarrow X-\bar{X}$ where $\bar{X}$ contains the column means. Centering is essential because the covariance matrix measures variation around the mean. In finance, we typically do not scale the data when working with returns, because asset returns are already on a similar scale. However, when combining series with very different units, scaling may be necessary to prevent high-variance series from dominating the analysis.

Eigenvalue Decomposition: Mathematical Foundation

Eigenvalue decomposition is the mathematical operation that underlies PCA. For a symmetric matrix $\Sigma$ (like a covariance matrix), we can decompose it as:

where:

• $P$ is an orthogonal matrix (columns are orthonormal eigenvectors): $P^{T}P=I$
• $\Lambda$ is a diagonal matrix containing eigenvalues: $\Lambda =\text{diag}({\lambda }_1,{\lambda }_2,...,{\lambda }_n)$ with ${\lambda }_1\ge {\lambda }_2\ge \cdots \ge {\lambda }_n\ge 0$

Why Eigenvalue Decomposition? The eigenvectors of the covariance matrix point in directions where data varies most. The first eigenvector (corresponding to the largest eigenvalue) is the direction of maximum variance. The second eigenvector (orthogonal to the first) captures the maximum remaining variance, and so on. This decomposition is unique (up to sign) for symmetric matrices.

Geometric Interpretation: Think of the data as a cloud of points. The eigenvectors are the principal axes of this cloud. The eigenvalues tell us how "spread out" the data is along each axis. PCA finds these principal axes automatically.

PCA proceeds through eigenvalue decomposition of the covariance matrix. For centered data $X$, we compute the covariance matrix:

which captures how each pair of series co-varies. The decomposition:

yields eigenvectors $P$ (principal directions, orthonormal columns) and eigenvalues $\Lambda$ (variances along each direction, ordered ${\lambda }_1\ge {\lambda }_2\ge \cdots \ge {\lambda }_n\ge 0$). The eigenvectors point in directions where data varies most, with the first eigenvector capturing maximum variance, the second capturing maximum remaining variance orthogonal to the first, and so on.

Complete PCA Example: Step-by-Step Calculation

Let's work through a concrete numerical example to illustrate PCA.

Given Data: 3 asset returns over 5 time periods (after centering):

Step 1: Compute Covariance Matrix

Step 2: Eigenvalue Decomposition

Solving $\Sigma P=P\Lambda$ (or equivalently $(\Sigma -\lambda I)\mathbf{p}=0$), we find:

Eigenvalues (ordered): ${\lambda }_1=1.124,{\lambda }_2=0.000,{\lambda }_3=0.000$

Eigenvectors (normalized):

Step 3: Extract First Principal Component

Using $k=1$ (one factor):

Step 4: Compute Factor Values

Project data onto first eigenvector:

For $t=1$: $f_1=0.816(1.0)+0.408(0.5)+0.408(0.8)=1.224$

Computing for all periods:

Step 5: Reconstruct Data

For $t=1$:

The reconstruction is close to the original (since first PC explains most variance).

Step 6: Variance Explained

Total variance: $\text{tr}(\Sigma )=0.596+0.148+0.380=1.124$

Variance explained by first PC: ${\lambda }_1=1.124$

Proportion explained: $1.124/1.124=100\%$ (in this example, data lies on a line, so one PC explains everything)

Why Project? Geometric and Algebraic Interpretation

Geometric Interpretation: Projecting data onto eigenvectors means finding coordinates in a new coordinate system. The original data lives in $n$-dimensional space (one dimension per series). The eigenvectors define a new coordinate system aligned with directions of maximum variation. Projecting means finding where each data point lies in this new coordinate system.

Algebraic Interpretation: The projection $f_t=P_k^T{x}_t$ computes the dot product between the data vector $x_t$ and each eigenvector (column of $P_k$). This measures how much the data aligns with each principal direction. Large values mean the data varies strongly in that direction.

Why This Works: Since eigenvectors point in directions of maximum variance, projecting onto them captures the most important patterns in the data. We discard dimensions with little variance (noise) and keep dimensions with high variance (signal).

We project data onto the first $k$ eigenvectors to get factor values:

where $P_k$ contains the first $k$ columns of $P$ and $f_t\in {\mathbb{R}}^k$ are the factor values at time $t$. The factors reconstruct the original data via:

Understanding Factor Loadings

Factor loadings are the coefficients that tell us how much each observed series responds to each factor. In the matrix $P_k$, element $P_k[i,j]$ is the loading of series $i$ on factor $j$.

Interpretation:

• High positive loading (e.g., 0.8): Series $i$ moves strongly in the same direction as factor $j$
• Low loading (e.g., 0.1): Series $i$ is relatively insensitive to factor $j$
• Negative loading (e.g., -0.5): Series $i$ moves opposite to factor $j$

Example: If $P_k[1,1]=0.816$ and $P_k[2,1]=0.408$, then series 1 loads twice as strongly on factor 1 as series 2. When factor 1 increases by 1 unit, series 1 increases by 0.816 units, while series 2 increases by 0.408 units.

where $P_k$ contains factor loadings: $P_k[i,j]$ tells us how much series $i$ loads on factor $j$. Projecting data onto eigenvectors means finding coordinates in a new coordinate system aligned with directions of maximum variation. In financial terms, this identifies the common risk factors that drive asset returns.

Two Equivalent Formulations of PCA

PCA has two equivalent formulations that lead to the same solution:

1. Variance Maximization: Find directions (eigenvectors) that maximize variance of projected data:

2. Reconstruction Error Minimization: Find directions that minimize reconstruction error:

Why They're Equivalent: Maximizing variance is equivalent to minimizing reconstruction error. This is because:

Total variance is fixed, so maximizing $\text{Var}({\hat{x}}_t)$ (variance of reconstruction) minimizes $\text{Var}(x_t-{\hat{x}}_t)$ (reconstruction error).

Mathematical Proof: The variance maximization problem leads to the eigenvalue equation $\Sigma \mathbf{p}=\lambda \mathbf{p}$, which is solved by eigenvectors. The reconstruction error minimization problem leads to the same eigenvalue equation. Therefore, both formulations yield identical solutions.

This fundamental property means PCA simultaneously maximizes variance in the latent space and minimizes reconstruction error in the original space.

The proportion of variance explained by the $i$-th principal component is ${\lambda }_i/{\sum }_{j=1}^n{\lambda }_j$, and the cumulative variance explained by the first $k$ components is ${\sum }_{i=1}^k{\lambda }_i/{\sum }_{j=1}^n{\lambda }_j$. In equity returns, the first PC often explains 30-50% of variance (market factor), the second 10-15% (size or sector factor), and subsequent components progressively less [@stock2002forecasting]. This rapid decay justifies using only a few factors, allowing investors to focus on key drivers rather than tracking hundreds of individual assets.

PCA factors are linear combinations of returns:

where $P_{ij}$ is the loading of series $i$ on factor $j$. In factor model notation, $x_t=P_k{f}_t+{\epsilon }_t$ where $\Lambda =P_k$ (loadings are eigenvectors), $f_t=P_k^T{x}_t$ (factors are projections), and ${\epsilon }_t=x_t-P_k{P}_k^T{x}_t$ (reconstruction error). The total variance decomposes as:

where the first term is factor variance (systematic risk) and the second is idiosyncratic variance (diversifiable risk). This decomposition is fundamental to portfolio theory: factor risk cannot be diversified away, while idiosyncratic risk can be reduced through diversification [@fama1992common].

Despite our use of time-series data, PCA treats each time period independently—this is static dimension reduction, not dynamic modeling. PCA computes factors $f_t$ from cross-sectional data at time $t$, with no connection to factors at other time periods. This static approach ignores temporal dependencies that are crucial in finance: volatility clustering, regime switches, momentum, and mean reversion. To estimate dynamic models, we need methods that explicitly model how states evolve: $x_t$ depends on $x_{t-1}$ through the transition equation, as we defined in Section 6-01. This motivates the Kalman filter, which captures temporal dependencies while maintaining the factor structure.

Despite its limitations, PCA provides crucial initial values for the EM algorithm through a three-step process: extract first $k$ principal components to get initial loadings, use PCA factors as initial factor estimates, and regress factors on lagged factors to get initial transition matrix. This PCA initialization is crucial for EM convergence, providing reasonable starting values that the algorithm then refines. Without good initialization, EM may converge to poor local optima or fail to converge at all, making PCA initialization essential for practical applications.

From States to Factors: Core Concepts

Before introducing dynamic methods, we clarify the terminology: "states" and "factors" both refer to latent variables, but with different emphasis. Factors are the conceptual drivers (e.g., "market risk", "credit cycle") that affect multiple observed series. States are the time-evolving realizations of these factors (e.g., "market risk is high today"). In state-space models, states evolve through time following the transition equation, while factors are the underlying drivers that states represent. Factor loadings measure how much each observed series responds to each factor, appearing in the observation matrix $H_t$.

A factor model expresses returns as:

where $r_{i,t}$ is the return of asset $i$ at time $t$, ${\alpha }_i$ is the asset-specific intercept, ${\beta }_i\in {\mathbb{R}}^k$ are factor loadings (sensitivities to factors), $f_t\in {\mathbb{R}}^k$ are common factors (latent drivers), and ${\epsilon }_{i,t}$ is the idiosyncratic return (asset-specific shock). Assuming factors and idiosyncratic returns are uncorrelated, the variance decomposes as:

where ${\Sigma }_f$ is the factor covariance matrix. The term ${\beta }_i^T{\Sigma }_f{\beta }_i$ is factor risk (systematic risk), and ${\sigma }_{{\epsilon }_i}^2$ is idiosyncratic risk (diversifiable risk). This decomposition is fundamental to portfolio theory: factor risk cannot be diversified away because all assets load on common factors, while idiosyncratic risk can be reduced through diversification as asset-specific shocks average out. In volatile markets, factor loadings may evolve—assets may become more or less sensitive to market risk during crises, a phenomenon known as "beta instability." In the current linear setup, loadings are constant, which is easy to interpret but does not model evolving sensitivities, motivating nonlinear extensions (Section 6-04) that allow loadings to depend on the state or regime.

State Estimation: Kalman Filter

The Kalman filter provides optimal state estimation for linear-Gaussian state-space models. We compute the posterior $p(x_t|y_{1:t})=N({\hat{x}}_{t|t},P_{t|t})$ recursively using Bayesian updating.

Derivation from Bayes' theorem: The posterior combines prediction (prior) and observation (likelihood):

Complete Kalman Filter Example: Step-by-Step Calculation

Let's work through a numerical example with a simple 1D state-space model.

Model Setup:

• State equation: $x_t=0.9x_{t-1}+w_t$, $w_t\sim N(0,0.1)$
• Observation equation: $y_t=x_t+v_t$, $v_t\sim N(0,0.5)$
• Initial state: $x_0\sim N(0,1)$
• Observations: $y_1=0.8,y_2=0.6,y_3=0.4$

Parameters:

• $F=0.9$ (transition)
• $H=1$ (observation matrix)
• $Q=0.1$ (process noise variance)
• $R=0.5$ (observation noise variance)
• ${\hat{x}}_{0|0}=0,P_{0|0}=1$ (initial state)

Time $t=1$:

Prediction Step:

Update Step: Kalman gain:

Innovation:

Posterior mean:

Posterior covariance:

Time $t=2$:

Prediction Step:

Update Step:

Time $t=3$:

Prediction Step:

Update Step:

Interpretation: The filter recursively updates state estimates as new observations arrive. Notice how:

• Uncertainty decreases after each update ($P_{t|t}<P_{t|t-1}$)
• Kalman gain decreases over time (more confident predictions)
• State estimates track observations while smoothing noise

For linear-Gaussian systems, both terms are Gaussian, so the posterior is also Gaussian. The prediction step propagates the previous posterior through the transition:

Detailed Derivation of Prediction Step

The prediction step computes $p(x_t|y_{1:t-1})$ by integrating over the previous state:

Why This Integral? We're marginalizing over $x_{t-1}$: we don't know the exact previous state, only its distribution. We average over all possible values of $x_{t-1}$, weighted by their probability.

Mathematical Derivation: Since $p(x_t|x_{t-1})=N(F_t{x}_{t-1},Q_t)$ and $p(x_{t-1}|y_{1:t-1})=N({\hat{x}}_{t-1|t-1},P_{t-1|t-1})$, we have:

The mean of $x_t$ is:

The covariance of $x_t$ is:

Therefore, the integral yields:

where we've included the control term $G_t{u}_t$ (often zero in financial applications).

The update step combines prediction and observation. The likelihood is $p(y_t|x_t)=N(H_t{x}_t,R_t)$, and the prior is $p(x_t|y_{1:t-1})=N({\hat{x}}_{t|t-1},P_{t|t-1})$. The posterior mean and covariance are:

Detailed Derivation of Kalman Gain

The Kalman gain $K_t$ is derived by minimizing the posterior covariance $P_{t|t}$. Let's derive it step by step.

Step 1: Posterior Mean

The posterior mean combines prediction and observation:

where $K_t$ is a matrix to be determined. The innovation $y_t-H_t{\hat{x}}_{t|t-1}$ measures prediction error.

Step 2: Posterior Covariance

The posterior covariance is:

Substituting the update equation:

Expanding and using $y_t=H_t{x}_t+v_t$:

Step 3: Minimize Trace

To minimize uncertainty, we minimize $\text{tr}(P_{t|t})$ (trace = sum of diagonal = sum of variances).

Taking derivative with respect to $K_t$ and setting to zero:

Solving for $K_t$:

Step 4: Intuition

The Kalman gain balances two sources of information:

• Prediction uncertainty $P_{t|t-1}$: Large uncertainty → trust observations more → larger $K_t$
• Observation uncertainty $R_t$: Large uncertainty → trust prediction more → smaller $K_t$

Limiting Cases:

• $R_t\to \infty$ (very noisy observations): $K_t\to 0$ (ignore observations, use prediction)
• $P_{t|t-1}\to \infty$ (very uncertain prediction): $K_t\to H_t^{-1}$ (trust observations completely, if $H_t$ is invertible)

The innovation $y_t-H_t{\hat{x}}_{t|t-1}$ measures prediction error. The gain balances prediction and observation uncertainty: $K_t\to 0$ when $R_t$ is large (unreliable data), and $K_t\to H_t^{-1}$ when $P_{t|t-1}$ is large (uncertain prediction).

The filter provides MMSE estimates: $E[x_t|y_{1:t}]={\hat{x}}_{t|t}$ minimizes $E[||x_t-{\hat{x}}_{t|t}|{|}^2]$. Missing observations are handled by setting $R_t[i,i]=\infty$ for missing series $i$, making $K_t[i,:]=0$ and ignoring that observation.

Kalman Smoother: Using All Information

Detailed Derivation of Kalman Smoother

The smoother estimates $p(x_t|y_{1:T})$ using all observations, refining past estimates with future information.

Intuition: The filter uses only past and current observations ($y_{1:t}$). The smoother uses all observations ($y_{1:T}$), including future ones. This allows us to refine past estimates: if we know what happened later, we can better estimate what the state was earlier.

Mathematical Derivation: We factor the joint distribution:

The key insight: $x_t$ is conditionally independent of future observations $y_{t+1:T}$ given $x_{t+1}$ (Markov property). So $p(x_t|x_{t+1},y_{1:T})=p(x_t|x_{t+1},y_{1:t})$.

Deriving the Conditional: The conditional $p(x_t|x_{t+1},y_{1:t})$ is derived from the joint $p(x_t,x_{t+1}|y_{1:t})$. Using properties of multivariate Gaussians:

where $J_t=P_{t|t}{F}_{t+1}^T{P}_{t+1|t}^{-1}$ is the smoother gain (analogous to Kalman gain, but for backward propagation).

Integrating: Integrating over $x_{t+1}\sim p(x_{t+1}|y_{1:T})$ yields the Rauch-Tung-Striebel (RTS) smoother equations:

Interpretation: The smoother corrects filter estimates using future information: ${\hat{x}}_{t|T}={\hat{x}}_{t|t}+\text{correction}$. The correction term $J_t({\hat{x}}_{t+1|T}-{\hat{x}}_{t+1|t})$ propagates future information backward. If the future smoothed estimate differs from the filter's prediction, we adjust the current estimate accordingly.

Complete Kalman Smoother Example

Continuing from the Kalman filter example above, let's compute smoothed estimates.

From Filter (we computed earlier):

• $t=1$: ${\hat{x}}_{1|1}=0.516,P_{1|1}=0.323,{\hat{x}}_{1|0}=0,P_{1|0}=0.91$
• $t=2$: ${\hat{x}}_{2|2}=0.521,P_{2|2}=0.210,{\hat{x}}_{2|1}=0.464,P_{2|1}=0.362$
• $t=3$: ${\hat{x}}_{3|3}=0.445,P_{3|3}=0.175,{\hat{x}}_{3|2}=0.469,P_{3|2}=0.270$

Backward Pass (starting from $t=3$):

Time $t=3$: Already at end, so ${\hat{x}}_{3|3}={\hat{x}}_{3|T}=0.445,P_{3|3}=P_{3|T}=0.175$

Time $t=2$: Smoother gain:

Smoothed estimate:

Smoothed covariance:

Time $t=1$:

Key Observations:

• Smoothed estimates are more accurate (use all information)
• Smoothed covariances are smaller (less uncertainty)
• Estimates are refined using future information
• The smoother is essential for EM, which requires $E[x_t|y_{1:T}]$ and $E[x_t{x}_{t-1}^T|y_{1:T}]$

Expectation-Maximization Algorithm: Complete Mathematical Pipeline

EM estimates parameters $\theta =\{F,H,Q,R\}$ by iterating between state estimation (E-step) and parameter estimation (M-step). The likelihood $p(y_{1:T}|\theta )$ is intractable (requires integrating over all possible state sequences), but the complete-data likelihood factors as:

Why EM? We can't directly maximize $p(y_{1:T}|\theta )$ because states $x_t$ are unobserved. EM works with the complete-data likelihood $p(x_{1:T},y_{1:T}|\theta )$, which factors nicely, and handles missing states by taking expectations.

E-Step: State Estimation

E-step: Compute expected complete-data log-likelihood:

This requires smoothed moments: $E[x_t|y_{1:T}]$, $E[x_t{x}_t^T|y_{1:T}]$, and $E[x_t{x}_{t-1}^T|y_{1:T}]$, computed via the Kalman smoother.

Mathematical Details: Expanding the expectation:

Each term involves expectations over the smoothed distribution $p(x_t|y_{1:T})$, which we compute using Kalman smoother outputs.

M-Step: Parameter Estimation

M-step: Maximize $Q(\theta |{\theta }^{(k)})$ with respect to $\theta$. For linear-Gaussian systems, this yields closed-form solutions.

Derivation for Transition Matrix $F$:

The relevant term is:

Taking derivative with respect to $F$ and setting to zero:

Solving:

Therefore:

This is the regression of $x_t$ on $x_{t-1}$ using smoothed moments (weighted by uncertainty).

Derivation for Observation Matrix $H$:

Similarly, for the observation equation:

Derivation for Covariances:

For process noise $Q$:

For observation noise $R$:

The covariances are residual covariances after accounting for estimated relationships.

Complete EM Algorithm Example

Let's work through a complete EM iteration with numerical values.

Initialization (PCA):

• Data: 3 series, 5 time periods
• PCA extracts: $P_1=[0.816,0.408,0.408{]}^T$ (first eigenvector)
• Initial factors: $f_t^{(0)}=P_1^T{x}_t$ for $t=1,...,5$
• Initial loadings: $H^{(0)}=P_1$
• Initial transition: $F^{(0)}=0.9$ (from regressing $f_t$ on $f_{t-1}$)
• Initial covariances: $Q^{(0)}=0.1,R^{(0)}=0.5$

EM Iteration 1:

E-Step: Run Kalman filter and smoother (as in examples above) to get:

• $E[f_t|y_{1:5}]$ for $t=1,...,5$
• $E[f_t{f}_t^T|y_{1:5}]$ for $t=1,...,5$
• $E[f_t{f}_{t-1}^T|y_{1:5}]$ for $t=2,...,5$

M-Step: Update parameters

Update $F$:

Suppose smoothed moments give:

• ${\sum }_{t=2}^{5}E[f_t{f}_{t-1}|y_{1:5}]=1.85$
• ${\sum }_{t=2}^{5}E[f_{t-1}^2|y_{1:5}]=2.05$

Then: $F^{(1)}=1.85/2.05=0.902$

Update $H$:

Suppose:

• ${\sum }_{t=1}^5{y}_{t}E[f_t|y_{1:5}]=[2.1,1.05,1.05{]}^T$
• ${\sum }_{t=1}^{5}E[f_t^2|y_{1:5}]=2.5$

Then: $H^{(1)}=[0.84,0.42,0.42{]}^T$

Update $Q$:

Suppose this equals 0.095.

Update $R$:

Suppose this equals 0.48 (diagonal matrix).

Convergence Check:

• Compute log-likelihood $\ell ({\theta }^{(1)})$
• Check: $|\ell ({\theta }^{(1)})-\ell ({\theta }^{(0)})|<ϵ$?
• If not converged, set $k=2$ and repeat

Key Properties:

• EM guarantees $\log p(y_{1:T}|{\theta }^{(k+1)})\ge \log p(y_{1:T}|{\theta }^{(k)})$, converging to a local maximum
• Each iteration improves (or maintains) the likelihood
• Convergence is typically achieved in 10-100 iterations

PCA Initialization: Detailed Procedure

The EM algorithm is sensitive to initialization. PCA initialization provides excellent starting values through a three-step process:

Step 1: Extract Initial Loadings

• Apply PCA to centered data $X$
• Extract first $k$ principal components: $P_k\in {\mathbb{R}}^{n\times k}$
• Set initial loadings: $H^{(0)}=P_k$

Step 2: Extract Initial Factors

• Project data onto first $k$ eigenvectors: $f_t^{(0)}=P_k^T{x}_t$
• This gives initial factor estimates for all $t=1,...,T$

Step 3: Estimate Initial Transition Matrix

• Regress factors on lagged factors: $f_t^{(0)}=A{f}_{t-1}^{(0)}+\text{error}$
• OLS solution: $F^{(0)}=({\sum }_{t=2}^T{f}_t^{(0)}(f_{t-1}^{(0)}{)}^T)({\sum }_{t=2}^T{f}_{t-1}^{(0)}(f_{t-1}^{(0)}{)}^T{)}^{-1}$

Step 4: Initialize Covariances

• $Q^{(0)}$: Residual covariance from factor regression
• $R^{(0)}$: Residual covariance from observation equation

This initialization is crucial because PCA factors capture the main directions of variation, and the algorithm starts close to a good solution. Without good initialization, EM may converge to poor local optima or fail to converge at all.

EM Algorithm Limitations and Strengths

Limitations:

• May converge to local optima (not global maximum)
• Convergence can be slow (10-100 iterations)
• Each iteration requires full forward-backward pass through Kalman filter and smoother (computationally expensive)

Strengths:

• Provides closed-form updates (no numerical optimization needed)
• Guarantees monotonic likelihood increase
• Works well with proper initialization (PCA)
• Handles missing data naturally (via Kalman filter)
• Standard method for linear dynamic factor models

Despite limitations, EM remains the standard method for estimating linear dynamic factor models because it provides closed-form updates, guarantees monotonic likelihood increase, and works well with proper initialization.

Dynamic Factor Model: Practical Guide

This section provides a hands-on tutorial for building Dynamic Factor Models (DFMs) with the dfm-python package. We follow the workflow used by the Federal Reserve Bank of New York [@frbnynowcast] for nowcasting and forecasting. By the end, you'll be able to extract latent factors from mixed-frequency data and use them for forecasting.

A complete working example is available in codes/6_modeling_dynamics_3_dfm.py, tested with dfm-python version 0.4.51.

Recap: Why Dynamic Factor Models?

A Dynamic Factor Model has two main equations building on the state-space framework (Section 6-01):

Factor dynamics: $f_t=A{f}_{t-1}+w_t,w_t\sim N(0,Q)$. Factors $f_t$ (typically 1-5) follow an autoregressive process with transition matrix $A$ and innovations $w_t$ with covariance $Q$.

Observation equation: $y_t=\Lambda f_t+{\epsilon }_t,{\epsilon }_t\sim N(0,R)$. Observed series $y_t$ are linear combinations of factors weighted by loading matrix $\Lambda$, with observation errors ${\epsilon }_t$ having covariance $R$.