Sequences & Time Series

A standard MLP takes a fixed-size input and produces a fixed-size output. That is fatal when your input is a sentence, a stock price history, an audio waveform, or a video. These have variable length, and the order of the tokens matters. You can pad to a fixed length and feed it through an MLP, but you will pay the full quadratic-in-length parameter cost every time, and the model has no inductive bias that says "the same operation applied at every position".

Karpathy's framing in 24-founder-blogs/karpathy-rnn-effectiveness §sequences lists five sequence regimes worth distinguishing: one-to-many (image captioning), many-to-one (sentiment classification), many-to-many synchronised (frame-by-frame video tagging), many-to-many shifted (translation), and many-to-many same-length (POS tagging). All of them are awkward for an MLP. All of them are natural for a recurrent network.

A recurrent network does one thing differently: it maintains a hidden state that is updated at every position, using the same weights at every position. Formally, given a sequence $x_1, x_2, \ldots, x_T$:

where $f$ is a learned function. The output at position $t$ is some function of $h_t$. The same parameters are shared across all positions. This is the recurrent analogue of the convolutional weight-sharing trick: the architecture exploits the fact that the same operation applies at every step. Karpathy's one-liner is worth memorising: "if training vanilla neural nets is optimization over functions, training recurrent nets is optimization over programs". An RNN is closer to a tiny CPU with one fixed instruction.

The simplest recurrent cell uses a single linear layer on the concatenated previous hidden state and current input, followed by a tanh:

The output, if the task is many-to-many synchronised:

That is the whole equation. Three weight matrices, two bias vectors, one nonlinearity. Karpathy's blog has this in 4 lines of NumPy.

rnn = nn.RNN(input_size=64, hidden_size=128, num_layers=1, batch_first=True, nonlinearity='tanh')
output, h_final = rnn(x)   # x: (32, 50, 64)
# output: (32, 50, 128); h_final: (1, 32, 128). Add a separate nn.Linear(128, vocab) for the y head.

class VanillaRNN:
    def __init__(self, vocab_size, hidden_size):
        self.W_hh = np.random.randn(hidden_size, hidden_size) * 0.01
        self.W_xh = np.random.randn(hidden_size, vocab_size) * 0.01
        self.W_hy = np.random.randn(vocab_size, hidden_size) * 0.01
        self.b_h = np.zeros((hidden_size, 1))
        self.b_y = np.zeros((vocab_size, 1))

    def forward(self, x_seq):
        h = np.zeros((self.W_hh.shape[0], 1))
        hs, ys = [], []
        for x in x_seq:
            h = np.tanh(self.W_hh @ h + self.W_xh @ x + self.b_h)
            y = self.W_hy @ h + self.b_y
            hs.append(h); ys.append(y)
        return hs, ys

That is the whole API. nn.RNN with nonlinearity='tanh' is the equation above, vectorised across batch and time.

To train an RNN you run it forward over the entire sequence, accumulate the loss at every timestep, and then run gradient flow backwards through the same shared weights at every timestep. This is called backpropagation through time (BPTT). The unrolling is conceptual; in code it is the same backprop you already know, but with loss.backward traversing $T$ time steps before reaching the parameters.

The gradient of the loss at position $t$ with respect to the hidden state at position $k < t$ involves a product of Jacobians:

Each Jacobian factor has the form $\text{diag}(\tanh'(\cdot)) W_{hh}$. The product is therefore governed by the spectral norm of these factors — the largest factor by which the matrix can stretch a vector, equal to its largest singular value (for $W_{hh}$ alone, the relevant quantity is its largest-magnitude eigenvalue, the spectral radius). If that norm is below 1, the product shrinks exponentially toward zero as the gap $t - k$ grows; if above 1, it explodes. This is the matrix version of d2l's observation that repeated powers of $W_{hh}^\top$ have "eigenvalues smaller than 1 vanish and eigenvalues larger than 1 diverge". Both are bad. The vanishing case means the network cannot learn dependencies more than ≈20 steps apart. The exploding case means training diverges.

Practical fixes:

• Gradient clipping. Cap the global gradient norm at some threshold (typically 1.0 or 5.0). This stops explosions cold. Pascanu et al. 2013 popularised it.
• Truncated BPTT. Instead of unrolling over the entire sequence, unroll over a fixed window (say 35 steps) and detach the hidden state between windows. This is what every production RNN does.
• Smarter cells (LSTM, GRU). The structural fix for vanishing gradients, covered next.

# Truncated BPTT in PyTorch
hidden = None
for chunk in chunks_of_sequence:
    if hidden is not None:
        hidden = hidden.detach()   # cut the gradient graph
    output, hidden = rnn(chunk, hidden)
    loss = criterion(output, target_chunk)
    loss.backward()
    torch.nn.utils.clip_grad_norm_(rnn.parameters(), max_norm=1.0)
    optimizer.step()
    optimizer.zero_grad()

The vanishing-gradient problem in RNNs is not a hyperparameter issue. It is a structural property of the architecture. The hidden state at position $t$ has to flow through a $\tanh$ at every position from 1 to $t$. The derivative of $\tanh$ is bounded by 1 (and is much smaller almost everywhere). The product of $t$ such terms, multiplied by powers of $W_{hh}$, shrinks fast.

Olah's 24-founder-blogs/olah-2015-08-understanding-lstms §long-term-dependencies has the cleanest informal statement: "The clouds are in the sky" can be predicted from local context, but "I grew up in France. I speak fluent French" requires the model to remember "France" across an unbounded number of intervening tokens. RNNs in theory can; in practice they cannot.

The LSTM and GRU fix this by introducing a separate cell-state pathway that has linear (additive) updates instead of multiplicative tanh squashes. Gradients can flow through the cell state without being attenuated at every step. We will derive this next.

The LSTM (Hochreiter and Schmidhuber 1997) keeps two state vectors at every timestep: a cell state $c_t$ and a hidden state $h_t$. The cell state is the long-term memory; the hidden state is the working memory and the output.

Four gates, each a sigmoid-activated linear function of $[h_{t-1}, x_t]$:

• $f_t$ (forget gate): which cell-state components to keep
• $i_t$ (input gate): which new candidate values to add
• $\tilde{c}_t$ (candidate values): the tanh-squashed proposal
• $o_t$ (output gate): which cell-state components to expose as the hidden state

The update:

The key line is the cell-state update: $c_t = f_t \odot c_{t-1} + i_t \odot \tilde{c}_t$. This is an additive update on the cell state, gated multiplicatively. If $f_t \approx 1$ and $i_t \approx 0$, then $c_t \approx c_{t-1}$ and information from many timesteps ago flows forward untouched. Gradients flow backward through this addition without the tanh-derivative attenuation. That is the entire trick.

lstm = nn.LSTM(input_size=embed, hidden_size=hidden, batch_first=True)
output, (h_n, c_n) = lstm(x)   # x: (B, T, embed)
# output: (B, T, hidden); h_n, c_n: (1, B, hidden)
# single-step equivalent: cell = nn.LSTMCell(embed, hidden); h, c = cell(x_t, (h, c))

class LSTMCell(nn.Module):
    def __init__(self, input_size, hidden_size):
        super().__init__()
        self.gates = nn.Linear(input_size + hidden_size, 4 * hidden_size)
        with torch.no_grad():
            self.gates.bias.zero_()
            self.gates.bias[:hidden_size].fill_(1.0)   # forget-gate bias = 1
        self.hidden_size = hidden_size

    def forward(self, x, state):
        h, c = state
        combined = torch.cat([h, x], dim=-1)
        proj = self.gates(combined)
        f, i, g, o = proj.split(self.hidden_size, dim=-1)
        f = torch.sigmoid(f); i = torch.sigmoid(i)
        g = torch.tanh(g);    o = torch.sigmoid(o)
        c_new = f * c + i * g
        h_new = o * torch.tanh(c_new)
        return h_new, c_new

The GRU (Cho et al. 2014) is the LSTM with one fewer gate. It merges the cell state and hidden state into a single state vector, and uses only two gates: an update gate $z_t$ (how much of the new candidate to mix in) and a reset gate $r_t$ (how much of the previous state to use when computing the candidate).

GRUs have about 75% of the parameters of an equivalent LSTM (three weight matrices versus four). On most sequence tasks the accuracy difference is within noise. The original GRU paper and subsequent comparisons (16-d2l-sections/chapter_recurrent-modern__gru §comparison) found GRU faster to train and marginally weaker on the hardest long-range tasks. Use whichever your library defaults to; do not spend cycles choosing between them.

Library path:

gru = nn.GRU(input_size=64, hidden_size=128, batch_first=True)
x = torch.randn(32, 50, 64)
output, h_n = gru(x)
# Same API as nn.RNN, fewer parameters than nn.LSTM

For tasks where you have the full sequence available at inference time (POS tagging, named-entity recognition, sequence classification), running an RNN in both directions and concatenating the hidden states gives you context from both sides of each position.

bilstm = nn.LSTM(input_size=64, hidden_size=128, bidirectional=True, batch_first=True)
x = torch.randn(32, 50, 64)
output, _ = bilstm(x)
# output: (32, 50, 256) — forward 128 + backward 128 concatenated

Bidirectional only works for non-causal tasks. You cannot use it for autoregressive language modeling, where position $t$ should not see position $t + k$. For tagging, classification, or any task where the entire input is available, bidirectional is strictly better than unidirectional.

You can stack RNN layers the way you stack MLP layers. Layer 1's output sequence becomes layer 2's input sequence. Each layer has its own hidden state. Stacking gives the model more representational capacity per timestep.

deep_lstm = nn.LSTM(input_size=64, hidden_size=128, num_layers=3, batch_first=True, dropout=0.3)
x = torch.randn(32, 50, 64)
output, (h_n, c_n) = deep_lstm(x)
# h_n, c_n: (3, 32, 128) — one (h, c) pair per layer

In practice, 2 to 3 layers is the standard. Beyond that, the marginal gain shrinks and training stability decreases. Note the dropout argument: PyTorch's nn.LSTM applies dropout between layers (not within a layer's recurrent step). Recurrent dropout (within a step, applied with the same mask across timesteps) requires manual implementation or third-party libraries.

For tasks where the input and output sequences have different lengths and you need to produce the output one token at a time (translation, summarisation, dialog), the standard pre-transformer architecture was the encoder-decoder RNN (Sutskever et al. 2014, Cho et al. 2014).

The encoder is an RNN that reads the input sequence and compresses it into a single context vector (typically the final hidden state). The decoder is another RNN that takes that context vector as its initial hidden state and generates the output sequence one token at a time, conditioned on its own previous outputs.

class EncoderDecoder(nn.Module):
    def __init__(self, src_vocab: int, tgt_vocab: int, d: int = 256):
        super().__init__()
        self.src_emb = nn.Embedding(src_vocab, d)
        self.tgt_emb = nn.Embedding(tgt_vocab, d)
        self.encoder = nn.GRU(d, d, batch_first=True)
        self.decoder = nn.GRU(d, d, batch_first=True)
        self.head = nn.Linear(d, tgt_vocab)

    def forward(self, src_ids, tgt_ids):
        _, h_enc = self.encoder(self.src_emb(src_ids))
        out, _ = self.decoder(self.tgt_emb(tgt_ids), h_enc)
        return self.head(out)

Here nn.Embedding(vocab, d) is a learned lookup table — a $(\text{vocab} \times d)$ weight matrix whose $i$-th row is the $d$-dimensional feature vector for token index $i$, fetched directly by index and trained with the rest of the model. (This is the token-embedding sense, distinct from the t-SNE/UMAP "embedding" of another chapter.) A token is one discrete unit of the sequence — a character or word — and the vocabulary is the fixed set of tokens the model can read or emit.

At training time, the decoder is fed the ground-truth previous tokens (teacher forcing); at inference it consumes its own previous predictions instead, which is where errors compound. Generating one token at a time by feeding each prediction back in is autoregressive decoding, and taking the single highest-probability token at every step (the strategy used in this chapter) is greedy decoding.

This architecture is the direct ancestor of the transformer, which kept the encoder-decoder framing and replaced the recurrent computation with attention. another chapter develops the attention layer that sits on top of this. The bottleneck you should anticipate: a single fixed-size context vector cannot carry all the information from a 50-word input sentence. Attention solves it.

The widget below is a preview of the idea another chapter formalises, not a detour. Here is the one-sentence version: instead of forcing the decoder to read from one frozen context vector, attention lets it build a fresh context at every output step, $c_{t'} = \sum_t \alpha_{t',t}\, h_t$ — a weighted sum of all the encoder hidden states, where the weights $\alpha$ are a softmax over learned alignment scores between the current decoder state and each source position. The decoder, in effect, re-reads the source and focuses on the part it needs for the token it is about to emit. Watch the alignment weights light up as you step through a translation; that is the whole mechanism, and another chapter only adds the algebra.

A 1D convolution is to sequences what a 2D convolution is to images. A filter of size $k$ slides along the sequence axis and produces a new sequence. Same arithmetic for output shape: $L_{\text{out}} = (L_{\text{in}} + 2P - k)/S + 1$.

# Input: (batch, channels, length). Output: (batch, channels_out, length_out)
conv1d = nn.Conv1d(in_channels=64, out_channels=128, kernel_size=3, padding=1)
x = torch.randn(32, 64, 100)
out = conv1d(x)
print(out.shape)   # torch.Size([32, 128, 100])

For sequences, 1D convs have two important properties RNNs do not:

• Parallelism. Every timestep is computed independently in a single matmul. RNNs serialise across time; convs do not.
• Fixed receptive field. A stack of 1D convs has a finite, calculable receptive field. RNNs have a theoretically infinite (in practice limited) memory.

The trick that made 1D convs competitive on sequence modeling is dilated convolution. A dilated conv with dilation $d$ skips $d - 1$ inputs between each filter tap. Stack convs with exponentially increasing dilation (1, 2, 4, 8,..., 512) and the receptive field grows exponentially, while the parameter count grows linearly. WaveNet (van den Oord et al. 2016) used this to model raw audio with ~5000-sample receptive fields.

class DilatedBlock(nn.Module):
    def __init__(self, channels: int, kernel: int, dilation: int):
        super().__init__()
        self.conv = nn.Conv1d(channels, channels, kernel,
                              padding=(kernel - 1) * dilation, dilation=dilation)
    def forward(self, x):
        out = self.conv(x)
        # Causal: trim the right-pad so position t only depends on t-positions
        return out[..., :x.size(-1)]

That right-trim is the causal convolution trick. It makes the conv autoregressive (position $t$ does not see position $t + k$), which lets you use it as a language model. WaveNet was an autoregressive raw-audio model, and its causal-dilated-conv stack is the conceptual ancestor of the modern Mamba / S4 / state-space-model lineage.

Forecasting is the prototypical sequence task: given $x_1, \ldots, x_t$, predict $x_{t+1}, \ldots, x_{t+k}$. The toolbox is wider than people from a deep-learning background expect.

Three baselines you must implement before reaching for an RNN:

• Naive last-value. $\hat{x}_{t+1} = x_t$. Surprisingly hard to beat on noisy data.
• Seasonal naive. $\hat{x}_{t+1} = x_{t+1-s}$ where $s$ is the season length (7 for weekly, 24 for hourly, 12 for monthly). Strong on anything with periodicity.
• ARIMA / ETS. The classical statistical forecasting toolbox — ARIMA models the series as an autoregressive process (each value regressed on its own recent past, $P(x_t \mid x_{t-1}, \ldots, x_{t-\tau})$) plus a moving-average term; ETS fits exponential-smoothing trend and seasonality. You drive them through auto-tuning libraries (statsmodels, pmdarima) rather than deriving them here; the lab notebook walks through their main diagnostic, the autocorrelation function, from its definition. Often within a few percent of an RNN on standard benchmarks.

If those don't suffice, then a small RNN:

class Forecaster(nn.Module):
    def __init__(self, n_features: int, hidden: int = 64, horizon: int = 24):
        super().__init__()
        self.rnn = nn.GRU(n_features, hidden, batch_first=True)
        self.head = nn.Linear(hidden, horizon)
    def forward(self, x):
        # x: (B, L, n_features). Returns (B, horizon).
        _, h = self.rnn(x)
        return self.head(h.squeeze(0))

Train with MSE on the rolling-window targets. Standardise features per-series. Use teacher forcing only if you are doing multi-step decoder-style outputs; for direct multi-horizon output (as above) it does not apply.

Transformers have eaten most of what RNNs used to be good for. Honest cases where RNNs still make sense:

• Tiny models on edge devices. A 100k-parameter LSTM runs on a microcontroller. A transformer of similar capacity does not.
• Streaming inference with strict latency budgets. RNNs have $O(1)$ per-token memory and compute. Transformers without KV-cache are $O(L)$; with KV-cache they are still $O(L)$ memory.
• Very long sequences with structured dependencies. State-space models (Mamba, S4, S5) are the modern descendants of RNNs and beat transformers on sequences of length 10k+ in benchmarks like Long Range Arena. They are recurrent in a more carefully-engineered way.
• Pedagogy. Building an LSTM from scratch is the cleanest way to internalise the vanishing-gradient problem and the gate-based fix.

What you should not reach for an RNN for in 2026: general NLP, machine translation, dialog systems, language modeling at scale, document classification, or anything where a pre-trained transformer is available. The cost of pre-training has shifted the trade-off permanently.