Module 9: Advanced Deep LearningBy Ashok Kumar Yadav4 min read

RNN & LSTM Networks: Sequence Modeling and Time Series

Text, sensor streams and prices are ordered data: what came before matters. RNNs carry a hidden state through time, and LSTMs add gates that decide what to remember.

Unrolled recurrent neural network and LSTM cell with input, forget and output gates

Quick answer: An RNN reuses the same weights at every time step and passes a hidden state forward. An LSTM adds input, forget and output gates plus a cell state, which lets it learn long-range dependencies that plain RNNs lose to vanishing gradients.

RNN, LSTM and GRU compared

A plain RNN updates its hidden state as h_t = tanh(W x_t + U h_(t-1)). Gradients are multiplied through every step, so they shrink or explode over long sequences. LSTM adds a cell state with gates to control what is written, kept or exposed. GRU merges some gates and has fewer parameters.

Model	Gates	Strength
RNN	None	Simple, short sequences
LSTM	Input, forget, output	Long dependencies
GRU	Update, reset	Faster, fewer parameters

Prepare sequences for a forecaster

Slide a window over the series: the previous 30 values are the input and the next value is the target. Scale using statistics from the training portion only to avoid leakage.

Create sliding windows without data leakage

import numpy as np, torch

def make_windows(series, window=30):
    X, y = [], []
    for i in range(len(series) - window):
        X.append(series[i:i+window])
        y.append(series[i+window])
    X = torch.tensor(np.array(X), dtype=torch.float32).unsqueeze(-1)
    y = torch.tensor(np.array(y), dtype=torch.float32).unsqueeze(-1)
    return X, y

t = np.arange(0, 400, 0.1)
series = np.sin(t) + 0.1 * np.random.randn(len(t))
split = int(len(series) * 0.8)
mu, sd = series[:split].mean(), series[:split].std()
scaled = (series - mu) / sd
Xtr, ytr = make_windows(scaled[:split])
Xte, yte = make_windows(scaled[split:])
print(Xtr.shape, Xte.shape)

Build and train an LSTM in PyTorch

The model reads the window and predicts the next value from the last hidden state.

Training loop with gradient clipping

import torch.nn as nn

class Forecaster(nn.Module):
    def __init__(self, hidden=64, layers=2):
        super().__init__()
        self.lstm = nn.LSTM(1, hidden, layers, batch_first=True, dropout=0.1)
        self.head = nn.Linear(hidden, 1)
    def forward(self, x):
        out, _ = self.lstm(x)
        return self.head(out[:, -1])

model = Forecaster()
opt = torch.optim.Adam(model.parameters(), lr=1e-3)
loss_fn = nn.MSELoss()
for epoch in range(15):
    model.train()
    perm = torch.randperm(len(Xtr))
    for i in range(0, len(perm), 64):
        idx = perm[i:i+64]
        opt.zero_grad()
        loss = loss_fn(model(Xtr[idx]), ytr[idx])
        loss.backward()
        nn.utils.clip_grad_norm_(model.parameters(), 1.0)
        opt.step()
    model.eval()
    with torch.no_grad():
        val = loss_fn(model(Xte), yte).item()
    print(f"epoch {epoch+1:2d}  val MSE {val:.4f}")

Evaluate honestly

Compare the LSTM against a naive baseline that predicts the previous value. If the LSTM does not beat it on held-out data, the extra complexity is not helping. Use walk-forward validation for real time series so each fold only trains on the past.

Practical tipGradient clipping (shown above) stops exploding gradients, which are common in recurrent models.

Baseline to beat

naive = Xte[:, -1]                       # predict last observed value
print("naive MSE:", nn.functional.mse_loss(naive, yte).item())

Hands-on practice

Run the examples in the downloadable Jupyter notebook. The extra checks below focus on mistakes that appear in real projects, not just syntax.

Build sliding windows without leakage

Practice example

import numpy as np
s=np.arange(10)
X=np.array([s[i:i+3] for i in range(len(s)-3)])
y=np.array([s[i+3] for i in range(len(s)-3)])
print(X[:2]); print(y[:2])

[[0 1 2]
 [1 2 3]]
[3 4]

Create train/validation splits by time before fitting scalers.

Inspect LSTM output shapes

Practice example

import torch, torch.nn as nn
lstm=nn.LSTM(input_size=1,hidden_size=8,batch_first=True)
out,(h,c)=lstm(torch.randn(4,12,1))
print(out.shape,h.shape)

torch.Size([4, 12, 8]) torch.Size([1, 4, 8])

Shape checks catch many sequence-model bugs early.

Self-check

Can you explain why the first practice result is trustworthy rather than just plausible?
What would you monitor or test before using this RNN & LSTM workflow in production?
Which assumption in the example would break first on a larger or messier dataset?

Frequently asked questions

Why does LSTM solve the vanishing gradient problem?

The cell state is updated additively and controlled by gates, so gradients can flow across many steps without shrinking repeatedly.

LSTM or Transformer for time series?

LSTMs are a solid choice for small and medium datasets. Transformers can win on large datasets and long contexts, but always compare both against a simple baseline.

What is the difference between LSTM and GRU?

GRU uses two gates and no separate cell state, so it is lighter and often trains faster with similar accuracy.

Primary references

Use these official sources when exact syntax, defaults or definitions matter beyond the examples here.

Written by Ashok Kumar Yadav
Ashok Kumar Yadav writes Xenors tutorials on applied AI, programming and data workflows. Every lesson is built around code you can run and check yourself.
GitHub · LinkedIn