Recurrent Neural Network · RNN/LSTM
Recurrent Neural Network / Long Short-Term Memory
What is a recurrent neural network?
- Processes ordered data with memory
- LSTM improves long context
- Workhorse before Transformers
What the RNN really changed is that it does not treat the input as one block. Instead of pushing a fixed-size input through all at once, as in image classification, it applies the same set of weights at every time step and passes the hidden state on to the next one. This lets you handle data of varying lengths with a single model, so you no longer need separate models for 10-word and 50-word sentences. The trade-off is that each step can be computed only after the previous one finishes, so training time grows in proportion to sequence length.
The family splits into three main branches. The vanilla RNN is simple, but past roughly 10–20 time steps its gradients vanish and it effectively cannot use the early part of the sequence. The LSTM adds a cell state and three gates (input, forget, output), a structure that creates a separate channel for memory to flow through. The GRU cuts the gates to two, giving about 25% fewer parameters. In practice, with a few thousand records, a GRU often converges more stably than an LSTM.
The line between this and the Transformer is drawn by “length” and “resources.” Attention compares every time step with every other, so computation grows with the square of the length, while RNN-family models grow linearly with length. That is why, for sensor logs with tens of thousands of time steps or edge devices with only a few MB of memory, LSTM/GRU is still the realistic choice. Conversely, for tasks like translation and summarization, where the whole sentence must be weighed in both directions at once, sequential processing itself became the bottleneck.
The most common point of failure in practice is how the data is split. With time series, an analyst who splits training and validation by random shuffle out of habit lets future periods leak into training. The team gets sign-off on 95% validation accuracy, then finds in production that the model does worse than the existing rules. “It’s old tech, so just use a Transformer” is another common misconception. With only a few thousand labeled records of in-house data, a large model overfits first, and training costs multiply for nothing. The safer order is to set an LSTM baseline first and compare against it.
Sungil KimKAIST and Samsung Electronics alumnus and global pitching expert who coached winning teams at the UN Tourism and Select USA investor pitch competitions
Hyerang ParkContent storytelling and AI-driven brand content expertProposed in 1997 by Hochreiter and Schmidhuber, the LSTM addressed the RNN’s long-term memory problem.
Consider a hypothetical case. Company A, an auto-parts manufacturer with KRW 120 billion (≈ US$89,780,000) in annual revenue and 340 employees, had four press machines that stopped without warning an average of 3.2 times a month, halting the entire line. Vibration and current sensors logged data every second, but the rule was “alarm when a threshold is exceeded,” so more than 40 false alarms a day went off, and the floor worked with the alarm window switched off. The maintenance team validated LSTM failure prediction in a three-month proof of concept.
- Mr. Kim, a maintenance team manager, pulled 87 cases of the 30 minutes right before actual failures from the last two years of logs, and built the training set at a 1:5 ratio against normal periods.
- An outside advisory data scientist pointed out that “if you shuffle and split, the future teaches the past,” so the team split by time: 2023 for training, 2024 for validation.
- Feeding in 1,800 steps at 1-second intervals took three days per training run. Compressing them to 180 steps of 10-second averages cut training time to 4 hours.
- The head of production made it clear: “If false alarms exceed 5 a day, nobody will look.” The team raised the decision threshold from 0.5 to 0.8, prioritizing false-alarm suppression over recall.
- Three months later, unplanned stoppages fell from 3.2 to 1.1 per month and false alarms dropped from 40 to about 4 per day, recovering about 210 hours of annual downtime.