Recurrent networks and LSTM
3 milestones used this technique.
Andrew M. Dai and Quoc V. Le Introduced Semi-Supervised Sequence Learning
In November 2015, Andrew M. Dai and Quoc V. Le at Google Brain published 'Semi-Supervised Sequence Learning', showing that pre-training recurrent neural networks with unsupervised objectives, language modelling or sequence autoencoding, before supervised fine-tuning improved text classification accuracy and training stability, anticipating the pre-train-then-fine-tune paradigm later adopted widely in NLP.
Neural Turing Machine Introduced by Alex Graves, Greg Wayne, and Ivo Danihelka
In October 2014, Alex Graves, Greg Wayne, and Ivo Danihelka at Google DeepMind published 'Neural Turing Machines', a preprint proposing a neural network architecture augmented with an external memory matrix and differentiable read/write operations, enabling the system to learn algorithms such as sorting and copying from examples alone.
Long Short-Term Memory Introduced by Sepp Hochreiter and Jürgen Schmidhuber
In 1997, Sepp Hochreiter at Technische Universität München and Jürgen Schmidhuber at IDSIA published 'Long Short-Term Memory' in Neural Computation, introducing a recurrent neural network architecture with gated memory cells that could learn dependencies across long sequences without suffering from the vanishing gradient problem.