Andrew M. Dai and Quoc V. Le Introduced Semi-Supervised Sequence Learning
In November 2015, Andrew M. Dai and Quoc V. Le at Google Brain published 'Semi-Supervised Sequence Learning', showing that pre-training recurrent neural networks with unsupervised objectives, language modelling or sequence autoencoding, before supervised fine-tuning improved text classification accuracy and training stability, anticipating the pre-train-then-fine-tune paradigm later adopted widely in NLP.
Background
By 2015, recurrent neural networks, and in particular the long short-term memory architecture (LSTM, a type of recurrent network designed to remember information across long sequences without the signal degrading), had become the standard tool for text classification tasks. Given enough labelled training examples, an LSTM trained in the ordinary supervised way could do well. The problem was that labelled data was expensive to produce. Someone had to read each example and assign it a category, and for many tasks there simply was not enough of that kind of annotated text to train a deep network reliably.
Unlabelled text, on the other hand, was everywhere. Web pages, books, forum posts: the supply was effectively unlimited. Researchers knew this and had experimented with unsupervised pre-training in other contexts. Geoffrey Hinton and colleagues had used layerwise pre-training to help deep networks in the mid-2000s, and there was a broader tradition of pre-training in computer vision. But for sequential text models, no one had demonstrated a clean, general method for using that raw unlabelled data to give a network a head start before supervised training began.
What happened
Andrew M. Dai and Quoc V. Le at Google Brain published “Semi-Supervised Sequence Learning” in November 2015, first appearing on arXiv on 4 November. The paper tested two ways of pre-training an LSTM on unlabelled text before fine-tuning it on a labelled classification task.
The first method was straightforward language modelling: the network learned to predict the next word in a sequence, step by step, across a large corpus of unlabelled text. The second was a sequence autoencoder. The autoencoder reads an entire input sequence and compresses it into a single fixed-size vector, then tries to reconstruct the original sequence from that vector alone. Both objectives forced the network to develop a general sense of how language fits together, before it ever saw a labelled example. After pre-training by either method, the same network was then fine-tuned on standard supervised text classification benchmarks.
The results were consistent across several tasks. On the IMDB sentiment classification benchmark, pre-training with the sequence autoencoder pushed test error down to 7.64%, compared to 8.93% for the baseline LSTM trained from scratch without pre-training. Pre-training also made training more stable: networks that had been initialised this way were less sensitive to the choice of hyperparameters. Dai and Le also found that the gains were larger when the labelled training set was small, which made sense. A network that had already learned the structure of language from millions of unlabelled sentences needed fewer labelled examples to adapt to a specific task.
The paper did not claim a finished product or a general-purpose system. It was a demonstration that the pre-train-then-fine-tune pattern worked for recurrent sequence models on text. At the time, that was not obvious, and the empirical results gave the research community concrete numbers to argue from.
Why it mattered
The paper provided empirical evidence that unsupervised pre-training of recurrent networks on large unlabelled corpora could substantially improve downstream supervised performance, reducing the labelled data needed for competitive text classification. It prefigured the transfer-learning methodology that became central to NLP with ELMo, GPT, and BERT. The sequence autoencoder objective also demonstrated that encoder–decoder architectures could serve as general-purpose feature extractors for sequential data.
People
Andrew M Dai, Quoc V Le
Organisations
Sources
- Semi-Supervised Sequence Learning.arXiv.Primary source
- Semi-Supervised Sequence Learning, NeurIPS 2015 Proceedings.Neural Information Processing Systems Foundation.Official
- Semi-Supervised Sequence Learning, Google Research Publications.Google Research.Official
Cite this page
AI Achievements. (2015). Andrew M. Dai and Quoc V. Le Introduced Semi-Supervised Sequence Learning. Retrieved 2026-08-22, from https://achievements.ai/milestone/andrew-semi-supervised-sequence-learning
@misc{achievements_andrew_semi_supervised_sequence_learning,
title = {Andrew M. Dai and Quoc V. Le Introduced Semi-Supervised Sequence Learning},
author = {{AI Achievements}},
year = {2015},
url = {https://achievements.ai/milestone/andrew-semi-supervised-sequence-learning}
}