AI milestones in 2023
14 documented milestones from 2023.
GNoME predicts 2.2 million crystal structures, and the claim is disputed
Google DeepMind published GNoME in Nature on 29 November 2023, reporting 2.2 million predicted crystal structures of which 381,000 were judged stable, and describing this as an order of magnitude expansion in the stable materials known. A companion paper from Berkeley Lab reported a robotic laboratory synthesising 41 of the predicted compounds.
GraphCast forecasts ten days of global weather in under a minute
Google DeepMind published GraphCast in Science on 14 November 2023. Trained on about forty years of reanalysis data, it predicts hundreds of atmospheric variables ten days ahead at quarter-degree resolution, and beat the European Centre's operational forecast on 90 per cent of 1,380 verification targets. A forecast that takes hours on a supercomputer takes it under a minute.
AlphaMissense scores every possible missense variant in the human proteome
Google DeepMind published AlphaMissense in Science on 19 September 2023, an adaptation of AlphaFold that scores how likely a single amino acid change is to cause disease. It classified 89 per cent of all 71 million possible missense variants, against roughly 0.1 per cent confirmed by laboratory or clinical work.
3D Gaussian Splatting achieves real-time novel-view synthesis at 1080p
Bernhard Kerbl and colleagues applied 3D Gaussian splatting to novel-view synthesis with end-to-end optimisation and adaptive density control, achieving high-quality results at 1080p resolution and real-time frame rates, without the slow neural rendering that previous approaches required.
FlashAttention-2 roughly doubles attention speed on A100 GPUs
Tri Dao's FlashAttention-2 improved GPU work partitioning to deliver roughly 2x the speed of FlashAttention, reaching 50–73% of the theoretical maximum FLOPs/s in the configurations benchmarked in the paper, and up to 225 TFLOPs/s per A100 in the specific GPT-style training configurations reported.
RFdiffusion designs proteins that do not exist in nature
Watson, Juergens and colleagues at the Baker Lab published RFdiffusion in Nature on 11 July 2023. By fine-tuning the RoseTTAFold prediction network on denoising tasks, they turned a model that reads protein structures into one that invents them. Hundreds of the designs were then made in the laboratory.
Pangu-Weather is the first learned model to beat operational forecasting
Bi and colleagues at Huawei Cloud published Pangu-Weather in Nature on 5 July 2023. Trained on 39 years of recorded atmospheric data, it was the first machine-learned model reported to beat the operational numerical forecast on accuracy as well as speed, producing a global forecast in about 1.4 seconds on one graphics processor.
Process supervision outperforms outcome supervision for LLM reasoning
Researchers showed that giving a model step-by-step feedback on its reasoning, rather than judging only the final answer, raised the solve rate to 78% on a representative subset of the MATH benchmark, and released 800,000 human feedback labels to support further work.
Direct Preference Optimisation reduces need for RL in LM alignment
Rafael Rafailov and colleagues introduced Direct Preference Optimization (DPO), a method that aligns language models with human preferences using only a simple classification loss, bypassing the complex reinforcement learning pipeline that existing approaches required.
QLoRA enables finetuning of 65B-parameter models on a single 48GB GPU
Tim Dettmers and colleagues showed that a 65-billion-parameter language model could be finetuned on a single 48GB GPU without measurable quality loss on the benchmarks tested, by combining 4-bit quantisation with low-rank adapter training.
ESMFold predicts structure without alignments and folds 617 million proteins
Lin and colleagues at Meta published ESMFold in Science on 16 March 2023. It predicts a protein's structure from its sequence alone, with no search for related sequences, because the evolutionary information such a search supplies is already inside a language model of 15 billion parameters. They used that speed to fold over 617 million metagenomic proteins.
OpenAI releases GPT-4, the first of its models to accept images
OpenAI released GPT-4 on 14 March 2023, a model accepting both image and text input and producing text. Its technical report gives human-level performance on a range of professional and academic examinations, including a simulated bar exam in roughly the top ten per cent of test takers.
LLaMA matches leading models on public data with far fewer parameters
LLaMA, a collection of foundation language models from 7B to 65B parameters trained exclusively on publicly available data, was submitted to arXiv on 27 February 2023. The 13B model outperformed GPT-3 at 175B parameters on most benchmarks, and the weights were released to the research community.
ControlNet adds structured spatial conditioning to pretrained text-to-image diffusion models
Lvmin Zhang, Anyi Rao and Maneesh Agrawala introduced ControlNet, a neural network architecture that adds spatial conditioning controls such as edges, depth, segmentation and human pose to large pretrained text-to-image diffusion models without degrading their existing capabilities.