The transformer & GPT timeline

Noah

The transformer & GPT timeline

Timeline of Transformers, GPT, and ChatGPT. The timeline shows that GPT evolved from the original transformer architecture and gained its ability through many iterations.
2015. OpenAI was founded by Sam Altman, Elon Musk, Greg Brockman, Peter Thiel, and others. OpenAI develops many different AI models other than GPT.
2017. Google published the paper Attention is All You Need, which introduced the transformer architecture [Vaswani, Ashish, et al. “Attention is all you need.” Advances in neural information processing systems 30 (2017)]. The transformer is a neural network architecture that lays the foundation for many state-of-the-art (SOTA) large language models (LLM) like GPT.
2018. GPT is introduced in Improving Language Understanding by Generative Pre-training [Radford, Alec, et al. “Improving language understanding by generative pre-training.” (2018)]. It’s based on a modified transformer architecture and pre-trained on a large corpus.
2019. GPT-2 is introduced in Language Models are Unsupervised Multitask Learners [Radford, Alec, et al. “Language models are unsupervised multitask learners.” OpenAI blog 1.8 (2019): 9], which can perform a range of tasks without explicit supervision when training.
2020. GPT-3 is introduced in Language Models are Few-Shot Learners [Brown, Tom, et al. “Language models are few-shot learners.” Advances in neural information processing systems 33 (2020): 1877–1901], which can perform well with few examples in the prompt without fine-tuning.
2022. InstructGPT is introduced in Training language models to follow instructions with human feedback [Ouyang, Long, et al. “Training language models to follow instructions with human feedback.” arXiv preprint arXiv:2203.02155 (2022)], which can better follow user instructions by fine-tuning with human feedback.
2022. ChatGPT, a sibling of InstructGPT, is introduced in ChatGPT: Optimizing Language Models for Dialogue. It can interact with humans in conversations, thanks to the fine-tuning with human examples and reinforcement learning from human feedback (RLHF).