Pre-trained transformer language models have become a stepping stone towards artificial general intelligence (AGI), with some researchers reporting that AGI may evolve from our current language model technology. While these models are trained on increasingly larger datasets, the documentation of basic metrics including dataset size, dataset token count, and specific details of content is lacking. Notwithstanding proposed standards for documentation of dataset composition and collection, nearly all major research labs have fallen behind in disclosing details of datasets used in model training. The research synthesized here covers the period from 2018 to early 2022, and represents a comprehensive view of all datasets—including major components Wikipedia and Common Crawl—of selected language models from GPT-1 to Gopher.
2022 has brought about an explosion in the world of artificial intelligence. Some have described the current AI trajectory as a shift more profound than the discovery of fire, or the birth of the internet. Google's Transformer architecture—introduced just a few years ago—has permanently changed the landscape, giving rise to hundreds of new AI models around the world, from Abu Dhabi to Switzerland.
The capabilities of these models are only limited by human imagination, as they generate new code, conceptualize images from scratch, write books, control robots, and much more. Discover the latest models and frameworks beyond GPT-3, from Meta NLLB to Google Pathways.
See popular AI applications and use cases. And understand what all of this means for the future of artificial general intelligence, and you.
Artificial Intelligence: You Are Here
Artificial Intelligence: You Are Here
Artificial Intelligence: You Are Here