GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow

2021137 citationsPreprintgreen Open Access

Authors

Dan Hendrycks

Steven Basart

Abstract

GPT-Neo is an implementation of model & data-parallel GPT-2 and GPT-3-like models, utilizing Mesh Tensorflow for distributed support. This codebase is designed for TPUs. It should also work on GPUs, though we do not recommend this hardware configuration.

Topics & Keywords

Software Engineering Research Software Testing and Debugging Techniques Machine Learning and Data Classification

UN Sustainable Development Goals

Quality Education

Publication Details

Published in: arXiv (Cornell University)

DOI: 10.48550/arxiv.2105.09938