Meta announces a GPT3-size language model you can download

Meta announces a GPT3-size language model you can download

This is another unbelievable constituent!

[Submitted on 2 May 2022]

Authors:Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, Luke Zettlemoyer

Download PDF

Abstract: Large language models, which are often trained for hundreds of thousands of
compute days, have shown remarkable capabilities for zero- and few-shot
learning. Given their computational cost, these models are difficult to
replicate without significant capital. For the few that are available through
APIs, no access is granted to the full model weights, making them difficult to
study. We present Open Pre-trained Transformers (OPT), a suite of decoder-only
pre-trained transformers ranging from 125M to 175B parameters, which we aim to
fully and responsibly share with interested researchers. We show that OPT-175B
is comparable to GPT-3, while requiring only 1/7th the carbon footprint to
develop. We are also releasing our logbook detailing the infrastructure
challenges we faced, along with code for experimenting with all of the released

Submission history From: Susan Zhang [view email]

Mon, 2 May 2022 17:49:50 UTC (9,196 KB)

Read More
Share this on to discuss with people on this topicSign up on now if you’re not registered yet.

Related Articles

What’s recent in Emacs 28.1?

By Mickey Petersen It’s that time again: there’s a new major version of Emacs and, with it, a treasure trove of new features and changes.Notable features include the formal inclusion of native compilation, a technique that will greatly speed up your Emacs experience.A critical issue surrounding the use of ligatures also fixed; without it, you…

ToaruOS 2.0

ToaruOS has consumed the greater part of my life for the last eleven years. This release has been a long time coming: My first plans for a 64-bit, SMP-capable port of the OS date back to before the 1.0 release. The repository in which I built the new kernel was something I had already set…

A Python Guide for the Ages

Jure Šorn #Contents ToC = { ‘1. Collections’: [List, Dictionary, Set, Tuple, Range, Enumerate, Iterator, Generator], ‘2. Types’: [Type, String, Regular_Exp, Format, Numbers, Combinatorics, Datetime],…

Wikimedia voting on stopping accepting cryptocurrency donations

This is a subpage; for more information, see the Requests for comments page. The Wikimedia Foundation currently accepts cryptocurrency donations in currencies including Bitcoin, Bitcoin Cash, and Ethereum, as explained on the “Other ways to give” page. I propose that we stop accepting cryptocurrency donations. Accepting cryptocurrency signals endorsement of the cryptocurrency space by the…


Your email address will not be published. Required fields are marked *