# GPT-NeoX-20B 正式发布

- 来源：EleutherAI：Blog
- 发布时间：2022-02-03 00:00
- AIHOT 链接：https://aihot.virxact.com/items/cmnxbjhcw004jsln0xfwj229m
- 原文链接：https://blog.eleuther.ai/announcing-20b

## AI 摘要

GPT-NeoX-20B 正式发布。这款拥有 200 亿参数的大规模语言模型由开发团队与云计算服务商 CoreWeave 合作训练完成。该模型在参数量级上达到 200 亿规模，通过双方协作完成训练流程，为自然语言处理研究和应用提供了新的基础模型选择。

## 正文

As of February 9, 2022, GPT-NeoX-20B checkpoints are available for download from The Eye under Apache 2.0. More in-depth information on GPT-NeoX-20B can be found in the associated technical report on arXiv.

Looking for a demo? Try GPT-NeoX-20B via CoreWeave and Anlatan's inference service, GooseAI!

After a year-long odyssey through months of chip shortage-induced shipping delays, technical trials and tribulations, and aggressively boring debugging, we are happy to finally announce EleutherAI's latest open-source language model: GPT-NeoX-20B, a 20 billion parameter model trained using our GPT-NeoX framework on GPUs generously provided by our friends at CoreWeave.

GPT-NeoX-20B is, to our knowledge, the largest publicly accessible pretrained general-purpose autoregressive language model, and we expect it to perform well on many tasks.

We hope that the increased accessibility of models of this size will aid in research towards the safe use of AI systems, and encourage anyone interested in working in this direction to reach out to us.

As a thank you to our generous compute donors, we are delaying the public downloadable release of the model by 7 days. On February 9, 2022, the full model weights will be downloadable for free under a permissive Apache 2.0 license from The Eye.

There will be a #20b channel set up in our Discord for discussions of this model. Please note that much like our other language models and codebases, GPT-NeoX and GPT-NeoX-20B are very much research artifacts and we do not recommend deploying either in a production setting without careful consideration. In particular, we strongly encourage those looking to use GPT-NeoX-20B to read the paper and datasheet on our training data. There are still bugs to be ironed out and many inefficiencies that could be addressed---but hey, we do this in our free time, give us a break lol

Task Category Babbage Curie GPT-J-6B FairSeq-13B GPT-NeoX-20B DaVinci LAMBADA Sentence Completion 62.49% 69.51% 68.29% 70.95% 72.00% 75.16% ANLI R1 Natural Language Inference 32.40% 32.80% 32.40% 34.00% 34.00% 36.30% ANLI R2 Natural Language Inference 30.90% 33.50% 34.00% 33.00% 34.40% 37.00% ANLI R3 Natural Language Inference 33.75% 35.50% 35.50% 34.75% 35.40% 36.83% WSC Coreference Resolution 54.54% 49.54% 49.54% 55.44% 50.00% 59.18% WinoGrande Coreference Resolution 59.51% 64.56% 64.01% 67.40% 66.10% 69.93% HellaSwag Sentence Completion 40.38% 54.81% 36.53% 57.69% 53.50% 63.46% Average 44.85% 48.60% 45.75% 50.43% 49.34% 53.98%

Accuracy on standard language modeling tasks.

Subject Group Babbage Curie GPT-J-6B FairSeq-13B GPT-NeoX-20B DaVinci Humanities 27.01% 26.48% 28.07% 27.27% 28.70% 32.30% Social Science 27.94% 29.24% 28.73% 27.94% 30.80% 35.87% STEM 25.83% 24.25% 25.71% 24.63% 27.20% 28.60% Other 26.86% 28.84% 27.95% 27.33% 29.20% 36.85% Average 26.91% 27.20% 27.62% 26.79% 28.98% 33.41%

Zero-shot accuracy of factual knowledge by subject group, as measured by the HendrycksTest evaluation.