RedPajama

Introducing the RedPajama-INCITE family of models by Together! The latest release presents a groundbreaking step in AI model availability, with a focus on base, instruction-tuned, and chat AI models. At the core of this release are the 3 billion (3B) and 7 billion (7B) parameter models, designed to closely replicate the LLaMA recipe for high-performance AI. The models have been meticulously trained on the expansive 5-terabyte RedPajama base dataset, showcasing superior capabilities and compatibility with a wide range of hardware including GPUs as old as the RTX 2070. Ideal for few-shot learning and downstream applications such as entity extraction, classification, and summarization, these models set new benchmarks in instruction tuning on the HELM benchmarking scale, not only outperforming peers but also introducing new avenues for AI research and real-world applications. Users will be particularly interested in the 7B model's ongoing training progress, already showcasing a competitive edge over similar models. All models are released under the Apache 2.0 license, promoting research and commercial uses and advancing open collaborations in AI.

Key Features

• Open-Source AI Models: The project offers a suite of leading open-source models including base, instruction-tuned, and chat variants.

• Enhanced Performance: Models have been trained on a comprehensive 5TB dataset, resulting in high-performing AI capabilities.

• Wide Compatibility: Engineered to operate on various hardware, including older models like the RTX 2070.

• Instruction-Tuned Advances: Models demonstrate strong results on benchmarks, ideal for few-shot learning and entity extraction.

• Community-Driven Development: The RedPajama project encourages community contributions and feedback for continuous improvement.

FAQ

What is RedPajama-INCITE?

RedPajama-INCITE is a range of AI models developed by Together, including base, instruction-tuned, and chat models, with a focus on open-source distribution and high-performance, replicating the LLaMA recipe.

What are the advantages of the RedPajama-INCITE models?

The models outperform other similar-sized open models on benchmarks, are ideal for few-shot prompts, and run efficiently even on hardware like the RTX 2070 released over five years ago.

Under what license are the RedPajama-INCITE models released?

These models are released under the Apache 2.0 license, which allows for free usage in both research and commercial applications.

Is the RedPajama-INCITE 7B model still improving?

Yes, the 7B model is partway through its training on the RedPajama base dataset with a roadmap to complete 1 trillion token training, showing continuous quality improvement.

Are the RedPajama-INCITE models designed to work on older-generation hardware?

The project values broad hardware compatibility and aims to be accessible to a wide array of users, as demonstrated by even the 3B model's operability on an RTX 2070.