Recurrent-Pretraining

Visit Tool

recurrent-pretraining is an Open Source & Models tool that provides code for pretraining and inference of a large-scale depth-recurrent language model. It supports training on platforms like AMD GPUs and includes benchmarking tools.

Claim this tool

No Views Yet

At a glance

Pricing

Open Source

Free tier

Yes

API

Skill level

Technical

About

What is recurrent-pretraining?

recurrent-pretraining offers the complete code used to train a large-scale depth-recurrent language model, Huginn-0125, on 4096 AMD GPUs. This repository serves as a valuable reference for researchers and engineers interested in the exact methodologies and configurations employed for such a demanding task, especially within the constraints of AMD systems. It includes code for pretraining, inference, tokenizer generation, and data preparation. The project also provides detailed instructions for reproducing benchmark scores using the lm-eval harness and supports fast inference via vllm. The entire training dataset is available on Hugging Face, making it a comprehensive resource for those looking to understand or replicate advanced recurrent depth model training.

Best used for

Ideal for developers and data scientists who need to train large-scale depth-recurrent language models, reproduce benchmark scores for models like Huginn-0125, and understand the practicalities of deploying such models on AMD GPU clusters. Especially valuable for researchers and engineers working with advanced recurrent neural network architectures.

Common actions

train language models

perform model inference

benchmark AI models

develop recurrent networks

"AI Agents"github copilotdeepfakeopen-sourceworkflowscollaborationface swappingautomated workflowlow-code/no-code

Capabilities

Key features

Pretraining code
Inference code
Tokenizer generation
Data preparation scripts
lm-eval benchmarking
vllm fast inference
AMD GPU support

Target Audience

developerdata scientistprofessorstudent

Integrations

Not yet documented

Pricing & Plans

Open Source

Free

FAQs

What is the Huginn-0125 model?

Huginn-0125 is a large-scale depth-recurrent language model developed by the tomg-group-umd. The recurrent-pretraining repository provides all the code used to train and infer with this specific model, including its tokenizer and data preparation scripts.

Can I use this code to pretrain my own recurrent depth model?

While the repository provides the full code and methodology for training Huginn-0125, the authors note that it's primarily intended as a reference. Users are encouraged to adapt it for their own pretraining efforts, especially if working with AMD systems, but it's not presented as a plug-and-play solution for general model pretraining.

How can I reproduce the benchmark scores mentioned in the paper?

The repository includes instructions and commands for reproducing benchmark scores using the lm-eval harness. This covers standard benchmarks and specific tasks like GSM8k and Humaneval, with options for chat templates and system prompts.

Is there support for fast inference with this model?

Yes, the recurrent-pretraining repository supports fast inference through vllm. A dedicated vllm folder contains details and a plugin that can be installed into recent vllm v1 versions, allowing for optimized model execution.

Trending

Subcategories trending in Coding & Development

Code Assistants DevOps & Infrastructure No-Code / Low-Code Testing & QA Backend & APIs Prompt Engineering

Trending

Explore

Browse AI tools by category

Content & Design Productivity & Business Coding & Development AI Agents & Automation Research & Education Wellness & Lifestyle Career Development Marketing & Growth Data & Analytics Customer Support & CX Finance E-commerce