distributed-llm-pretraining-torchtitan

Name: distributed-llm-pretraining-torchtitan
Rating: 5 (7 reviews)
Author: braxtonROSE4

Provides PyTorch-native distributed LLM pretraining using torchtitan with 4D parallelism (FSDP2, TP, PP, CP). Use when pretraining Llama 3.1, DeepSeek V3, or custom models at scale from 8 to 512+ GPUs with Float8, torch.compile, and distributed checkpointing.

7stars

Updated 2 months ago

View on GitHub ↗License: MIT

How to add

/plugin marketplace add braxtonROSE4/zorro-agent

The exact command may vary by repository. Check the README on GitHub.

For the skill author

Drop this on your repo README

Shows your skill is listed on Skillteca, generates a backlink and trackable traffic.

[![Listada na Skillteca](https://www.skillteca.com.br/api/badge/distributed-llm-pretraining-torchtitan-braxtonrose4/svg)](https://www.skillteca.com.br/skills/distributed-llm-pretraining-torchtitan-braxtonrose4?utm_source=badge&utm_medium=readme&utm_campaign=badge)

#llm #ai

Related skills

See all in Outros →

template-skill

143.8k

Replace with a description of the skill and when Claude should use it.

Outrosby anthropics

slack-gif-creator

143.8k

Knowledge and utilities for creating animated GIFs optimized for Slack. It provides constraints, validation tools, and animation concepts, useful when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack".

Outros#aiby anthropics

baoyu-compress-image

19.9k

Compresses images to WebP (default) or PNG with automatic tool selection. Use when the user asks to compress image, optimize image, convert to webp, or reduce image file size.

Outrosby JimLiu

zzz-one-dragon-player

6.4k

Zenless Zone Zero's all-in-one automatic game assistant, enabling AI Agents to fully automate daily game routines.

Outros#aiby OneDragon-Anything

Category alert

Get new Outros skills every Monday

One short email with only the new Outros skills. 4 minutes of reading, no spam, unsubscribe with one click.

You confirm your email on the first send. No spam. Unsubscribe with one click.

TorchTitan - PyTorch Native Distributed LLM Pretraining

Quick start

TorchTitan is PyTorch's official platform for large-scale LLM pretraining with composable 4D parallelism (FSDP2, TP, PP, CP), achieving 65%+ speedups over baselines on H100 GPUs.

Installation:

# From PyPI (stable)
pip install torchtitan

# From source (latest features, requires PyTorch nightly)
git clone https://github.com/pytorch/torchtitan
cd torchtitan
pip install -r requirements.txt