Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →openlair/distributed-llm-pretraining-torchtitan
Provides PyTorch-native distributed LLM pretraining using torchtitan with 4D parallelism (FSDP2, TP, PP, CP).
Passed 19 of 20 test cases with the skill, and 11 of 20 without it.
and 10 more in the full report
9 fixed12 passed either way1 still failing0 broke
Tokens, turns and time describe resource use, not a dollar price.
One of 8 test cases the skill fixed. Same model, same prompt, with and without the skill.
What is the standard path to the default Llama 3 8B training configuration file bundled within the repository structure?
| Checked | Without skill | With skill |
|---|---|---|
| points to ./torchtitan/models/llama3/train_configs/llama3_8b.toml | Missed | Passed |
The instructions your model reads when the skill loads.
TorchTitan is PyTorch's official platform for large-scale LLM pretraining with composable 4D parallelism (FSDP2, TP, PP, CP), achieving 65%+ speedups over baselines on H100 GPUs.
Installation:
# From PyPI (stable)
pip install torchtitan
# From source (latest features, requires PyTorch nightly)
git clone https://github.com/pytorch/torchtitan
cd torchtitan
pip install -r requirements.txt
Download tokenizer:
# Get HF token from https://huggingface.co/settings/tokens
python scripts/download_hf_assets.py --repo_id meta-llama/Llama-3.1-8B --assets tokenizer --hf_token=...
Start training on 8 GPUs:
CONFIG_FILE="./torchtitan/models/llama3/train_configs/llama3_8b.toml" ./run_train.sh
Copy this checklist:
Single Node Pretraining:
- [ ] Step 1: Download tokenizer
- [ ] Step 2: Configure training
- [ ] Step 3: Launch training
- [ ] Step 4: Monitor and checkpoint
test cases passed, vs 11 / 20 without
Published skills whose descriptions closely match this one, with their headline benchmark lift.
More skills from the same source and nearby in the registry, with their headline benchmark lift.