Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →nousresearch/distributed-llm-pretraining-torchtitan
Pretrain LLMs at scale with PyTorch 4D parallelism.
Passed 22 of 22 test cases with the skill, and 11 of 22 without it.
and 15 more in the full report
11 fixed11 passed either way0 still failing0 broke
Tokens, turns and time describe resource use, not a dollar price.
One of 11 test cases the skill fixed. Same model, same prompt, with and without the skill.
In which Python file relative to the TorchTitan repository root are Llama 3 run configs and model variants registered?
| Checked | Without skill | With skill |
|---|---|---|
| Specifies torchtitan/models/llama3/config_registry.py as the config registry location. | Missed | Passed |
The instructions your model reads when the skill loads.
TorchTitan is PyTorch's official platform for large-scale LLM pretraining with composable 4D parallelism (FSDP2, TP, PP, CP), achieving 65%+ speedups over baselines on H100 GPUs.
Installation:
# From PyPI (stable)
pip install torchtitan
# From source (latest features, requires PyTorch nightly)
git clone https://github.com/pytorch/torchtitan
cd torchtitan
pip install -r requirements.txt
Download tokenizer:
# Get HF token from https://huggingface.co/settings/tokens
python scripts/download_hf_assets.py --repo_id meta-llama/Llama-3.1-8B --assets tokenizer --hf_token=...
Start training on 8 GPUs:
# Configs are selected by name from the Python config registry
# (torchtitan/models/llama3/config_registry.py), not by TOML path
MODULE=llama3 CONFIG=llama3_8b ./run_train.sh
Copy this checklist:
Single Node Pretraining:
- [ ] Step 1: Download tokenizer
- [ ] Step 2: Configure training
test cases passed, vs 11 / 22 without
Published skills whose descriptions closely match this one, with their headline benchmark lift.
More skills from the same source and nearby in the registry, with their headline benchmark lift.