Vision Model Training on Hugging Face Jobs

Vision Model Training on Hugging Face Jobs

Train object detection, image classification, and SAM/SAM2 segmentation models on managed cloud GPUs. No local GPU setup required — results are automatically saved to the Hugging Face Hub. For text/language model fine-tuning (SFT/DPO/GRPO via TRL), use this pack's huggingface-llm-trainer skill instead.

When to Use

Fine-tuning object detection models (D-FINE, RT-DETR v2, DETR, YOLOS), image classification models (any timm/ model or Transformers classifier), or SAM/SAM2 segmentation models (bbox or point prompts) on custom datasets — locally or on Hugging Face Jobs.

Local Script Execution

Helper scripts use PEP 723 inline dependencies:

uv run scripts/dataset_inspector.py --dataset username/dataset-name --split train
uv run scripts/estimate_cost.py --help

Prerequisites Checklist

• Hugging Face account with Pro/Team/Enterprise plan (Jobs require a paid plan). Authenticated login (hf auth whoami), token with write permissions passed in job secrets.
• Object detection: dataset on the Hub with an objects column (bbox, category, optional area). Bboxes in xywh (COCO) or xyxy (Pascal VOC) — auto-detected/converted. Categories can be integers or strings (auto-remapped). image_id optional, auto-generated.
• Image classification: an image column (PIL images) and a label column (integer or string class IDs, ClassLabel or plain — auto-remapped). Common alt names (labels, class, fine_label) auto-detected.
• SAM/SAM2 segmentation: an image column, a mask column (binary ground-truth mask), and a prompt — either a prompt column with JSON ({"bbox": [...]} or {"point": [...]}), or dedicated bbox/point columns (xyxy, absolute pixels). Example dataset: merve/MicroMat-mini.
• Always validate unknown datasets first (see Dataset Validation below).
• Timeout must exceed expected training time — default 30min is too short, use 2-4h minimum for vision training.
• Hub push enabled: push_to_hub=True, hub_model_id="username/model-name", token in secrets.

Dataset Validation

Validate BEFORE launching GPU training — the #1 cause of training failures is format mismatches. Skip only for well-known defaults (e.g. cppe-5). Run via Jobs (avoids local SSL/dependency issues), locally with uv run scripts/dataset_inspector.py --dataset ... --split train, or via HfApi().run_uv_job(script="scripts/dataset_inspector.py", script_args=[...], flavor="cpu-basic", timeout=300). Output markers: ✓ READY or ✗ NEEDS FORMATTING (with mapping code).

The object detection training script auto-handles bbox format detection/conversion, sanitization, image_id generation, and category remapping — no manual preprocessing needed beyond having objects.bbox/objects.category.

Training Workflow

1. Verify prerequisites (account, token, dataset).
2. Validate dataset format with the inspector, before spending GPU time.
3. Ask the user about dataset size (quick 10% test vs. full) and whether to create a validation split, and which GPU hardware to use — present as explicit options rather than assuming.
4. Prepare the training script: scripts/object_detection_training.py (OD), scripts/image_classification_training.py (IC), or scripts/sam_segmentation_training.py (SAM). All use HfArgumentParser — configure via CLI-style script_args, not by editing Python variables. See references/timm_trainer.md for timm details and references/finetune_sam2_trainer.md for SAM2 details.
5. Save the script to submitted_jobs/<dataset>_<timestamp>.py, submit the job, and report the job ID, monitoring URL, Trackio dashboard (https://huggingface.co/spaces/{username}/trackio), expected time, and estimated cost. Wait for the user to request status checks — don't poll; jobs are asynchronous and can take hours.