HF Jobs — training mechanics + cost model

reference · tech-lead · updated 2026-06-27 · source

Research for effort #943 (sub-issue #944) — SPIKE evaluating Hugging Face as the training backend for a multi-tenant model platform. This doc covers the training mechanics (HF Jobs, TRL methods, hardware/flavors, monitoring) and the cost model for our two targets: the internal builder model and per-tenant LoRA adapters. Sibling docs: multi-tenant isolation · serving & integration · datasets & trace pipeline.

Sources: local skills huggingface-llm-trainer, trl-training, hf-mem, huggingface-trackio, huggingface-best (.agents/skills/); live hf jobs hardware (read 2026-06-27, logged in as josetedos / org tuempresadigital); HF docs and pricing (cited inline).

---

1. HF Jobs — what it is and how training runs

HF Jobs runs a containerized compute task on Hugging Face's managed infrastructure — you define a command + Docker image (or a UV script) + a hardware flavor, and HF schedules it on a cloud GPU. No local GPU, no infra to manage. CLI surface: hf jobs {run,uv,scheduled,ps,logs,inspect,cancel,wait,ssh,stats} (HF Jobs docs).

Execution model — the four load-bearing facts

Fact	Detail	Why it matters for us
Asynchronous	run/uv submits and returns immediately; the job runs in the background for minutes–hours. Poll with hf jobs logs/inspect/ps; logs lag ~30–60s.	Our orchestrator fires a training job and checks back later — never blocks.
Ephemeral environment	The container's filesystem is destroyed when the job ends.	The trained weights/adapter MUST be pushed to the Hub or all training is lost. This is the #1 footgun.
Mandatory Hub push	Config: push_to_hub=True, hub_model_id="org/model"; job: secrets={"HF_TOKEN": "$HF_TOKEN"} (write token). Optional hub_strategy="every_save" to push checkpoints.	The Hub model repo is the durable training artifact — see serving & integration for how it's consumed.
Timeout	Default 30 min — too short for real training. Format "90m"/"2h"/1.5h/seconds. Reaching timeout kills the job and loses unsaved progress. Add 20–30% buffer.	Every job we submit sets an explicit, padded timeout.

Billing stops the instant a job ends/cancels/fails (auto-suspends on repeated failure), so a killed-on-timeout job still costs the wasted GPU-minutes — set timeouts deliberately.

How a training job is actually submitted

The recommended path is a UV script with PEP 723 inline dependencies — a self-contained Python file whose deps are declared in a header comment, passed inline (no local file; jobs run in isolated containers with no access to your filesystem — local paths fail). Sketch:

hf jobs uv run --flavor a10g-large --timeout 2h --secrets HF_TOKEN <script-url>
# or inline via the hf_jobs("uv", {...}) MCP tool with the script as a string

The PEP 723 header (# /// script … # dependencies = ["trl>=0.12.0", "peft>=0.7.0", "trackio"] # ///) makes the job reproducible and dependency-pinned. Scripts can be inline strings, or public URLs (Hub resolve/main/..., GitHub raw, gist). CLI syntax gotcha: flags go before the script URL; it's hf jobs uv run (not run uv); the flag is --secrets (plural).

Other submission paths:

• TRL maintained scripts — official, battle-tested per-method scripts run straight from a URL with --script_args (no code to write).
• trl-jobs package — uvx trl-jobs sft --model_name ... --dataset_name ...: one-liner with optimized defaults, auto-Trackio, auto-Hub-push. Good for terminal use.
• hf jobs run <image> <cmd> — generic container job (e.g. GGUF conversion, dataset prep).

Scheduled jobs — hf jobs scheduled

hf jobs scheduled run "<cron>" <image> <cmd> (and scheduled uv for UV scripts) registers a recurring job on a cron expression ("0 0 * * *" = nightly), with list/inspect/suspend/resume/delete/labels. This is the native primitive for "retrain every tenant's adapter weekly" — no external scheduler needed; HF runs it. Pair with labels (env=prod, tenant=<id>) for attribution.

---

2. TRL training methods + LoRA/PEFT + Unsloth

TRL (Transformer Reinforcement Learning, built on Transformers + Accelerate) is the trainer library HF Jobs uses. Methods (references/training_methods.md, trl-training skill):

Method	One-liner	Data needed	When to use
SFT (Supervised Fine-Tuning)	Standard instruction/demonstration tuning	demonstrations (messages / text / prompt-completion)	The default and starting point — teach a task/domain. This is what the builder model + tenant adapters need.
DPO (Direct Preference Optimization)	Align to preferences from chosen/rejected pairs, no reward model	preference pairs (prompt/chosen/rejected)	Post-SFT quality alignment when you have paired preferences.
GRPO (Group Relative Policy Optimization)	Online RL, optimizes vs. group performance on verifiable rewards	prompts + a reward function	Tasks with automatic reward signals (code runs, math checks).
Reward Modeling (trl reward)	Train a model that scores response quality	preference pairs	Build the reward component of an RLHF pipeline.
KTO	Alignment from unpaired binary good/bad labels (no pairs)	per-sample 👍/👎	When you have thumbs-up/down signal but not matched pairs.
(also RLOO, PPO)	Online RL variants	prompts + reward	Advanced RL; out of scope for v1.

Recommended pipeline for us: SFT first (and likely only SFT for v1). DPO/GRPO/Reward are later refinements once we have preference/outcome signal from the effort→PR trace (merged-clean = positive, reverted = negative — see datasets & trace pipeline).

LoRA/PEFT vs full fine-tune — the decision that drives cost

• Full fine-tune updates all weights → max VRAM, produces a full model copy per run. Only needed for the largest-impact base-model changes.
• LoRA/PEFT trains small low-rank adapter matrices (e.g. LoraConfig(r=16, lora_alpha=32)), freezing the base. ~Fraction of the VRAM, far cheaper, and the artifact is a tiny adapter (MBs, not GBs) that loads on top of a shared base. The skill mandates LoRA for models >7B. LoRA is the architecture for per-tenant adapters — one frozen base, N cheap adapters; see multi-tenant isolation.

Unsloth — when

Drop-in FastLanguageModel/FastVisionModel over TRL: ~2× faster, ~60% less VRAM. Use it when: training >13B, VRAM-constrained, speed matters, or VLMs. It lets a bigger model fit a smaller (cheaper) flavor — directly lowers our builder-model cost if it grows past 7B.

---

3. Hardware flavors + real pricing (live hf jobs hardware, 2026-06-27)

Billed per minute, only while Starting/Running (no charge during build); auto-suspend on failure (Jobs pricing). Exposed ports add a flat $0.01/hr. Full live list below — these are the authoritative prices (the skill's bundled estimate_cost.py carries a stale, inflated cost table, e.g. it lists a10g-large at $5/hr vs the real $1.50/hr — use these numbers, not the script's).

GPU flavors → $/hr → model-size fit

Flavor	GPU	GPU mem	$/hr	$/min	Fits (LoRA SFT)
t4-small	1× T4	16 GB	$0.40	$0.0067	<1B demos / tiny adapters
t4-medium	1× T4	16 GB	$0.60	$0.0100	1–3B small jobs
l4x1	1× L4	24 GB	$0.80	$0.0133	1–7B LoRA — best $/perf for tenant adapters
a10g-small	1× A10G	24 GB	$1.00	$0.0167	1–7B dev
a10g-large	1× A10G	24 GB	$1.50	$0.0250	3–13B LoRA (workhorse)
l40sx1	1× L40S	48 GB	$1.80	$0.0300	7–13B, longer context
a100-large	1× A100	80 GB	$2.50	$0.0417	7–34B LoRA / builder model
rtx-pro-6000	1× RTX PRO 6000	96 GB	$2.75	$0.0458	13–34B
a10g-largex2	2× A10G	48 GB	$3.00	$0.0500	13B multi-GPU
l4x4	4× L4	96 GB	$3.80	$0.0633	multi-GPU mid
a10g-largex4	4× A10G	96 GB	$5.00	$0.0833	13B+
h200	1× H200	141 GB	$5.00	$0.0833	30–70B
l40sx4	4× L40S	192 GB	$8.30	$0.1383	70B
a100x4	4× A100	320 GB	$10.00	$0.1667	70B full / 100B+ LoRA
h200x2	2× H200	282 GB	$10.00	$0.1667	70B+
rtx-pro-6000x4	4×	384 GB	$11.00	$0.1833	large
a100x8 / h200x4	8×A100 / 4×H200	640 / 564 GB	$20.00	$0.3333	very large / multi-node
rtx-pro-6000x8	8×	768 GB	$22.00	$0.3667	very large
l40sx8	8× L40S	384 GB	$23.50	$0.3917	very large
h200x8	8× H200	1128 GB	$40.00	$0.6667	frontier

CPU flavors (dataset prep, dataset validation, GGUF): cpu-basic $0.01/hr, cpu-upgrade $0.03, cpu-xl $1.00, cpu-performance $1.90. Dataset validation on CPU costs ~$0.01 — always validate format before a GPU run (50%+ of failures are format mismatches).

Use hf-mem (uvx hf-mem --model-id <id> --experimental --json-output) to estimate a model's VRAM (weights + KV cache) from the Hub without downloading, to pick the right flavor.

Paid-plan requirement

Jobs require a positive credit balance, which in practice means a paid plan — the huggingface-llm-trainer skill states Jobs need Pro / Team / Enterprise (Jobs pricing; HF pricing):

Plan	Price (2026-06)	Notes
Free	$0	No Jobs (no positive balance / credits).
PRO	$9 / month	Personal; unlocks Jobs pay-per-minute, ZeroGPU quota, 1TB private storage.
Team	$20 / user / mo	Org-level; bill Jobs to the org namespace (--namespace org).
Enterprise	$50 / user / mo	+ Resource Groups for per-group cost attribution (--namespace <rg-id>).

The plan fee is a subscription floor; GPU time is pay-as-you-go per minute on top. For a multi-tenant org, Team (org namespace + per-job labels for tenant attribution) is the likely fit. Sources: huggingface.co/pricing, docs.huggingface.co/.../jobs-pricing.

---

4. Cost model for our two targets

Estimation method. The skill's estimate_cost.py heuristic: hours ≈ 0.1 × model_B × (dataset_examples/1000) × epochs × hw_multiplier (0.1h = base for a 1B model per 1k examples on a10g-large; multipliers ≈ a100-large 0.7, a10g-large 1.0, l4x1 1.2, a10g-small 1.3, t4-small 2.0). cost = hours × $/hr using the live prices in §3. These are rough (the script itself says "approximations; actual times vary widely") — treat as order-of-magnitude, validate with a real first run, and add a 30% timeout buffer.

Assumptions (stated explicitly)

• SFT + LoRA for both targets (cheapest, adapter artifacts).
• Builder model: 7B base, trained on the effort→PR trace corpus, ~3,000 examples, 3 epochs (corpus is small early — efforts + sub-issues number in the low thousands; see datasets & trace pipeline).
• Tenant adapter: small base (1–3B), small per-tenant corpus (~500–1,000 examples, 3 epochs).
• "Weekly" = 4.3 runs/month. Prices = live §3. Excludes the plan subscription floor and Hub storage.

(a) Builder model — one larger model, periodic retrain

Config	Flavor	Est. time	Est. cost / run
7B LoRA, 3k ex, 3 ep	a100-large ($2.50)	~4.4 h	~$11
7B LoRA, 3k ex, 3 ep	a10g-large ($1.50)	~6.3 h	~$9.5
7B LoRA, 1k ex, 3 ep (early/small corpus)	a100-large	~1.5 h	~$4
13B LoRA (if it grows), 3k ex, 3 ep	a100-large	~8 h	~$20

Monthly: retrain weekly → ~$45–50/mo; monthly → ~$10/mo. Unsloth (~2× faster) roughly halves these. Builder cost is small and bounded — it's one model.

(b) Per-tenant LoRA adapters — many small, one per client

Config	Flavor	Est. time	Est. cost / run
1B LoRA, 500 ex, 3 ep	t4-small ($0.40)	~0.3 h	~$0.12
3B LoRA, 1k ex, 3 ep	l4x1 ($0.80)	~1.1 h	~$0.86
3B LoRA, 1k ex, 3 ep	t4-small ($0.40)	~1.8 h	~$0.72
3B LoRA, 1k ex, 3 ep	a10g-small ($1.00)	~1.2 h	~$1.17

Headline: a per-tenant adapter retrain is ~$0.10–$1.20. Pick l4x1/t4-small for best $/run.

Monthly fleet scenarios (weekly retrain, ~$1/run mid-case; ~$0.30/run small-case):

Tenants	Weekly @ ~$1/run	Weekly @ ~$0.30/run	Monthly retrain @ ~$1/run
10	~$43/mo	~$13/mo	~$10/mo
50	~$215/mo	~$65/mo	~$50/mo
100	~$430/mo	~$129/mo	~$100/mo

Combined rough monthly (100 tenants weekly + builder weekly + Team plan): ~$430 (tenants) + ~$48 (builder) + ~$20/user (plan) ≈ ~$500/mo order-of-magnitude — before volume discounts, Unsloth savings, or moving cadence to monthly. Cost scales linearly with tenant count × cadence, so cadence (weekly vs monthly vs event-triggered) and flavor choice are the two cost knobs. The dominant marginal cost is the tenant fleet, and each tenant run is cheap; the model is economically viable at the per-tenant LoRA granularity.

---

5. Monitoring — Trackio

Trackio is HF's experiment-tracking library; every training script should include it (huggingface-trackio skill). Add trackio to deps and set the trainer's report_to="trackio"

• a meaningful project/run_name. Three interfaces:

• Python logging — trackio.init(project=..., space_id="org/trackio") → trackio.log({...}) → trackio.finish(). Passing space_id syncs metrics to a HF Space dashboard so they persist after the ephemeral instance dies — essential given §1's ephemeral env.
• Alerts — trackio.alert(title, text, level=INFO|WARN|ERROR): printed to logs, stored, shown on the dashboard, optional Slack/Discord webhook. The primitive for autonomous iteration — an orchestrator inserts alerts for loss divergence / NaN / stalls and polls them.
• CLI retrieval — trackio list projects/runs/alerts --json, trackio get metric ... --json: programmatic readout for our orchestrator to gate "did this run succeed?" by construction.

A single org/trackio Space gives one dashboard across all tenant + builder runs (group by project/run_name/labels). This is our training observability surface.

---

Implications for the ADR (#948)

1. HF Jobs fits the multi-tenant retraining shape. Async submit + hf jobs scheduled (cron) + mandatory Hub push + per-job labels = a native "retrain N tenant adapters on a schedule, each artifact durable on the Hub" pipeline with no self-managed GPU fleet.
2. Economics are favorable at LoRA granularity. Per-tenant adapter retrain ≈ $0.10–$1.20; 100 tenants retrained weekly ≈ ~$130–430/mo; the builder model ≈ ~$10–50/mo. Combined order-of-magnitude ~$500/mo at 100 tenants weekly — linear in tenants × cadence, so cadence and flavor are the cost knobs. No idle-GPU burn (billed per running minute only).
3. Architecture = SFT + LoRA, one shared base + N adapters. Cheapest, smallest artifacts, matches per-tenant isolation. Full fine-tune reserved for the builder base. Adopt Unsloth if the builder grows past ~13B (halves cost/time).
4. Paid plan required — recommend Team ($20/user/mo) for org-namespace billing + tenant cost attribution via labels (Enterprise adds Resource Groups if hard per-tenant accounting is needed).
5. Hard operational rules to bake in: always push_to_hub + HF_TOKEN secret (ephemeral env); explicit padded timeout (default 30m is a trap); CPU dataset-validation pre-flight (~$0.01); Trackio with space_id for durable metrics + alerts for autonomous gating.
6. Open items for sibling docs: adapter storage/serving model → serving & integration; per-tenant data separation + the shared-base/adapter boundary → multi-tenant isolation; trace-corpus schema + outcome labels → datasets & trace pipeline.
7. Caveat: cost numbers are heuristic (the skill's own estimator is approximate and ships a stale price table — this doc uses live hf jobs hardware prices). Calibrate against one real builder run + one real tenant run before committing budget in the ADR.