ADR-011 — Hugging Face as the multi-tenant model-training platform

reference · tech-lead · updated 2026-06-27 · source

Status

Proposed — recommendation from spike #943 (effort), pending José's sign-off. On acceptance, flip frontmatter to canonical, set this section to "Accepted (José, )", and the phased rollout below becomes the work plan. No code or paid commitment is made by this ADR.

Context

tedos needs to train models in two distinct shapes, and today can do neither:

• Target A — the internal builder model. One model, trained on the effort→PR trace corpus (problem → plan → patch → verify → outcome). The corpus already exists in Postgres plan_version (ADR-006) and a JSONL exporter is built (#535, salvage #881). What's missing is a training backend.
• Target B — per-tenant client models. One fine-tune/adapter per PyME client, specialized on that client's own data — multi-tenant in the product sense.

The current model layer (@tedos/model-client, #874/#875) is inference-only: OpenAI /v1 over local (Ollama) / hosted (vLLM) / mock, with an x-tedos-tenant header for attribution. There is no per-tenant model, adapter, or training path, and we do not want to stand up and operate our own GPU fleet.

This ADR records the spike's finding on whether — and how — Hugging Face fills that gap. The full investigation lives in four reference docs (the evidence base for everything below):

• Jobs & cost
• Multi-tenant isolation
• Serving & integration
• Datasets & trace pipeline

Scope of the spike (decided with José, #943): HF only (no Modal/Replicate/own-GPU comparison), both targets, research/ADR output — no shippable code, $0 GPU spend.

Decision (recommendation)

Adopt Hugging Face as the training backend for both targets, with serving kept behind the existing @tedos/model-client seam. The four pillars:

1. Train on HF Jobs. Managed cloud GPUs, no infra to operate. Submit via UV scripts (PEP 723) / the TRL Jobs package; schedule retrains with hf jobs scheduled. Method: SFT + LoRA as the default — one shared base model + N tiny per-tenant adapters (MBs each). Reserve a full fine-tune for the builder base; adopt Unsloth if the builder grows past ~13B.

2. Isolate on the Hub. One org (tuempresadigital); one private dataset repo + one private model repo per tenant, each in its own per-tenant Resource Group, reachable only by a per-tenant fine-grained token injected as the Job's HF_TOKEN secret. The builder corpus + model live in a separate internal-only Resource Group no tenant token can reach. This is the Hub encoding of the hard rule "never mix tenant data into the dev-workflow corpus, nor across tenants" (ADR-006, workflow-vs-product.md).

3. Serve behind the seam, unchanged wire contract. Default = shared hosted vLLM with multi-LoRA (one base, per-request hot-swappable adapters — cheap for many tenants). Promote a tenant to a dedicated HF Inference Endpoint only as a premium/regulated tier. The /v1 contract stays frozen (non-breaking for the Python satellites); ModelPolicy gains additive adapterId / fineTunedEndpoint fields resolved inside providerFor().

4. Feed it from the satellites. The chat-datasets satellite owns the export → private HF Dataset → TRL-shape + dataset_inspector validation step (for both targets); tedos-builder only triggers it. SFT (messages) format is primary; DPO (prompt/chosen/rejected) is a later track once outcome_label negatives accumulate.

Cost (order-of-magnitude, from live hf jobs hardware 2026-06-27 — calibrate before committing)

Item	Estimate
Per-tenant adapter retrain (1–3B, SFT+LoRA, t4-small/l4x1)	~$0.10–1.20 / run
Builder model retrain (7B, ~3k examples)	~$9–11 / run (~$10/mo monthly cadence)
100 tenants retrained weekly + builder weekly + plan	~$500 / mo order-of-magnitude
Plan floor	Team ($20/user/mo: Resource Groups + token policy); Enterprise ($50/user/mo) for per-tenant cost attribution, SCIM, token revocation, full RG-at-create automation

Cost is linear in tenants × cadence — cadence and GPU flavor are the only knobs. (Note: the llm-trainer skill's bundled estimate_cost.py ships a stale, inflated price table — the live CLI numbers above supersede it.)

Consequences

Positive

• No GPU fleet to buy or operate; pay-per-minute, billed only while a Job runs.
• LoRA-per-tenant makes Target B cheap and storage-light; one base serves all via multi-LoRA.
• Isolation is enforced at-rest (Resource Group) and at-runtime (scoped token) — defence in depth.
• The model seam absorbs all of this additively; satellites and the /v1 contract are untouched.
• Reuses work already done: the trace corpus + JSONL exporter (#535/#881).

Negative / risks

• A paid HF tier is required (Team min; Enterprise for attribution at scale) — a recurring cost.
• Cost numbers are heuristic; a format-mismatch or runaway job wastes GPU $ — the dataset_inspector gate + padded timeouts + mandatory Hub push are mandatory guardrails.
• One open discrepancy to confirm at purchase: HF docs say Resource Groups are Team & Enterprise, but hf repos create --resource-group-id self-reports Enterprise-only (isolation doc).
• Vendor concentration on HF for train + (optionally) serve; mitigated by GGUF export + the seam keeping serving swappable.

Phased rollout (on acceptance)

• Phase 0 — Calibrate ($). Buy Team; run one real builder PoC + one real tenant LoRA PoC end-to-end (dataset → Job → private model on Hub → serve a test query). Replace the heuristic cost table with measured numbers. This is the first thing that spends money — gate it on José's OK.
• Phase 1 — Builder pipeline. Wire chat-datasets export → private HF Dataset → scheduled SFT Job → builder model on Hub (internal RG).
• Phase 2 — Per-tenant LoRA. Per-tenant dataset/model repos + Resource Groups + scoped tokens; shared vLLM multi-LoRA serving; ModelPolicy adapterId routing.
• Phase 3 — Scale. Dedicated Inference Endpoints for premium tenants; Enterprise tier for per-tenant cost attribution + token lifecycle automation.

References

• Spike effort #943 (subs #944–#948) · evidence docs linked in Context above.
• Related: ADR-006 (effort workflow = training trace), #770/#796 (builder model + telemetry), #535/#881 (trace exporter), #874/#875 (@tedos/model-client seam), #922 (multi-tenant client config).

Local agent skills (not vendored)

The HF training/work is supported by agent skills that are installed locally per developer and NOT committed to the repo (gitignored, #943): hf-cli, hf-mem, huggingface-best, huggingface-llm-trainer, huggingface-local-models, huggingface-trackio, train-sentence-transformers, trl-training (under .agents/skills/ and .claude/skills/). They are referenced here as the toolchain behind the spike; install on demand rather than vendoring their bundles into the tree.