Skip to main content

HF — multi-tenant isolation on the Hub

reference · tech-lead · updated 2026-06-27 · source

Spike output for effort #943 (sub #945). Decides how TED OS keeps each client (PyME tenant) and the internal builder model cleanly separated — data, weights, and access — on the Hugging Face Hub. Siblings: jobs & cost · serving · datasets.

The tedos hard rule this must satisfy (.claude/rules/effort-model.md § Training-trace hard rules): "Never mix tenant/client data into the dev-workflow corpus" and (by extension) never mix tenant A's data with tenant B's. .claude/rules/workflow-vs-product.md draws the same line: dev-workflow artifacts (efforts, plans, the builder corpus) are internal, tenant data is product — the two never share a store. This doc maps that boundary onto Hub primitives.

0. The one constraint that shapes everything: HF has no nested namespaces​

A Hub repo id is exactly two levels — owner/repo, where owner is a user or an org. There is no org/team/repo nesting. So "a namespace per tenant" inside one org is not a Hub concept; the only intra-org partition is:

  1. a naming prefix on the repo name (cosmetic / discovery only — not a security boundary), and
  2. private + a Resource Group (the real access-control boundary).

Everything below follows from this: the security boundary is the Resource Group ACL, not the name.

1. Repo isolation — layout options​

Each tenant needs at minimum a private dataset repo (its corpus) and a private model repo (its weights/adapters). Three topology options:

OptionShapeProsCons
A. One org + per-tenant prefix + per-tenant Resource Group (recommended)tuempresadigital/tenant-<slug>-corpus (dataset), tuempresadigital/tenant-<slug>-model (model); each pair in its own RG rg-tenant-<slug>One org to administer + bill; RG gives a hard per-tenant ACL; private repos in an RG are invisible to non-members; one plan/seat poolRepo names share a flat namespace (prefix discipline needed); RG requires Team+
B. One org + per-tenant prefix, no Resource Groupssame names, all repos just --privateWorks on free/Pro (no Team needed)Every org member with read sees every private repo — no per-tenant ACL. Fails the hard rule the moment there are 2+ tenants + >1 internal person. Acceptable only single-operator / pre-MVP
C. Separate HF org per tenantorg tenant-acme/..., org tenant-beta/...Strongest isolation: separate namespace, billing, member list, planN orgs to create/administer/pay per tenant; doesn't scale to many PyMEs; white-label friction; cross-org tooling overhead

Decision: Option A. Resource Groups give Option-C-grade ACL isolation without N orgs. Reserve Option C only for a client that contractually demands a dedicated tenant boundary (data-residency, separate billing entity).

Repos are created private and (for tenant corpora) gated:

hf repos create tuempresadigital/tenant-acme-corpus --type dataset --private \
    --resource-group-id <rg-id>          # --resource-group-id: Enterprise-org flag (see §2)
hf repos settings tuempresadigital/tenant-acme-corpus --repo-type dataset --gated manual

2. Access control — tokens, gating, Resource Groups, and which tier​

Three independent layers; use all three.

2a. Resource Groups (the per-tenant ACL — Team & Enterprise)​

security-resource-groups

  • An RG groups related repos; a repo belongs to exactly one RG. Org members must be added to an RG to see its repos. A private repo in an RG is visible only to that RG's members — this is the primitive that enforces tenant-A-can't-see-tenant-B.
  • Four roles per RG: read · contributor (write only to repos the user created) · write (full write to all repos in the RG) · admin (manage members + repos of the RG). Org admins can manage all RGs.
  • Plan: the Resource Groups feature is Team & Enterprise. Cost attribution to an RG (per-team billing of Jobs/Spaces/Endpoints), SCIM, and auto-join policies are Enterprise. Note: the hf repos create --resource-group-id flag's help text says "only available for Enterprise Hub organizations" — treat Enterprise as the safe assumption for full RG-at-create + cost attribution; Team is the floor for the ACL feature itself. Confirm the current tier matrix at purchase.

2b. Repo gating (hf repos settings --gated [auto|manual|false])​

datasets-gated · models-gated

  • Gating adds an access-request gate in front of a repo (request form → auto or manual approval; author can reject/cancel anytime; manage via UI or API). Works on any tier.
  • For our model: gating is defence-in-depth, not the primary boundary — private + RG already blocks outsiders. Use --gated manual on tenant corpora as an explicit, auditable approval log of who/what got access (good for compliance posture), and to gate any model we ever expose more widely.

2c. User Access Tokens — fine-grained vs classic (all tiers; policy = Team+)​

security-tokens

  • Classic tokens are read or write across everything the principal can access — too broad for multi-tenant; a leaked write token touches every tenant.
  • Fine-grained tokens scope to specific repos and/or a specific org, with specific permissions (read/write contents, manage settings, etc.). A fine-grained token can be scoped to a single repository — this is the per-tenant credential we want.
  • Org token-management policy (Team & Enterprise) can require fine-grained tokens (classic rejected with 403) and require admin approval (org-scoped tokens enter Pending until an admin approves; deny/revoke supported, revoke = Enterprise). Enterprise also offers OAuth Token Exchange (RFC 8693) to mint per-member tokens programmatically — the clean path for an internal platform that issues tenant tokens on demand.
  • Rule: one fine-grained token per tenant, scoped to that tenant's repos only; one separate fine-grained token for the builder repos. Never a classic org-wide write token in a job.

3. Secrets handling — how a training job gets a per-tenant token​

HF Jobs / Spaces / repo-create all accept --secrets KEY=value (or --secrets HF_TOKEN to pass the caller's token) and --secrets-file (hf jobs run … --secrets HF_TOKEN=<tok>; hf repos create … --secrets …; hf spaces secrets add). The Python run_job(..., secrets={"HF_TOKEN": ...}) is the same seam. Discipline:

  • One token per job, scoped to one tenant. The job that trains tenant ACME is launched with the ACME fine-grained token (scoped to tenant-acme-* only). It can read ACME's corpus and write ACME's model and nothing else — a bug or prompt-injection inside the job cannot reach tenant BETA or the builder corpus, because the credential physically can't.
  • Never inject a classic org-wide token or a broad write token into a tenant job — that re-opens the cross-tenant blast radius the whole design closes.
  • The orchestrator (our API) holds the tenant→token map server-side (Vercel/Fly env, not in the repo), mints/rotates via the org token policy, and passes the right one as secrets at launch. Tokens never land in issue bodies, diffs, or the dataset (effort-model.md: scrub secrets).
  • Rotate per tenant independently; revoke (Enterprise) on offboarding without touching other tenants.

4. Training-corpus segregation (the HARD RULE), mapped to Hub primitives​

BoundaryWhere it lives on the Hub
Tenant A corpus ≠ Tenant B corpusSeparate private dataset repos, each in its own Resource Group; private-in-RG = B's members can't even list A's repo
Tenant data ≠ dev-workflow (builder) corpusThe builder corpus is a different repo in a different RG (internal-only members); no tenant token can read it and no builder token can read tenant data
Job-time enforcementPer-tenant fine-grained token as the job secret — the credential is the runtime fence, RG is the at-rest fence
Audit--gated manual on tenant corpora records an explicit access-grant log

The boundary is redundant by design: at-rest (private + RG ACL) and at-runtime (scoped token). Either alone would technically satisfy "don't mix"; both together make a single misconfiguration non-catastrophic.

5. Builder model vs per-tenant split​

The internal builder model is trained on effort→PR traces (effort-model.md: the effort schema is the dataset schema) — non-tenant, internal-only dev-workflow data. It must never share a store with any client model/corpus, and tenant jobs must never be able to read it.

Concrete topology (Option A):

  • Builder corpus → tuempresadigital/builder-corpus (private dataset) in RG rg-internal (members = internal team only).
  • Builder model → tuempresadigital/builder-model (private model) in rg-internal.
  • Builder token → fine-grained, scoped to builder-* repos only.
  • Every tenant lives in its own rg-tenant-<slug> with its own token; no tenant token is ever scoped to rg-internal, and the builder token is never scoped to a tenant RG.

This is the Hub realization of the workflow-vs-product.md boundary: builder = workflow (internal), tenant models = product (client).

HF org: tuempresadigital  (single org, Team min / Enterprise for cost-attribution + SCIM + revoke)
│
├─ rg-internal                         ← Resource Group: internal team only
│   ├─ builder-corpus     (dataset, private)        ← effort→PR traces (NON-tenant)
│   └─ builder-model      (model,   private)
│        token: hf_builder  (fine-grained → builder-* only)
│
├─ rg-tenant-acme                      ← Resource Group: ACME-scoped members only
│   ├─ tenant-acme-corpus (dataset, private, gated manual)
│   └─ tenant-acme-model  (model,   private)
│        token: hf_acme     (fine-grained → tenant-acme-* only)   ← job secret HF_TOKEN
│
├─ rg-tenant-beta                      ← Resource Group: BETA-scoped members only
│   ├─ tenant-beta-corpus (dataset, private, gated manual)
│   └─ tenant-beta-model  (model,   private)
│        token: hf_beta     (fine-grained → tenant-beta-* only)   ← job secret HF_TOKEN
│
└─ …one rg-tenant-<slug> + token per client
TenantRepos (tuempresadigital/…)Resource GroupToken (fine-grained scope)Job secret
Internal / builderbuilder-corpus (ds), builder-modelrg-internalhf_builder → builder-* r/wHF_TOKEN=hf_builder
acme (client)tenant-acme-corpus (ds, gated), tenant-acme-modelrg-tenant-acmehf_acme → tenant-acme-* r/wHF_TOKEN=hf_acme
beta (client)tenant-beta-corpus (ds, gated), tenant-beta-modelrg-tenant-betahf_beta → tenant-beta-* r/wHF_TOKEN=hf_beta
…each new clienttenant-<slug>-corpus, tenant-<slug>-modelrg-tenant-<slug>hf_<slug> → tenant-<slug>-*HF_TOKEN=hf_<slug>

Provisioning per new tenant (4 steps): create the two private repos with --resource-group-id → --gated manual the corpus → create the RG with the tenant's members → mint the fine-grained token scoped to tenant-<slug>-*. Tear-down on offboarding: revoke the token (Enterprise) + delete/move the repos; no other tenant is touched.

Implications for the ADR (#948)​

  1. Adopt Option A — single org tuempresadigital, per-tenant repo prefix for discovery, and a Resource Group per tenant as the actual ACL boundary. The name is cosmetic; the RG is the fence (HF has no nested namespaces, §0).
  2. Tier: Team plan is the floor (Resource Groups + token-management policy). Go Enterprise if we want per-tenant cost attribution of training Jobs, SCIM, token revocation, or OAuth Token Exchange to mint tenant tokens programmatically — likely needed at scale. The --resource-group-id create flag self-reports Enterprise-only, so budget Enterprise for the full automated flow. (Free/Pro = single-operator pre-MVP only; it cannot enforce per-tenant ACLs.)
  3. Defence-in-depth is non-negotiable: at-rest (private + RG) and at-runtime (per-tenant fine-grained token as the job HF_TOKEN secret). Never a classic org-wide token in a job.
  4. Builder corpus/model live in rg-internal, fully disjoint from any tenant RG/token — the Hub encoding of the "never mix tenant data into the dev-workflow corpus" hard rule.
  5. Org policy: turn on require fine-grained tokens + admin approval so a stray classic token can't reach org resources.
  6. Open follow-ups for the ADR: confirm the exact Team-vs-Enterprise line for resource_group_id at create time with HF; decide where the tenant→token map + rotation live in our API; gate cost (training Jobs billing per RG) — see jobs & cost; serving isolation (private Inference Endpoints per tenant) — see serving; corpus build/scrub pipeline — see datasets.

Sources​