Skip to main content

HF — serving + integration with @tedos/model-client

reference · tech-lead · updated 2026-06-27 · source

SPIKE (#943, sub-issue #946) — research/design only, nothing to ship. Scope: once a model or LoRA adapter is trained on Hugging Face (see jobs & cost, datasets), how do we serve it and plug it into the existing tedos model seam, per tenant? Isolation deep-dive: isolation.

1. The current seam (what we extend)​

@tedos/model-client (packages/model-client/) is the single, inference-only boundary to the open-source model. Grounding facts from CONTRACT.md + apps/api/MODEL_SETUP.md + src/policy.ts:

  • Wire protocol = OpenAI-compatible /v1. POST {baseUrl}/chat/completions, standard request
    • response. Any provider that speaks it works (Ollama, vLLM, LM Studio, managed serving). No SDK required.
  • Three modes in ModelPolicy.mode: local (Ollama on the client machine, $0, key ignored) · hosted (your cloud vLLM, default) · mock (no network, dev/tests).
  • ModelPolicy is the per-job/per-tenant knob: { mode, endpoint?, model?, token?, tenantId? }.
  • providerFor(policy) → an LLMProvider (request/response, used by @tedos/engine); resolveModelConfig(policy) → { baseUrl, model, apiKey, headers } (streaming, used by apps/chat via createOpenAICompatible). Env fallbacks (TEDOS_MODEL_URL / TEDOS_MODEL / TEDOS_MODEL_TOKEN, OLLAMA_*) resolve in exactly one place — policy.ts.
  • x-tedos-tenant header (TENANT_HEADER) — per-client attribution / isolation; sent only when tenantId is supplied. Same string for TS and the Python satellites.
  • No runtime deps, native fetch, framework-agnostic (Node engine + Next.js chat both consume).

The integration job is therefore narrow: resolve the right per-tenant model behind the same /v1 wire shape — ideally without changing the wire contract at all (changing it is breaking for the Python satellites). Everything below stays inside policy.ts resolution.

2. Serving options compared​

Three realistic ways to serve a trained checkpoint/adapter, all OpenAI-/v1 compatible so the seam is unchanged downstream:

(a) HF Inference Endpoints (dedicated). Fully-managed, per-model dedicated container with autoscaling + scale-to-zero. Real CLI surface (verified hf endpoints --help): deploy / catalog deploy (deploy from a Hub repo or the curated catalog), update (--min-replica / --max-replica / --scale-to-zero-timeout / --scaling-metric pendingRequests|hardwareUsage), pause / resume / scale-to-zero, describe, list. Deploy flags include --accelerator, --instance-size/-type, --region, --vendor, --type public|protected|authenticated|private, plus --custom-image / --env / --secrets for a custom vLLM container. The catalog (hf endpoints catalog list) carries the bases we'd train from (Qwen2.5-Coder 7B/14B/32B, Llama-3.x, Mistral, gpt-oss, etc.). Pricing is per-minute by hardware: ~$0.60/hr T4, ~$1+/hr L4/A10G, ~$4–8/hr A100 80GB; CPU from ~$0.03/hr. Scale-to-zero idles after inactivity (default 1h) — bursty tenant traffic typically lands $20–60/mo per endpoint.

(b) Current hosted vLLM + multi-LoRA. Keep the existing $TEDOS_MODEL_URL vLLM box and load LoRA adapters on it. vLLM serves one base model + many adapters, adapter chosen per-request (the adapter name goes in the model field of the OpenAI request). Adapters can be hot-swapped at runtime — loaded on first request from local disk / S3 via the dynamic-LoRA API or a LoRAResolver plugin, LRU-evicted past capacity, with <1% per-layer compute overhead. This is the "fine-tune once, serve N tenants on one GPU" pattern.

(c) GGUF → Ollama/llama.cpp (the $0 local tier). Convert the merged checkpoint to GGUF (convert_hf_to_gguf.py + llama-quantize, default Q4_K_M; code/technical → Q5_K_M/Q6_K) and serve via llama-server / Ollama, which already expose OpenAI /v1. This is the mode: 'local' path — runs on the client machine, data never leaves, $0 inference. Adapters are merged into the base before quantizing (no per-request multi-adapter on this tier). See the huggingface-local-models skill.

OptionLatencyCostIsolationMulti-tenant fit
(a) HF Inference Endpoint (dedicated/tenant)Low warm; cold-start penalty after scale-to-zero$$ per endpoint (per-min GPU); cheap only if idle a lotStrong — separate container/GPU, separate authPoor at scale (N endpoints = N GPUs $$); good for a few premium/regulated tenants
(b) Shared vLLM + multi-LoRALowest (warm base, sub-ms adapter swap)$ best — one GPU amortized across many tenantsLogical only (shared process/GPU; per-request adapter + x-tedos-tenant)Best for many tenants — 1 base, many hot adapters
(c) GGUF → Ollama/localLocal HW dependent$0 (client compute)Strongest — runs on tenant's own machine, data stays homeN/A across tenants (single-tenant per install); great for the local/offline tier

3. Per-tenant adapter routing (DESIGN)​

Goal: providerFor(policy) / resolveModelConfig(policy) should resolve the right per-tenant model without breaking the /v1 wire contract. Two extension points, both additive to ModelPolicy:

  • adapterId — names a LoRA adapter on a shared vLLM (option b). It maps to the OpenAI model field (vLLM routes by adapter name), so the wire shape is unchanged — only the value of model differs per tenant. This is the cheap default path.
  • fineTunedEndpoint — names/points at a dedicated HF Inference Endpoint (option a) for a tenant that warrants its own isolated container; overrides endpoint (+ its own token).

A tenantModelFor(tenantId) lookup (config table / DB) yields the per-tenant routing record; the seam stays the single resolution point.

// DESIGN SKETCH — illustrative only, NOT to implement in this spike (#943).
// Additive extension of the existing ModelPolicy in packages/model-client/src/policy.ts.

interface ModelPolicy {
  mode: 'hosted' | 'local' | 'mock'
  endpoint?: string
  model?: string
  token?: string
  tenantId?: string

  // --- proposed, per-tenant fine-tune routing (design only) ---
  adapterId?: string // LoRA adapter name on a shared vLLM (option b). Becomes the
  //   OpenAI `model` field → wire contract unchanged.
  fineTunedEndpoint?: string // dedicated HF Inference Endpoint URL for this tenant (option a);
  //   overrides `endpoint`. Pair with its own `token`.
}

// Per-tenant routing record, resolved from config/DB (the new lookup, not the wire):
type TenantModel =
  | { kind: 'adapter'; adapterId: string } // shared vLLM + LoRA  (cheap default)
  | { kind: 'endpoint'; endpoint: string; token?: string } // dedicated HF endpoint (premium)
  | { kind: 'base' } // no fine-tune yet → shared base model

// resolveModelConfig(policy) gains, conceptually, before the env fallbacks:
//   const t = policy.tenantId ? tenantModelFor(policy.tenantId) : { kind: 'base' }
//   adapter  → model = t.adapterId,  baseUrl = shared $TEDOS_MODEL_URL  (+ x-tedos-tenant header)
//   endpoint → baseUrl = t.endpoint, token = t.token                    (+ x-tedos-tenant header)
//   base     → today's behavior, unchanged
// Net effect on the wire: only `model` (= adapterId) or `baseUrl` (= endpoint) changes per tenant.
// The `/v1` path, header name, and request/response shapes are untouched (non-breaking for the
// Python satellites — CONTRACT.md § Stability).

Key property: the wire contract does not change. Per-tenant routing is a resolution-table concern inside policy.ts; downstream callers (@tedos/engine, apps/chat) keep calling providerFor / resolveModelConfig exactly as today.

4. vLLM multi-LoRA vs endpoint-per-tenant​

The central cost/scale tradeoff:

  • Shared vLLM + many LoRAs (b). One base model resident on one GPU; each tenant is a small adapter (tens–low-hundreds of MB) hot-swapped per request with sub-ms overhead. Cost grows with aggregate traffic, not tenant count — the natural fit for many SMB tenants (tedos's B2B shape). Trade-off: logical (not physical) isolation, a shared blast radius, and a finite hot-adapter cache (LRU). Mitigations live in isolation.
  • Dedicated endpoint per tenant (a). Each tenant gets its own container/GPU + auth boundary — strong isolation, but N tenants ≈ N GPUs, and scale-to-zero only helps if a tenant is mostly idle (and adds cold-start latency on wake). Economically this only works for a handful of premium/regulated tenants, not a long tail.

Recommendation: default every tenant to shared vLLM multi-LoRA (b) — adapterId routing — and reserve dedicated HF Inference Endpoints (a) — fineTunedEndpoint routing — as an opt-in tier for premium/data-residency/regulated tenants. Both resolve behind the same seam, so a tenant can be promoted from shared→dedicated by flipping its routing record, with zero downstream code change.

Builder model (the internal model trained on the effort/PR trace corpus — the dev-workflow assistant): single shared hosted vLLM as today ($TEDOS_MODEL_URL). New builder versions land either as a swapped base or as a named LoRA adapter on that same box. HF Inference Endpoints are the fallback host if we don't want to run the vLLM box ourselves; GGUF→Ollama covers the local/offline $0 tier. No per-tenant routing — one model, one endpoint.

Per-tenant model (a tenant's fine-tune over their own data): shared vLLM + multi-LoRA by default — train a LoRA per tenant (HF Jobs, see jobs & cost), push it to a private Hub repo, and route via adapterId (loaded on the shared vLLM, hot-swapped per request, tagged with x-tedos-tenant). Promote to a dedicated HF Inference Endpoint (fineTunedEndpoint) only for premium/regulated tenants needing physical isolation or data residency. Local/offline tenants get the merged adapter as GGUF on Ollama (mode: 'local'). All three paths resolve in policy.ts behind the unchanged /v1 + x-tedos-tenant contract.

Implications for the ADR (#948)​

  • Don't change the wire contract. Per-tenant fine-tune routing is achievable as an additive ModelPolicy extension (adapterId / fineTunedEndpoint) + a tenantModelFor() resolution table inside policy.ts. The /v1 path, x-tedos-tenant header, and request/response shapes stay frozen (non-breaking for the Python satellites).
  • Adopt shared vLLM multi-LoRA as the default serving substrate; dedicated HF Inference Endpoints are an opt-in premium/isolation tier, not the baseline (cost: N tenants ≠ N GPUs).
  • GGUF→Ollama remains the mode: 'local' $0 tier; adapters merge into the base before quantizing (no multi-adapter locally).
  • Open decisions for the ADR: where the per-tenant routing table lives (DB vs config) and who owns adapter lifecycle/cache sizing; the trust boundary of shared-GPU multi-tenancy (defer to isolation); and whether we self-host vLLM or lean on HF Endpoints for the shared base.

Sources​