Skip to main content

Hosted vLLM serving — Databricks custom LLM endpoints

reference · tech-lead · updated 2026-06-28 · source

Reference notes from a Databricks walkthrough on custom LLM serving (beta). Not a decision — this is one concrete provider option for the "hosted vLLM" branch already decided in apps/admin/src/plans/tedos-tech-lead-real-model.mdx and the local | hosted | mock LLMProvider abstraction (#522). The engine is provider-agnostic via modelRouter (packages/engine/src/modelRouter.ts), so a Databricks endpoint is just a TEDOS_MODEL_URL + TEDOS_MODEL_TOKEN — no architecture change. Captured as the first output of the host-evaluation spike (#523).

What the feature is​

  • Custom LLM serving on Databricks model serving endpoints, with vLLM as the serving engine.
  • Final surface is a managed REST endpoint, callable with the Databricks SDK, the OpenAI SDK, or curl — same shape as any other model-serving endpoint.
  • Positioned against the serverless Foundation Model API (pre-packaged, provider-agnostic: OpenAI / Claude / Gemini / Qwen). FM API is the default; custom serving is for what FM API can't host.

When to use custom serving over FM API​

  • A model not on Foundation Model API.
  • Customization / fine-tuning — PEFT, QLoRA, or multi-adapter inference (vLLM).
  • You already run vLLM (e.g. on Kubernetes) and want the same tunables on a managed endpoint instead of self-managing hardware.

How it works (deployment flow)​

  1. MLflow logs/registers the model artifacts — natively pulls weights from a Hugging Face model ID onto disk.
  2. Provide a vLLM startup command alongside the model ID: tensor parallelism (e.g. max to auto-detect hardware), batch size, max model length, KV-cache and other low-level tunables. This is logged as part of the registered artifact.
  3. Pick workload size (concurrency) + GPU workload type (A10 / H100, with regional limits; beta, evolving).
  4. Get a managed REST endpoint → invoke via Databricks SDK / OpenAI SDK / curl.

Wiring into TED OS​

  • Maps to the hosted provider slot — set TEDOS_MODEL_URL=https://<workspace>/serving-endpoints, TEDOS_MODEL=<endpoint-name>, TEDOS_MODEL_TOKEN=<bearer>. The engine's OpenAICompatProvider posts to <base>/chat/completions, which the Databricks OpenAI-compatible route serves.
  • Auth: prefer a Clerk-issued JWT where a JWT path exists; static TEDOS_MODEL_TOKEN (a Databricks PAT) only as the fallback. See .claude/rules + memory jwt-prefer-clerk.
  • Databricks is one way to stand up "hosted vLLM" — alongside self-managed vLLM (DO GPU), Modal, or RunPod. The #523 spike picks the winner on cost / cold-start / setup effort.

Open question for the spike​

A10 + scale-to-zero is supported but cold starts on a 7B-class model are non-trivial — a real latency datapoint to weigh against always-on cost. That trade-off is what #523 resolves.