Skip to main content

HF — datasets + trace pipeline

reference · tech-lead · updated 2026-06-27 · source

SPIKE artifact for effort #943 / sub #947. Design only — no shippable code. This doc answers: where does the training DATA come from, and how does it reach Hugging Face in the right format, for the two training targets:

  • Target A — the builder model (the dev-workflow corpus: effort→PR trajectories).
  • Target B — per-tenant models (each PyME client's own corpus / knowledge).

Siblings: jobs & cost · isolation · serving.

0. The hard partition (read first)​

Two corpora, never mixed — not with each other, not across tenants. This is the load-bearing constraint of the whole design:

  • The dev-workflow corpus (Target A) is internal: our efforts, issues, PRs, diffs. It is the "workflow" side of workflow-vs-product.md.
  • The tenant corpora (Target B) are product data: each client's knowledge/content, one private dataset per tenant.
  • ADR-006 states the rule verbatim: "Tenant/client data never mixes with the dev-workflow corpus (two datasets; connectors are product)." (ADR-006)
  • The Plan Engine doc repeats it: the dev-workflow graph "must never mix with tenant/client data" (plan-engine.md §1, §7).

On HF this maps to: separate HF Dataset repos, ideally separate namespaces/orgs, separate tokens. Detailed isolation model (org vs namespace vs resource-group, token scoping) lives in isolation; this doc owns the data shapes and the pipeline.

1. Target A — the builder-model corpus​

1.1 The workflow IS the dataset​

ADR-006 ("the effort schema is the dataset schema") makes every effort a builder-agent trajectory: goal → plan → patch → verify → outcome. The unit of work and the unit of training are the same record (ADR-006, effort-model.md).

The parent-issue template maps 1:1 to dataset columns:

Issue sectionDataset columnMeaning
## Goalproblem_statementwhat outcome, in 1–2 lines
## Scopeplanthe sub-issue DAG (parallel | sequential)
## For agentsretrieval_contextexact file paths / entry points
per-sub-issue diffpatch / patches[]the code change (snapshot pre-squash)
## Verification + outcomeverification / outcome_labelreward signal

1.2 Where the trace lives — Postgres + a JSONL exporter (already built)​

  • Store: the Plan Engine Postgres graph (plan schema). plan_version is append-only trajectory history; outcome_label (positive | negative | gold) carries the reward signal (plan-engine.md §2).
  • Exporter (exists today): scripts/trace/export-traces.sh — "merged effort PRs → JSONL builder-trajectory records (ADR-006 Phase 5)". It reads merged effort PRs via gh, snapshots per-sub-issue diffs pre-squash (squash destroys the per-sub pairs), scrubs secrets from every diff/body, and emits one JSONL object per effort with exactly the columns above (problem_statement, plan, retrieval_context, repo_state, patches[], verification, outcome_label, gold, policy). Output is gitignored — only the tool is committed, never trace data. (Lineage: trace-export work tracked under issues #535 / #881; the live script is scripts/trace/export-traces.sh.)

So the front half of Target A is done: Postgres trace → export-traces.sh → JSONL. The HF spike only needs to add the last hop: JSONL → HF Dataset (private).

1.3 JSONL → HF Dataset (private)​

The JSONL is uploaded as a private HF Dataset under the org namespace, e.g. tuempresadigital/builder-traces (private). Two equivalent paths:

  • CLI: hf upload tuempresadigital/builder-traces ./effort-traces.jsonl --repo-type dataset --private (private on first create; the CLI's hf repos create … --private + hf upload also works).
  • Python: datasets.Dataset.from_json(...).push_to_hub("tuempresadigital/builder-traces", private=True) — private=True only takes effect on first creation (datasets upload docs).

Training jobs read it back with load_dataset("tuempresadigital/builder-traces", token=True) (token from the job's HF_TOKEN secret) — private datasets require the token at load time (datasets upload docs).

1.4 Which TRL format the trace maps to​

The trace is rich enough to feed two TRL methods, derived from the same JSONL:

  • SFT (primary) — the natural fit. Build a conversational record per effort: messages = [{role: system, ...}, {role: user, <problem_statement + retrieval_context + plan>}, {role: assistant, <patch>}]. TRL's SFTTrainer accepts the conversational messages type (or text / prompt-completion) (TRL dataset_formats).
  • DPO (later, from outcome_label) — efforts that were reverted/closed-unmerged are negative; the clean-merged version is positive. A prompt / chosen / rejected triple is derivable where both a good and a bad patch exist for the same problem. DPO requires exactly prompt, chosen, rejected columns (TRL dataset_formats). This is a follow-up; SFT lands first.

The shaping (JSONL → messages / prompt,chosen,rejected) is a dataset.map(...) step done in the export pipeline (§4), before push, so the HF Dataset is already TRL-ready.

2. Target B — per-tenant data​

  • One private HF Dataset per tenant, never a shared one. Source = the tenant's own knowledge corpus / content (the MemPalace-style palace tuempresadigital/knowledge that retrieval reads is the dev/knowledge corpus; tenant product data is separate per-client). Suggested naming: tuempresadigital/tenant-<tenantId>-corpus (private), or a dedicated per-tenant namespace — decided in isolation.
  • The x-tedos-tenant seam already exists. The model contract carries a per-tenant header (TENANT_HEADER = "x-tedos-tenant"), sent only when a tenantId is supplied — the same id keys the per-tenant dataset and any per-tenant model (CONTRACT.md).
  • HARD RULE (no exceptions): tenant data NEVER mixes with the dev corpus or across tenants (workflow-vs-product.md; ADR-006 "two datasets; connectors are product"). On HF this means: distinct dataset repos, distinct upload jobs, tenant-scoped tokens, and no cross-tenant load_dataset in any single job.
  • Format: same TRL shapes as Target A — most tenant fine-tunes are SFT over the tenant's Q&A / content (messages or prompt-completion). Per-tenant embeddings/RAG (a reranker or bi-encoder over the tenant corpus) is a separate, cheaper track via train-sentence-transformers — mentioned here only as an option; its dataset shapes (pairs / triplets / labeled pairs) and losses live in that skill, out of scope for this spike.

3. TRL dataset formats + validation​

3.1 Required column shapes per method​

MethodDataset typeRequired columnsNotes
SFTconversational / language-modeling / prompt-completionmessages or text or prompt+completionmost flexible; the default builder + tenant path
DPOpreference (explicit prompt)prompt, chosen, rejectedstrictest — exact names; ~90% of raw datasets need a map
GRPOprompt-onlypromptmodel generates; reward computed online by a reward fn (no stored completions)
Rewardpreferencechosen, rejectedfor an RLHF reward model

Source: TRL dataset_formats and huggingface-llm-trainer skill (references/training_methods.md).

For the builder model: SFT now, DPO when negative-outcome pairs accumulate. GRPO is plausible later (verifiable rewards = "did the patch make the build/tests pass") but needs an online reward function, not just a dataset — defer.

3.2 Validation BEFORE GPU spend (mandatory gate)​

The huggingface-llm-trainer skill is emphatic: validate the dataset format on CPU before any GPU job. ~50%+ of training failures are format mismatches; DPO especially (exact column names). CPU validation costs ~$0.01 and <1 min; a failed GPU job wastes $1–10 and 30–60 min.

  • Tool: dataset_inspector.py (in the skill scripts/, also at https://huggingface.co/datasets/mcp-tools/skills/raw/main/dataset_inspector.py).
  • Run: hf jobs uv run --flavor cpu-basic … dataset_inspector.py --dataset tuempresadigital/builder-traces --split train.
  • Output markers: ✓ READY (use directly) / ✗ NEEDS MAPPING (emits copy-paste map code) / ✗ INCOMPATIBLE.
  • Gate rule for both targets: export → push private HF Dataset → dataset_inspector → only on ✓ READY does a GPU training job launch. This is the contract between this doc and jobs & cost.

4. Where the pipeline lives — the Python satellites​

Two external Python repos already own the data/training side and stay out of the monorepo (they would poison the turbo graph: weights + MLX + Python). They couple to tedos only through the wire contract (/v1 + x-tedos-tenant) (CONTRACT.md):

SatelliteToday's jobFits export→HF?
tuempresadigital/chat-datasetsdataset processing + MLX (149 MB weights)Yes — owns dataset processing. Natural home for JSONL→TRL-shape map + push to HF Dataset + dataset_inspector gate.
tuempresadigital/tedos-buildercorpus + tb sync (pushes the knowledge corpus retrieval reads), Python dev toolOwns corpus sync/triggering, not dataset shaping.

Recommendation — split by concern​

  • chat-datasets OWNS the export→HF-Dataset step (shaping + validation + push) for both targets. It is already the "dataset processing" satellite; adding "shape JSONL → messages / prompt,chosen,rejected, push private HF Dataset, run dataset_inspector" is squarely its job. For Target A it consumes the JSONL emitted by scripts/trace/export-traces.sh; for Target B it consumes each tenant corpus, one isolated run per tenant.
  • tedos-builder TRIGGERS it. It already owns scheduled/post-merge sync (tb sync "must run on a schedule or post-merge"), and the repo runs no GitHub Actions CI (billing-blocked — gates are Vercel build + local), so the canonical home of any schedule is the satellite's own repo (CONTRACT.md §Knowledge sync). tedos-builder invokes the chat-datasets export entrypoint.

Cadence​

TargetTriggerWhy
A — builder corpuspost-merge (effort PR lands → export-traces.sh for that PR → refresh the HF Dataset), with a hf jobs scheduled nightly catch-up sweepmatches kit-effort-close (snapshot pre-squash is already part of close); the merge IS the new training example, so append it then
B — tenant corpusscheduled per tenant (hf jobs scheduled run "<cron>" …) + on-demand re-export when a tenant's content changes materiallytenant content changes continuously; a per-tenant cron keeps each dataset fresh without coupling tenants

hf jobs scheduled uv run <SCHEDULE> <SCRIPT> (cron syntax, e.g. "0 3 * * *") runs the export/push as a managed scheduled job on HF infra — no CI runner needed (verified live: hf jobs scheduled --help). Pushing/training requires HF_TOKEN passed as a job secret (secrets={"HF_TOKEN": "$HF_TOKEN"}); the environment is ephemeral, so the dataset must be pushed to the Hub or it is lost (llm-trainer skill).

5. End-to-end data-flow diagrams​

Target A — builder model (dev-workflow corpus)​

effort/<N>  →  sub-issues merge  →  effort PR merged to develop
        │                                   │  (kit-effort-close: snapshot sub-diffs PRE-squash)
        ▼                                   ▼
  Plan Engine Postgres  ───────────►  scripts/trace/export-traces.sh
   (plan_version, outcome_label)        │  • one JSONL record / effort
                                        │  • secrets scrubbed, diffs pre-squash
                                        ▼
                              effort-traces.jsonl  (gitignored, local)
                                        │
        post-merge + nightly `hf jobs scheduled`  (tedos-builder triggers → chat-datasets runs)
                                        ▼
                  chat-datasets:  map → TRL shape (SFT `messages`; DPO `prompt/chosen/rejected`)
                                        ▼
                  hf upload → HF Dataset (PRIVATE)  tuempresadigital/builder-traces
                                        ▼
                  dataset_inspector.py  (cpu-basic gate)  →  ✓ READY ?
                                        ▼ yes
                  hf jobs uv run (TRL SFTTrainer, GPU)  ──►  model on Hub  tuempresadigital/builder-model

Target B — per-tenant model​

tenant <tenantId> corpus / knowledge   (product data; x-tedos-tenant seam)
                  │   ONE isolated run per tenant — never combined
   scheduled per tenant: `hf jobs scheduled run "<cron>" …`  (tedos-builder triggers → chat-datasets runs)
                  ▼
   chat-datasets:  map → TRL shape (usually SFT `messages` / prompt-completion)
                  ▼
   hf upload → HF Dataset (PRIVATE, tenant-scoped)  tuempresadigital/tenant-<tenantId>-corpus
                  ▼
   dataset_inspector.py  (cpu-basic gate)  →  ✓ READY ?
                  ▼ yes
   hf jobs uv run (TRL SFT, GPU, tenant-scoped token)  ──►  model on Hub  tuempresadigital/tenant-<tenantId>-model
                  │
                  └─ (optional, separate track) train-sentence-transformers → per-tenant reranker/embeddings for RAG

Both flows end at a model on the Hub; how that model is served back through /v1 + x-tedos-tenant is serving; GPU flavor/cost/scheduling is jobs & cost; the repo/namespace/token boundary that keeps A≠B and tenant≠tenant is isolation.

Implications for the ADR (#948)​

  1. Reuse, don't rebuild. The front half of the builder pipeline already exists (scripts/trace/export-traces.sh, Postgres plan_version/outcome_label). The ADR only needs to adopt the last hop: JSONL → private HF Dataset, plus the TRL-shape map. Cheap delta.
  2. Two datasets, two HF homes, by construction. The ADR must mandate separate private HF Dataset repos (and tenant-scoped tokens) for dev-corpus vs each tenant — the existing hard rule (ADR-006 / workflow-vs-product) becomes an HF-repo-layout rule. No single job ever loads more than one tenant.
  3. Validation is a required gate, not optional. dataset_inspector on cpu-basic must pass (✓ READY) before any GPU job — make it a step in the pipeline contract.
  4. Format: SFT first, DPO later. Ship SFT (messages) for both targets; DPO (prompt/chosen/rejected) is a follow-up once outcome_label negatives accumulate. GRPO/reward are out of scope for the MVP.
  5. Ownership: chat-datasets owns export→HF-Dataset; tedos-builder triggers it. Cadence: builder = post-merge + nightly hf jobs scheduled catch-up; tenant = per-tenant cron + on-demand. No new CI — HF scheduled jobs replace the (absent, billing-blocked) GitHub Actions runner.
  6. Secrets + monolingual corpus discipline carry over. export-traces already scrubs secrets; keep that on the tenant path too. Artifacts/corpus stay English (clean monolingual builder corpus) per ADR-006.

Sources​