Vektor Technologies
The task factory
for frontier RL.
We build next-generation training tasks by inverse generation: generate the world, compute the answer, then write the question. Verification is exact by construction - no LLM judges, no static benchmarks, nothing to leak.
One generated world → a chain of verifiable steps → an exact answer.
The problem
Frontier data is running out of road.
Static tasksets rot
Fixed benchmarks leak into training corpora and saturate. Once the rows are public, the eval is measuring memory, not capability.
LLM judges misjudge
An independent audit of 789 LLM-judged rollouts from a frontier RL system found an 8.5% false-positive rate and a 24% false-negative rate - reward noise baked directly into training.
Human authoring doesn't scale
Hand-built tasks typically run $200-$2,000 each, and coverage is limited to whatever an author happened to think of. The failure space stays unmapped.
The method
Inverse generation.
Everyone else writes a question and hopes a judge can check the answer. We run it in reverse - the answer exists before the question does.
-
01
Generate the world
Realistic enterprise worlds - inboxes, ledgers, calendars, ticket queues - synthesized from templates plus controlled noise. The generator plants the signal on purpose: three unpaid invoices, one double-booked payment, one FX discrepancy.
-
02
Compute the answer
Because the generator planted every fact, ground truth is computed programmatically while the world is built. Verification is an exact check against that computed answer - a short script, not a model judging a model.
-
03
Write the question
The task is natural language over what was planted: “Reconcile Q3 invoices against payments. Flag the three discrepancies.” Hard to solve, impossible to misjudge.
Every instance ships with an exact, machine-checkable answer. A lab can verify our quality in an afternoon - because every check is exact.
Why it challenges frontier models
Difficulty is a dial,
not a hope.
Compounding
Each generated step can be individually easy. Chain thirty of them and a model at 95% per-step reliability finishes the task about 21% of the time. We control chain length - so we control exactly where frontier models bleed.
Freshness
Every world is minted new at purchase. There is nothing to memorize, which is why static benchmarks saturate and generated ones don't.
Noise
Signal buried in realistic distractor volume - near-miss vendors, duplicate records, irrelevant threads - is the documented weak spot of long-context models. We tune the signal-to-noise ratio directly.
Domains
Wherever ground truth
can be computed.
Flagship
Enterprise finance tool use
Ledgers, invoices, reconciliation across realistic tool stacks. Ground truth is pure arithmetic - the strongest possible fit for inverse generation.
Coding
The repo and its failing tests are generated together. The tests are the exact verifier - no judge involved.
SRE / DevOps
Simulated incidents with the root cause planted by the generator. Diagnosis checked against known truth.
Browser / computer use
Synthetic web apps with fully known underlying state. Every click has a computed consequence.
Data analysis
Spreadsheets and databases with programmatically computed answers over fresh, noisy data.
The evidence
The research agrees.
8.5% / 24%
False-positive and false-negative rates in an independent audit of 789 LLM-judged rollouts from a frontier RL system.
DeepSWE audit, 2026
43%
Of GRPO training groups produce zero learning signal at a 10% pass rate with group size 8 (all-success or all-fail).
Advantage Collapse in GRPO, 2026
>15%
Of verifiers across five public terminal benchmarks were confirmed hackable under adversarial testing.
Terminal Wrench, 2026
$200-$2,000
Typical market rate per hand-authored task; complex website-replica environments run near $20,000.
Epoch AI, State of RL Environments
| Human-authored tasks | LLM-generated + judged | Vektor inverse generation | |
|---|---|---|---|
| Verifier | Human / model judge | LLM judge | Exact programmatic check |
| Judge error | Varies; unaudited | Measured at 8.5% FP / 24% FN in one frontier audit | Zero by construction |
| Cost per task | $200-$2,000 typical | Generation + judge compute | Milliseconds of compute |
| Contamination | Static rows leak | Static rows leak | Minted fresh at purchase |
| Difficulty control | Whatever the author wrote | Whatever the model wrote | Dialed: chain length, noise, distractors |
| Coverage | What an author imagined | What a model imagined | Systematic over the planted failure space |
Sources
- DeepSWE audit - arxiv.org/html/2607.07946v1
- Advantage Collapse in GRPO - arxiv.org/html/2605.21125v2
- Terminal Wrench - arxiv.org/html/2604.17596
- Epoch AI, State of RL Environments - epoch.ai/gradient-updates/state-of-rl-envs
The bottleneck in post-training
is no longer the model.
It's tasks worth learning from.
We build the factory.