Ultramarine 1 Flash: Scaling Non-Verifiable RL

We present Ultramarine 1 Flash, an agentic model for professional knowledge work. Ultramarine applies large-scale reinforcement learning to a strong open base model, GLM-5.3 Flash, and improves on it across all six knowledge-work benchmarks we evaluate, by 4.3 points on average, while generating 12% fewer tokens. Coding performance holds steady, and on our internal knowledge-work suite Ultramarine scores within three points of Claude Opus 5.5 and GPT-6 Sol at about a tenth of GPT-6 Sol's cost and a fiftieth of Opus's.

Recent work has explored scaling agent post-training through synthetic tasks and environments [1] [2] [3] [4] [5] [6] [7] [8] [9], though this research has so far largely concentrated on verifiable tasks, where coding has consistently held the largest share of the RL mix. Most professional work is not verifiable: models are asked to produce artifacts ranging from diligence memos to litigation chronologies and therefore cannot be checked by regex parsers or unit tests. The quality of the answer often turns on judgment and taste, and, as such, a defensible reward for open-ended work has to be shaped per task. In this work, we scale non-verifiable task synthesis across agentic knowledge work.

We formalize each RL task the way recent work formalizes a verifiable task: an environment, a model instruction, and a verification system [10]. Each environment and task is synthesized with a custom synthesis pipeline per domain, and then used in a standard RL post-training setup.

Scaling Non-Verifiable Verification

Scaling post-training for professional knowledge work requires tasks that are realistic and open-ended, yet specified well enough to distinguish better work from a different interpretation of the assignment. The reward must be shaped to recognize correct work without requiring a particular solution [11].

Task and rubric co-design is the central problem of authoring high quality RL training data. Per our rubric-grading work [12], each criterion defines a boundary over possible answers. A criterion copied too closely from one golden reference rejects valid alternatives, while a vague criterion such as “provides an appropriate recommendation” can reward plausible sounding errors. The criterion must instead state the substantive condition that earns credit, together with the facts, thresholds, and context needed to check it against the deliverable. Naming the relevant entities, dates, and assumptions keeps the standard explicit, reducing judge variance and increasing alignment with expert human judgment.

A criterion therefore tests one coherent proposition. Conjunctions may be used to express conditions that hold together: a note for the managing director may need to state what was checked "and" the decision implication "and" what remains open, so the reader can distinguish verified findings from unresolved decisions. Similarly, "or"s admit genuinely equivalent ways to meet a requirement. Independent accomplishments remain separate criteria so that success on one can receive credit even when another is missed. Each proposition is judged met or unmet, and its wording defines the acceptable variants to support generalization.

These binary judgments combine into the raw reward, and the full rubric’s coverage and weights should reflect professional importance. Reliable grading thus depends on the boundaries of individual criteria and how they compose, rather than simply choosing a stronger or agentic judge.

There are three sources of data we considered to curate the Ultramarine curriculum: (1) production traces from real users driving agents, (2) expert-authored tasks, and (3) synthetically generated data.

Production traces offer realism, but users often leave assumptions unstated and rely on shared context [12]. Additionally, production state is non-stationary and converting traces into group-relative RL tasks requires reconstructing an environment with perfect fidelity to production in order to replay rollouts. The latter issue is solvable with careful state snapshotting, but the former is more involved: converting user queries into RL tasks means embedding implicit assumptions about user preference into what should be a generalized reward that spans every correct instance in the set of possible answers. As a toy example, a user asking for the margin profiles of the top 10 medical technology companies may have meant the top 10 by revenue, and so a successful task shaped from that trajectory would embed that implicit assumption in the reward, penalizing perfectly valid responses choosing market cap, enterprise value, or profit to rank them. Solving for this would require materially reworking the query to make it constrained and evaluable. We chose not to pursue this path given the aforementioned complexity, though we believe it remains a rich area of research.

Expert-authored tasks resolve that ambiguity, as annotators are given explicit instruction to construct tasks that are constrained, well-defined, and evaluable, but maintaining breadth and grading quality by hand is costly, and narrow tasks and rubrics still exclude defensible answers. As model intelligence grows at a clip, meeting all of these requirements per-task is difficult to scale, highlighted by the growing cost of human data and errors in human authored public benchmarks [13] [14].

As such, we continued with a purely synthetic recipe to scale RL task quality and grading fidelity, reliably, as a function of compute, across finance, law, consulting, and business operations. Unlike production data, it allows us full control over data distribution and lets us choose seeds from scratch, and unlike expert authored data, it enables us to create low-cost, high-throughput tasks and filter for quality and difficulty at scale.

Domains, Taxonomy, and Preventing Mode Collapse

The training curriculum spans four domains (finance, legal, consulting, and operations), 19 professional practice areas, and 88 distinct environment types (Table 1). Each practice area and environment type is expressed as a standalone generation pipeline that samples over a taxonomy of axes. For each instance of generation, the taxonomy is sampled across its axes: deliverable format, sources of complexity, stakeholders, source conflicts, hints, the stage of the transaction or matter, numerous domain-specific dimensions, and so on. Sampling across these axes gives each practice area a nominal configuration space on the order of 10⁷–10¹⁰ combinations. Each environment instantiates further into many concrete instances with different companies, documents, and figures. This combinatorial breadth helps prevent mode collapse in LLM-generated data [1] [15].

Anatomy of a Task

Tasks follow the Harbor specification [10]. Each task is packaged as a self-contained, runnable instance with three parts: the instruction, handed to the agent as a professional would receive it; the environment, the workspace and any live services, with a Dockerfile and service definition; and the grader, the task's rubric and the harness that scores the delivered work.

Appendix A presents a complete example, including the task instructions, workspace files and connected services, and full weighted rubric.

Environments

The curriculum is built on environments. An environment is a mounted workspace an agent inhabits the way a professional would:

• A filesystem of realistic documents, including models, memos, filings, trackers, decks, correspondence, and so on.
• A live system of record it reads and mutates as part of the work, including CRM and ERP/general-ledger systems, warehouse and transport management (WMS/TMS), manufacturing and process control (MES), GRC and ticketing, and property and work-order platforms.
• Synthesized resource interfaces spanning MCP, HTTP, and CLI

To build environments, we retrieve real documents and synthesize materials that have no suitable public source. The workspace combines:

• Regulatory filings: SEC EDGAR filings, including 10-Ks, 10-Qs, 8-Ks, proxy statements, and S-1s, plus exhibits such as credit agreements and indentures.
• Public web material: company sites, press releases, investor presentations, and regulatory notices.
• Research material: analyst reports, market-data snapshots, and sector studies, provided as point-in-time snapshots labeled with their source and date.
• Synthetic documents: realistic contracts, patents, internal reports, internal emails, and so on based on seed data.

Each environment also has a private ground-truth record to provide privileged information to task proposer agents.

Instructions and Graders

We construct filesystems, document corpora, and MCP servers through bespoke pipelines for each of the 88 environment types. Within each environment, task instructions and rubrics are adversarially generated along the sampling axes described above, spanning 10⁷–10¹⁰ possible combinations.

Generating frontier tasks is compute intensive: Producing a single task involves several dozen independent agent loops and multiple rounds of review, checking grounding, professional realism, difficulty, and consistency between the assignment and its grading criteria. Additional filters check for answer leakage and reward hacking, using classifiers iteratively added or refined in response to failures observed during training. A final curriculum filter screens tasks based on the base model's performance.

Grading Logic

Deliverables are practical work products, including Excel models, PowerPoint decks, and Word or PDF documents that the agent authors or edits in place. This result is then parsed and rendered as text to a naive LLM rubric grader.

We use our open-source rubric package [16], with PerCriterionOneShotGrader and a custom grading prompt tuned for agreement with human experts following the evaluation process described in Notes on Choosing a Rubric Judge [12].

Recent work uses multi-harness training to improve generalization [17] [9]. However, training on several harnesses at once was not feasible on our compute budget. Instead, we surveyed the tool contracts and execution structures of the coding harnesses in wide use (Claude Code [18], Codex [19], Cursor [20], Kimi Code [21], OpenAI Agents SDK [22], OpenCode [23], OpenHands [24] and its Software Agent SDK [25], Qwen Code [26], Mini SWE Agent [27], Terminus [28], and SWE-agent [29]), and created a new Python harness built on common shared conventions, aiming to train a policy with the most transfer across harnesses.

Training and evaluation share Feather's agent loop, tools, and compaction. Training routes model requests through Verifiers’ inference interception layer [30]; the asynchronous training loop refreshes inference weights mid-rollout [31].

Tool Interface

We compared the file-editing and process-control interfaces of existing harnesses and used paired evaluations and trajectory inspection to determine Feather's tool configuration.

File Editing Tools

Table 2. File-editing interface categories considered in this work.

¹ OpenCode uses apply_patch for selected GPT models and separate edit and write tools for other models.

Existing harnesses abstract file editing in two main ways: a patch interface describes changes through context lines and additions/deletions, while a string-replacement interface describes changes through old and new text.

To change timeout: 30 to timeout: 60 in config.yaml, Codex's apply_patch uses textual context to locate the patch and does not require the model to supply line numbers:

*** Begin Patch
*** Update File: config.yaml
@@
-  timeout: 30
+  timeout: 60
*** End Patch

String replacement requires only the old and new text, with the tool locating the match in the file:

edit(file_path="config.yaml",
     old_string="timeout: 30",
     new_string="timeout: 60")

A second design choice is whether to expose file operations through a single combined tool or through dedicated per-operation tools.

Table 3. Combined and dedicated file-operation interfaces.

We ran both configurations on the same set of tasks and inspected the generated calls. The trajectory analysis did not show a clear advantage for split tools. Feather adopted dedicated tools for their simpler per-operation schemas and consistency with several widely used harnesses.

Beyond the tool interface, execution order also matters for correct file edits. A model turn may contain multiple tool calls, but Feather executes them sequentially in model-produced order to preserve dependencies between reads, shell commands, and file modifications. Even with sequential execution, intervening shell commands or background processes may change a file after it has been read. Feather therefore records a content fingerprint—a hash of the complete file contents—on each read and checks it before editing or overwriting the file. If the contents have changed, the modification is rejected and the model is asked to reread the file.

Feather's tool set comprises read, edit, write, grep, and glob:

• read returns text with line numbers and accepts offset and limit parameters to bound the requested range. A successful read records a content fingerprint of the file.
• edit and overwrites through write check the content fingerprint before applying changes, rejecting the modification when the current contents do not match the record. edit requires the old text to match uniquely in the file by default, and supports explicit multi-occurrence replacement through replace_all.
• grep and glob provide structured search, avoiding the need for models to construct Bash commands and parse their output. grep returns matching files and line numbers, glob returns paths by filename pattern. Both enforce a result limit and include a truncation indicator when results exceed the limit, so the model knows to narrow its query.

Bash Tool

Across existing harnesses, shell interfaces differ primarily in what happens to a process after a call returns and how the model regains access to it.

Table 4. Shell interfaces after a call returns.

Feather uses explicit background tasks. Each call starts a fresh shell. Changes to the working directory and environment variables do not carry into later calls. The model starts background processes with run_in_background, reads their output with task_output, stops them with task_stop, and lists them with task_list.

Todo Tool

To let the model maintain a work plan over long-horizon tasks and preserve progress tracking across multiple rounds of context compaction, Feather provides a todo tool. The model can create or update the list at any point. The list persists across compaction within a single agent run and is reattached in its current state to the compressed context.

Final Tool Set

Table 5. Feather's default tool set.

Optional web_search, web_fetch, and MCP servers are configured per task.

System Prompts and Exploration

"Training on your production harness" is an emerging strategy in agent post-training, and it is especially helpful to familiarize the model with the tools, arguments, and interfaces it will use in production. However, this approach can restrict exploration when applied to prompts, tool descriptions, and other model instruction. These show up as prescriptive context that increases performance during test time, but constrains model exploration and conditions behavior to a specific prior during training. We only introduced production model instruction in final stages of training.

For the bulk of training, we rewrite all model-facing prose to be general, lightweight, and unopinionated. It is up to the policy to decide how to make do with the options and information presented to it, and there is no one prescribed workflow it should adhere to.

Our training system prompt is therefore three sentences, about 50 tokens, where the production prompts we surveyed in real harnesses run from roughly 1,800 to 4,400 (Table 6). Similarly, our tool descriptions state what a tool does and what it guarantees, rather than when or how to use it, and we omit the planning and verification workflows that production harnesses often prescribe.

You are an agent working in a Linux sandbox with access to the provided tools and workspace.

Do not expose, log, transmit, or write secrets or credentials.

When relevant, put requested file deliverables at the paths specified by the user or environment.

Example 1: Complete Feather System prompt

Feather's tool descriptions, also shown to the model, similarly opt for exposition rather than prescription.

Read a text file with line numbers. `offset` and `limit` bound the returned range. A successful read records the file revision required to edit or overwrite its current contents.

Example 2: Complete Feather read description

Table 6. Default system prompt and file-read tool descriptions.

Vision

We did not train with vision. To nudge the model against repeated image-read attempts, without imposing additional harness constraints, read returns a text observation for supported image extensions:

read({"file_path": "chart.png"})

→ Vision is not available in this environment.

Example 3: Feather image-read response

Tool Failure Handling

A failure should shape the policy only when the policy caused it. An error that follows from the model's own actions, such as a malformed call or a failing command, is part of its exploration, so we return the error and let the model correct course within the same trajectory. When the environment fails beneath it, such as a lost sandbox, the outcome carries no information about the policy, and therefore scoring it would only add noise to the reward. We discard that attempt, excluding it from training rather than assigning zero reward, and restart the task in a fresh environment.

Table 7. Tool failure classes and their handling.

Training agents on professional work requires environments that support both document editing and interactions with live services. Each rollout uses an isolated E2B sandbox with separate Docker Compose containers for agent execution, MCP services, and grading.

We built our sandbox architecture around four choices: restore initialized environments from snapshots, keep the agent loop outside the execution environment, isolate task execution from services and grading, and treat remote communication as both a latency cost and a source of recoverable failures.

Initialized Snapshots

Environment-level hot snapshots preserve filesystem and memory state, including running services [33]. Tasks and rollout replicas restore independent copies, then load task-specific overlays and the pinned Feather runtime. Task-input and Feather updates do not require rebuilding every world snapshot.

Agent Loop Outside the Sandbox

The agent loop runs on the orchestrator, while commands and file operations execute inside the sandbox [34]. Model inference credentials and agent-loop code remain outside the environment the model can modify, so orchestration updates can ship without rebuilding environment snapshots. More importantly, nothing the model does in the sandbox can alter its own harness, credentials, or grader.

Execution, Services, and Grading

The agent accesses application services through MCP, without direct access to service processes or their storage. Deliverables are writable, and grading artifacts and verifier state are outside the agent's execution boundary. Reusable verifier dependencies and world-level grading infrastructure are baked into snapshots in advance. Finally, task-specific grading material is supplied after the rollout for isolation.

Remote Calls and Recovery

Operations that look cheap locally can become expensive when each requires a remote request. Our initial file-loading path created directories and naively wrote files individually. On an upload test of about 50 files, packing them into a single tarball cut the time from 52 seconds to 2.2 seconds, a 96% reduction. Each tool call made several roundtrips, and fusing those roundtrips into fewer requests removed the same kind of delay.

We adapt the RL setup used for Kos-1 Lite [35] and Kos-1 Experimental [36] to full agent trajectories.

For a group of rollouts on the same task, our modified CISPO objective is [31] [37]:

$m_{i,t}$ marks model-generated training tokens, and $N$ counts those tokens in the batch.

In addition to CISPO we incorporate Rollout Routing Replay (R3) [38], mean-centered advantages [39], and zero-advantage filtering [40].

Async Updates and Mini-Steps

Our earlier synchronous setup used three policies (current, pre-update, and rollout) [41]: CISPO compared the current and pre-update policies, while token-level truncated importance sampling corrected the pre-update/rollout mismatch [42]. That implementation used an extra forward pass for pre-update log probabilities and multiple minibatch updates per rollout batch.

Here, completed task groups enter a buffer continuously, and a full batch triggers one optimizer step, with no mini-steps [31] [43]. Off-policyness therefore depends on trainer throughput relative to rollout throughput. The update uses the importance sampling ratio of the current policy to the rollout policy directly, which removes the separate pre-update policy and its forward pass.

We penalize generated output and excessive turns, alongside malformed tool calls and calls to nonexistent tools:

$T_i$ counts generated tokens across the trajectory, $n_i$ counts agent turns, and $m_i$ counts invalid tool calls. Each invalid call incurs a penalty $\lambda$, capped in total at $c_{\mathrm{tool}}$. The turn factor $f(n)=\operatorname{clip}((n-n_0)/(n_{\max}-n_0),0,1)$ fades positive reward from $n_0$ to zero at the turn cap $n_{\max}$. $G_g$ enables output penalties when the group mean reward after this fade is positive.

The output penalty grows at rate $\alpha/L$ beyond allowance $B$, up to $c_{\mathrm{out}}$. A separate extreme-length penalty starts at $B_{\mathrm{tail}}$, rises over $L_{\mathrm{tail}}$ tokens, and caps at $\eta$.

We evaluate Ultramarine 1 Flash on six knowledge-work benchmarks and report two coding benchmarks for reference (Table 8). Each benchmark runs on its official harness. Comparison scores are published results where available and our own runs otherwise.

Table 8. Knowledge-work and coding results.

Ultramarine improves on GLM-5.3 Flash on all six knowledge-work benchmarks, by 4.3 points on average, while generating 12% fewer tokens across them. Coding remains stable without regressing: both coding benchmarks stay within noise of the base model, at 6% more tokens.

On PI Knowledge Work, Ultramarine sits on the cost-quality frontier. It scores within three points of Claude Opus 5.5 and GPT-6 Sol at about a tenth of GPT-6 Sol's cost per task and a fiftieth of Opus's. Of the 33 other runs across twelve models and their reasoning settings, 21 cost more and score lower.

1. [1]

   Ivison, H., et al. (2026). TMax: A Simple Recipe for Terminal Agents. arXiv:2606.23321.

   ↩ 1↩ 2
2. [2]

   Wang, Z., et al. (2026). Agent World Model: Infinity Synthetic Environments for Agentic Reinforcement Learning. arXiv:2602.10090.

   ↩
3. [3]

   Zeng, Y., et al. (2026). EnvCraft: Synthesizing Executable Environments in Agentic RL for Claw-like Agent. arXiv:2609.05576.

   ↩
4. [4]

   Zuo, Y., et al. (2026). Qwen-AgentWorld: Language World Models for General Agents. arXiv:2606.24597.

   ↩
5. [5]

   Ramrakhya, R., et al. (2025). Scaling Synthetic Task Generation for Agents via Exploration. arXiv:2509.25047.

   ↩
6. [6]

   Li, Z., et al. (2026). Recursive Synthesis for Long-Horizon Terminal Tasks. arXiv:2608.05466.

   ↩
7. [7]

   Li, J., et al. (2026). Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Application. arXiv:2606.12191.

   ↩
8. [8]

   DeepSeek-AI (2026). DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression. Technical report.

   ↩
9. [9]

   LLM-Core, Xiaomi (2026). MiMo-V2.6: Scaling Reinforcement Learning Towards Self-Improvement. Technical report.

   ↩ 1↩ 2
10. [10]

    Harbor Framework Team (2026). Harbor: A framework for evaluating and optimizing agents and models in container environments. GitHub. Task specification.

    ↩ 1↩ 2
11. [11]

    Gunjal, A., et al. (2025). Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains. arXiv:2507.17746.

    ↩
12. [12]

    Khazi, D., et al. (2026). Notes on Choosing a Rubric Judge. Paper Instruments Blog.

    ↩ 1↩ 2↩ 3
13. [13]

    Epoch AI (2026, September 17). Introducing Benchmark Reviews. X post.

    ↩
14. [14]

    OpenAI (2026). Separating signal from noise in coding evaluations. OpenAI.

    ↩
15. [15]

    Yu, Y., et al. (2023). Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Bias. arXiv:2306.15895.

    ↩
16. [16]

    Paper Instruments (n.d.). Rubric: A Python library for LLM-based evaluation using weighted rubrics. GitHub.

    ↩
17. [17]

    NVIDIA (2026). Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning. arXiv:2606.15007.

    ↩
18. [18]

    Anthropic (n.d.). Claude Code overview. Official documentation.

    ↩
19. [19]

    OpenAI (n.d.). Codex. GitHub.

    ↩
20. [20]

    Cursor (n.d.). Cursor. Official issue-tracker repository. Harness engineering description.

    ↩
21. [21]

    Moonshot AI (n.d.). Kimi Code CLI. GitHub. Earlier Python implementation.

    ↩
22. [22]

    OpenAI (n.d.). OpenAI Agents SDK for Python. GitHub.

    ↩
23. [23]

    Anomaly (n.d.). OpenCode. GitHub.

    ↩
24. [24]

    OpenHands (n.d.). OpenHands. GitHub.

    ↩
25. [25]

    OpenHands (n.d.). OpenHands Software Agent SDK. GitHub.

    ↩
26. [26]

    Qwen (n.d.). Qwen Code. GitHub.

    ↩
27. [27]

    SWE-agent (n.d.). mini-SWE-agent. GitHub.

    ↩
28. [28]

    Terminal-Bench contributors (n.d.). Terminus. GitHub. Agent description.

    ↩
29. [29]

    SWE-agent (n.d.). SWE-agent. GitHub.

    ↩
30. [30]

    Brown, W. (2025). Verifiers: Environments for LLM Reinforcement Learning. GitHub.

    ↩
31. [31]

    Piché, A., et al. (2025). PipelineRL: Faster On-policy Reinforcement Learning for Long Sequence Generation. arXiv:2509.19128.

    ↩ 1↩ 2↩ 3
32. [32]

    Paper Instruments (n.d.). Paper Office. Paper Instruments.

    ↩
33. [33]

    E2B (n.d.). Sandbox persistence. Documentation.

    ↩
34. [34]

    Martin, L., Cemaj, G., & Cohen, M. (2026). Scaling Managed Agents: Decoupling the brain from the hands. Anthropic Engineering.

    ↩
35. [35]

    Paper Instruments (2026). Kos-1 Lite: SOTA Medical Model. Paper Instruments Blog.

    ↩
36. [36]

    Fang, Q., et al. (2026). Kos-1 Experimental: Env-Free RL on a 1T Parameter Agentic Prior. Paper Instruments Blog.

    ↩
37. [37]

    MiniMax (2025). MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention. arXiv:2506.13585.

    ↩ 1↩ 2
38. [38]

    Ma, W., et al. (2025). Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routers. arXiv:2510.11370.

    ↩
39. [39]

    Liu, Z., et al. (2025). Understanding R1-Zero-Like Training: A Critical Perspective. arXiv:2503.20783.

    ↩ 1↩ 2
40. [40]

    Yu, Q., et al. (2025). DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv:2503.14476.

    ↩
41. [41]

    Hilton, J., Cobbe, K., & Schulman, J. (2021). Batch Size-Invariance for Policy Optimization. arXiv:2110.00641.

    ↩
42. [42]

    Fang, Q., & Khazi, D. (2025). Mismatch Praxis: Rollout Settings and IS Corrections. Paper Instruments Blog.

    ↩
43. [43]

    Mistral-AI (2025). Magistral. arXiv:2506.10910.

    ↩
44. [44]

    Z.ai (2026). GLM-5.3-Flash. Model card.

    ↩
45. [45]

    Hu, E. J., et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685.

    ↩