Separate Agent decisions from LLM generation

Separate generation, judgment, and execution

General-purpose models excel at planning, explanation, generation, and multi-step reasoning. An Agent also contains narrower questions: which tool to call, whether a search result matters, how severe a risk is, or whether a person should intervene. Their shared property is not simplicity; it is that the output space can be defined in advance.

Jev's engineering value is not merely returning JSON. Structured output can do that too. Jev constrains the task itself to Choice, Score, or Noul, so the application receives a probabilistic decision rather than prose that needs another parsing step.

Which tasks belong in a fast decision layer

A task fits Jev when its options can be enumerated, calls are frequent, mistakes are recoverable, and semantic understanding is needed without a long explanation. The more conditions it satisfies, the stronger the case for separating it from generation.

Jev belongs less in the Agent’s brain than in its reflexes.

Case study: context compaction without summaries

Coding Agent context is often dominated not by user messages but by file contents, search results, command logs, and directory trees. Summaries save tokens, but may rewrite paths, error codes, stack locations, and command arguments into approximate prose.

fast-jev-compaction takes another route: preserve user and assistant text verbatim, and make retention decisions only for paired tool calls and results. Important evidence remains intact, secondary results are truncated, and irrelevant calls are removed.

From message history to three actions

1. 01
   collectToolCalls

   Pair calls with results by tool_use_id; pin the first and recent messages.

2. 02
   fitState

   Build only the state needed for judgment instead of resending every result.

3. 03
   batchCalls

   Split questions into batches and query Jev concurrently.

4. 04
   decideCall

   Combine two probabilities into keep, drop_result, or drop_call.

Split one compound question into two Noul judgments

questions = {
  call_t3: {
    type: 'noul',
    instructions: 'Is knowing this tool call still useful?'
  },
  result_t3: {
    type: 'noul',
    instructions: 'Must the full result remain available?'
  }
}

State budgets and progressive degradation

When state exceeds its budget, do not delete messages arbitrarily. Shorten inputs first, then collapse old text, and only then omit old messages or merge old calls. Every degradation stage should be observable.

Thresholds, fallbacks, and rollout

Thresholds such as 0.5 or 0.8 illustrate control flow; they are not production defaults. Confidence is not a safety proof. Calibrate thresholds against your labeled data, error costs, and action reversibility.

• Start in shadow mode without changing real behavior.
• Measure observed accuracy by probability band.
• Fall back on request failure, malformed output, or insufficient savings.
• Log model version, question version, threshold, action, and overrides.

Jev vs structured output vs classifiers

Type safety means the model stays inside the agreed output shape; it does not mean the selected answer is always correct.

A reusable Agent engineering pattern

fast-jev-compaction matters beyond compaction. It demonstrates a stable decision protocol: structure objects first, split judgments into atomic questions, combine probabilities in code, keep side effects under deterministic policy, and preserve a complete fallback.

The same pattern applies to tool routing, moderation, search reranking, permission requests, and quality gates. The reusable asset is not an example threshold; it is the boundary where models remain replaceable and the decision protocol stays stable.