Jev production reliability checklist

1. Version the decision contract

Record the model, state schema, question IDs, criteria, and threshold policy together. Changing any one of them can change behavior and should trigger a new evaluation run.

const decisionContract = {
  model: 'typesafe-ai/jev',
  stateSchema: 'support-ticket@3',
  questions: 'routing@5',
  policy: 'confidence-bands@2',
};

2. Separate failures from uncertainty

Timeouts, rate limits, overload, and invalid responses need retry or fallback handling. Low confidence is not a transport error; it is a valid signal that should follow an explicit review policy.

try {
  const result = await callJev({ timeoutMs: 2_000 });
  return applyConfidencePolicy(result);
} catch (error) {
  if (isRetryable(error)) return retryWithBackoff();
  return enqueueHumanReview({ reason: 'provider_failure' });
}

3. Log enough to reproduce decisions

Store request IDs, normalized state hashes, question versions, answers, distributions, confidence, latency, and the final application action without leaking sensitive content.

4. Evaluate and roll out gradually

Run labeled offline tests, shadow production traffic, compare human overrides, and use a kill switch. Expand automation only when the observed error cost matches the policy.