Jev confidence thresholds and human review
Design risk-aware Jev confidence thresholds, calibration tests, and human-review bands, informed by a production internal-link experiment where confidence did not separate accepted from rejected suggestions.
1. Confidence is not probability
For Choice and Score, confidence summarizes how decisive the full distribution is. A winning option can have the highest probability while the distribution is still ambiguous.
2. Use risk bands
Define three outcomes: automate, request review, or abstain. The boundary should be stricter for refunds, account changes, deletion, and other irreversible actions than for reversible routing suggestions.
3. Calibrate with your own data
Collect representative labeled cases, run the exact production questions, group results into confidence bands, and compare observed error rates. Recheck after changing models, criteria, or input structure.
4. Log the decision contract
Store question version, model version, selected answer, distribution, confidence, threshold, and final action. This makes regressions and review overrides measurable.
5. Third-party case: validate confidence against outcomes
In Loomaly's reported internal-link suggestion workflow, the mean confidence of accepted and rejected recommendations differed by only 0.004. The highest-confidence recommendation, at 0.89, was rejected as incorrect. We have not independently reproduced these results. Similar group means alone do not establish that confidence has no predictive value; sample sizes and distributions are needed. Use Jev to propose or rank links; let deterministic checks enforce URL validity, duplication, anchor rules, and other irreversible publishing decisions.
Work through a complete example and its failure paths
The complete file below has no placeholder functions. It prints a suggestion or review state; it does not update tickets, spend money, or publish content.
Verification: pinned-package type/syntax checks and mock transports, not live API performance or accuracy tests. Real SDK runs require your own server-side key and may incur charges. Calibration and input examples run offline.
Define correctness before choosing a threshold
The target is agreement with an independent team label, not whether somebody clicked Accept. Record prediction, label, confidence, and model/question versions. Split by time or customer to keep near-duplicate tickets out of both sets.
The following data is entirely synthetic. Coverage is proposed/total; selective error is errors/proposed. With no proposals, error rate is undefined, not zero. Cost combines assumed mistake and review costs.
Choose thresholds on development data, then evaluate once on held-out data. A failure calls for revised inputs/questions and another evaluation round, not repeated tuning to the holdout. Real reports also need sample counts, subgroup results and uncertainty intervals.
calibrate.py
"""Synthetic teaching fixture: NOT measured Jev accuracy or a recommended threshold."""
# (model confidence, predicted team, independently assigned correct team)
development = [(0.95, 'technical', 'technical'), (0.89, 'billing', 'technical'),
(0.8, 'billing', 'billing'), (0.7, 'technical', 'technical'),
(0.6, 'billing', 'technical'), (0.4, 'human', 'human')]
held_out = [(0.92, 'technical', 'billing'), (0.83, 'billing', 'billing'),
(0.55, 'technical', 'technical'), (0.3, 'billing', 'technical')]
def evaluate(rows, threshold, error_cost=10, review_cost=1):
proposed = [r for r in rows if r[0] >= threshold and r[1] != 'human']
errors = sum(predicted != label for _, predicted, label in proposed)
reviewed = len(rows) - len(proposed)
return {'threshold': threshold, 'n': len(rows), 'proposed': len(proposed), 'errors': errors,
'coverage': len(proposed) / len(rows) if rows else 0,
'selective_error': errors / len(proposed) if proposed else None,
'review_rate': reviewed / len(rows) if rows else 0,
'cost': errors * error_cost + reviewed * review_cost}
if __name__ == '__main__':
results = [evaluate(development, t) for t in [0.5, 0.7, 0.8, 0.9, 1.0]]
for row in results:
print('development', row)
chosen = min(results, key=lambda r: (r['cost'], -r['threshold']))['threshold']
print('held-out (do not retune on this set)', evaluate(held_out, chosen))
print('Synthetic illustration only. Repeat with labeled outcomes and uncertainty estimates.')
Run
python calibrate.pyHow to verify the run
API examples print structured fields such as action and team. Labels and scores may differ from mocks. Removing the key should produce review without exposing raw error text. The calibration script prints development and held-out metrics, not model-quality evidence.
Read the failure in the threshold exercise
The downloadable calibrate.py contains just six development cases and four held-out cases—all synthetic, not Jev measurements. With illustrative costs of 10 per error and 1 per human review, the development set selects 0.9: one of six cases is suggested automatically, with zero errors in that one. On the untouched held-out set, the same threshold suggests one of four cases and that one is wrong. A development-set optimum on tiny data is not a production policy.
Next, collect real labels, freeze the development/held-out split, and report coverage, error among automated suggestions, human review volume, and business loss together. If no cases are suggested automatically, the selective error is undefined—not 0%. Re-evaluate when questions or state shape change.
Download the synthetic calibration script ↓