Independent relevance is not a mutually exclusive choice
One tool’s Noul score need not decrease when another tool scores highly. The local top-K policy imposes the actual menu-size constraint.
Inspect the supporting source ↗REPOSITORY ANALYSIS
jev-toolspace
A research repository evaluating Jev relevance judgments for task-level tool menus using controlled TGB and BFCL protocols.
E2 · source reviewedPublic material or source evidence was reviewed. A fixed-version source review requires checking a pinned revision; this is not a reproduction, performance endorsement, or safety guarantee.
- Stars
- 1
- Forks
- 0
- License
- —
- Last commit
- 2026-09-23
What Jev does here
Independent Noul questions score each tool against shared task evidence; local aggregation and top-K rules produce menu masks for downstream evaluation.
Separates relevance scoring from downstream tool-use success and records model/state/question identities for cache reuse.
Fixed-version documentation and implementation material were inspected on 2026-10-24. No GPU benchmark or live scoring was run. Published sample results and historical service limits were not revalidated as current platform guarantees.
What task does this project fit?
Evaluate task-level tool menus rather than ask Jev to execute a tool.
Material analysed：
From input to action
- Benchmark traces or conversations define the relevance state.
- Each tool receives an independent Noul question; code aggregates scores and chooses top-K documentation budgets.
- TGB/BFCL evaluators measure downstream behavior under the resulting menu masks.
Implementation details that matter
The cache key includes the questions
client.py hashes model, state and questions, allowing exact request reuse. Prompt changes invalidate that key; the shipped mock is deterministic synthetic output, not model evaluation.
Inspect the supporting source ↗When to choose it
Useful for controlled tool-menu research. Keep build/test splits and compare downstream tool-use results; high relevance scores alone do not demonstrate task completion.
How to inspect it yourself
Read the README’s protocol and client.py _key/ask methods. Confirm that changing a question changes the cache identity and that JEV_MOCK records model=MOCK rather than real usage.
Limits of this review
- No GPU benchmark or live scoring was run.
- Published sample results and historical service limits were not revalidated as current platform guarantees.