The 0.89-confidence suggestion was wrong
Loomaly reports four production lessons from Jev audits and internal-link suggestions. This review adds a downloadable offline preflight that turns headings, existing links and uncaptured CTAs into repeatable counterexamples.
What changed after moving to Jev
per page; on one day, 163 of 190 audits timed out.
- Repeated runs produced different issue lists
- It reported issues unsupported by the supplied evidence
- Cost limited each run to 25 pages
per page at about $0.00003, making full-page coverage practical.
- 150 of 153 decisions matched across two runs
- A Jev run over the Pop Mart site cost about one cent
- Internal-link acceptance rose from 14% to 80%
These figures come from Loomaly design documents, calibration logs, and production records from September 2026. They are not universal performance claims for Jev.
High confidence is not correctness
Loomaly asked Jev whether a sentence should receive an internal link, then joined the answers with real accept and reject outcomes. Mean confidence barely separated the two groups.
“What are the alternatives to Google Analytics?”
The target article was relevant, but the source sentence was an H2. The model judged semantic relevance without the editorial rule that headings should not become inline links.
In the same reconciliation, the anchor-text Score separated outcomes better: 1.63 for accepted suggestions versus 1.39 for rejected suggestions. Loomaly therefore shows only suggestions above 1.4 in its free check. This does not mean Score always beats confidence; it means keeping the signal that actually predicts outcomes in your workflow.
It can judge only the evidence you provide
On the first Pop Mart audit, Jev marked 168 pages as thin and 56 live product pages as lacking a next step. The conclusions sounded plausible, but the input was incomplete: body content loaded through JavaScript and the crawl was nearly blank; button data was never supplied.
Decide whether the question is valid
The article uses 150 characters as a collection safeguard for nearly empty crawls. That is Loomaly’s engineering choice, not a content-quality or AdSense threshold. Our lab below checks evidence completeness without labeling short content low value. When buttons were not collected, CTA quality is not evaluable.
Judge the remaining semantic question
Only after evidence passes deterministic prerequisites should Jev judge specificity, CTA clarity, or intent fit.
The open-source Jev SEO project uses the same boundary: code owns crawling, rules, and scoring, while Jev handles semantic judgments such as page type, intent, helpfulness, and specificity.
Question design changes the answer
Avoid negative questions
“Is this page missing author information?” increased directional errors. Loomaly blocks negative phrasing before the call and asks “Does this page have author information?” instead.
Translate numbers in code first
Calculate page length, link counts, and other numeric facts in code, convert them into explicit categories, and pass the semantic state to Jev.
Judge one page at a time
Do not pack many page rows into one cross-row ranking decision. The Loomaly article reports an external test dropping from 0.97 to 0.51 but does not link the primary benchmark, so this site preserves it as an attributed observation rather than an independently verified result.
On launch, 40 of 50 suggestions already existed
Internal-link acceptance improved from 14% to 78% in testing. Yet among the first 50 production suggestions, 40 already existed on the customer site. The test script and production system read existing links differently, so offline evaluation never exposed the gap.
The largest gain did not come from another model tweak. Code rejected candidates that should never appear: existing links, previously rejected suggestions, image captions, table-of-contents items, headings, and anchors cut from the middle of a phrase. Image captions alone accounted for 14 of 26 rejections in one round.
Turn the case into a regression: audit evidence before content
The checker and synthetic inputs below were written by this site. They are not Loomaly source, production logs or a reproduction of its acceptance rate. They exercise collection and publication rules: headings and image alt text are not paragraph anchors; existing or rejected targets stay excluded; an uncaptured CTA cannot be called absent. Semantic relevance still requires editorial or model review.
- The crawler supplies DOM region, paragraph IDs, exact text offsets, existing inline-link ranges and capture status. The parser must produce kind/region; a model must not relabel an H2 as a paragraph. This example consumes structured evidence; it does not crawl or parse HTML.
- Missing or pending evidence returns not_evaluable. Seventy captured characters do not automatically mean thin content: length is diagnostic, not a quality rule. A complete empty CTA list differs from a CTA that was never captured.
- Only main-body paragraphs qualify. Exact offsets recover the anchor from source text. Existing targets, recorded rejections, linked text and duplicate or overlapping candidates are excluded. Latin mid-word cuts are rejected; Chinese phrase and grammatical boundaries still need editorial review.
| Synthetic case | Checker output |
|---|---|
| valid-paragraph | eligible_for_semantic_reviewdeterministic_checks_passed |
| heading | rejectednot_a_main_body_paragraph |
| alt-text | rejectednot_a_main_body_paragraph |
| existing-target | rejectedtarget_already_linked |
| previous-rejection | rejectedpreviously_rejected_target |
| repeat-target | rejectedduplicate_candidate_target |
| CTA never captured | not_evaluablecta_not_collected_or_incomplete |
eligible_for_semantic_review only means deterministic preflight passed. It neither establishes a correct recommendation nor authorizes insertion or proves target availability. URL comparison removes fragments and trailing slashes; it does not fetch redirects or infer canonicals, and query parameters remain. Share the collector and rules between test and production, then evaluate semantics against real editorial outcomes.
Download and run offline
node seo_input_audit.mjs
node --test seo_input_audit.test.mjs Download the complete checker ↓ · Download counterexample tests ↓
Use Node 20+. No dependencies, keys or network are needed; place both files together. To inspect your own collected evidence, save a {page, candidates} JSON file and run node seo_input_audit.mjs input.json. Missing records are never interpreted as no issue.
Inspect the program output for the synthetic fixture
{
"synthetic_fixture": true,
"content": {
"status": "ready_for_semantic_review",
"bodyCharacters": 70
},
"cta": {
"status": "not_evaluable",
"reason": "cta_not_collected_or_incomplete"
},
"decisions": [
{
"id": "valid-paragraph",
"status": "eligible_for_semantic_review",
"reason": "deterministic_checks_passed",
"anchor": "Google Analytics alternatives",
"targetUrl": "https://example.test/alternatives"
},
{
"id": "heading",
"status": "rejected",
"reason": "not_a_main_body_paragraph"
},
{
"id": "alt-text",
"status": "rejected",
"reason": "not_a_main_body_paragraph"
},
{
"id": "existing-target",
"status": "rejected",
"reason": "target_already_linked"
},
{
"id": "previous-rejection",
"status": "rejected",
"reason": "previously_rejected_target"
},
{
"id": "repeat-target",
"status": "rejected",
"reason": "duplicate_candidate_target"
}
],
"action": "review_candidates_only_no_page_changes"
}Inspect the complete shared implementation
/**
* Offline preflight for a crawler's structured evidence, not an HTML crawler.
* No network/model call, quality score, link insertion or page mutation.
*/
import { readFileSync } from 'node:fs';
import { pathToFileURL } from 'node:url';
function normalUrl(value, base) {
if (typeof value !== 'string' || !value.trim()) throw new Error('Missing URL');
const url = new URL(value, base);
if (!['http:', 'https:'].includes(url.protocol) || url.username || url.password) throw new Error('Unsupported URL');
url.hash = '';
if (url.pathname !== '/') url.pathname = url.pathname.replace(/\/+$/, '');
return url;
}
const isComplete = capture => capture?.status === 'complete';
const isBodyParagraph = block => block?.kind === 'paragraph' && block?.region === 'main-body';
const hasText = value => typeof value === 'string' && value.trim().length > 0;
const overlaps = (a, b) => a.start < b.end && b.start < a.end;
export function auditInput(page, candidates = []) {
const source = normalUrl(page.url);
const blocks = Array.isArray(page.bodyCapture?.blocks) ? page.bodyCapture.blocks : [];
const body = blocks.filter(isBodyParagraph).filter(block => hasText(block.text));
// Length is diagnostic only. A short page is not automatically thin content.
const bodyCharacters = body.map(block => block.text).join('\n').length;
const contentReady = isComplete(page.bodyCapture) && body.length > 0;
const content = contentReady
? { status: 'ready_for_semantic_review', bodyCharacters }
: { status: 'not_evaluable', reason: 'body_evidence_missing_or_incomplete', bodyCharacters };
const cta = !contentReady
? { status: 'not_evaluable', reason: 'body_context_missing_or_incomplete' }
: isComplete(page.ctaCapture) && Array.isArray(page.ctaCapture.items)
? { status: 'ready_for_semantic_review', capturedItems: page.ctaCapture.items.length }
: { status: 'not_evaluable', reason: 'cta_not_collected_or_incomplete' };
const linkEvidenceReady = isComplete(page.linkCapture) && Array.isArray(page.linkCapture.targets);
const rejectionEvidenceReady = isComplete(page.rejectionCapture) && Array.isArray(page.rejectionCapture.items);
const existing = new Set();
let malformedEvidence = !linkEvidenceReady || !rejectionEvidenceReady;
for (const href of Array.isArray(page.linkCapture?.targets) ? page.linkCapture.targets : []) {
try { existing.add(normalUrl(href, source).href); } catch { malformedEvidence = true; }
}
const rejected = new Set();
for (const item of Array.isArray(page.rejectionCapture?.items) ? page.rejectionCapture.items : []) {
try {
if (normalUrl(item.sourceUrl, source).href === source.href) rejected.add(normalUrl(item.targetUrl, source).href);
} catch { malformedEvidence = true; }
}
const selected = [];
const decisions = candidates.map(candidate => {
const result = (status, reason, extra = {}) => ({ id: candidate.id, status, reason, ...extra });
if (!contentReady) return result('not_evaluable', 'body_evidence_missing_or_incomplete');
if (!linkEvidenceReady || !rejectionEvidenceReady || malformedEvidence) {
return result('not_evaluable', 'link_or_rejection_evidence_incomplete');
}
const matching = blocks.filter(item => item.id === candidate.blockId);
if (matching.length > 1) return result('not_evaluable', 'ambiguous_block_identity');
if (matching.length === 0) return result('not_evaluable', 'source_block_not_found');
const block = matching[0];
if (!isBodyParagraph(block)) return result('rejected', 'not_a_main_body_paragraph');
if (!hasText(block.text)) return result('not_evaluable', 'paragraph_text_missing');
const { start, end } = candidate;
if (!Number.isInteger(start) || !Number.isInteger(end) || start < 0 || end <= start || end > block.text.length) {
return result('rejected', 'invalid_anchor_offsets');
}
const anchor = block.text.slice(start, end);
if (!hasText(anchor) || anchor !== anchor.trim()) return result('rejected', 'invalid_anchor_text');
// This catches Latin mid-word cuts only, not grammatical phrase boundaries.
const word = /[A-Za-z0-9_]/;
if ((start > 0 && word.test(block.text[start - 1]) && word.test(anchor[0]))
|| (end < block.text.length && word.test(block.text[end]) && word.test(anchor.at(-1)))) {
return result('rejected', 'anchor_cuts_a_word');
}
let target;
try { target = normalUrl(candidate.targetUrl, source); }
catch { return result('rejected', 'invalid_target_url'); }
if (target.origin !== source.origin) return result('rejected', 'external_target');
if (target.href === source.href) return result('rejected', 'self_link');
if (existing.has(target.href)) return result('rejected', 'target_already_linked');
if (rejected.has(target.href)) return result('rejected', 'previously_rejected_target');
if (!Array.isArray(block.linkSpans)) return result('not_evaluable', 'inline_link_ranges_not_collected');
if (block.linkSpans.some(span => !Number.isInteger(span.start) || !Number.isInteger(span.end)
|| span.start < 0 || span.end <= span.start || span.end > block.text.length)) {
return result('not_evaluable', 'invalid_inline_link_ranges');
}
if (block.linkSpans.some(span => overlaps(candidate, span))) return result('rejected', 'anchor_already_linked');
if (selected.some(item => item.target === target.href)) return result('rejected', 'duplicate_candidate_target');
if (selected.some(item => item.blockId === block.id && overlaps(candidate, item))) return result('rejected', 'overlapping_candidate');
selected.push({ blockId: block.id, start, end, target: target.href });
return result('eligible_for_semantic_review', 'deterministic_checks_passed', { anchor, targetUrl: target.href });
});
return { content, cta, decisions, action: 'review_candidates_only_no_page_changes' };
}
export function syntheticFixture() {
const text = 'Compare Google Analytics alternatives and review privacy requirements.';
const page = {
url: 'https://example.test/analytics',
bodyCapture: { status: 'complete', blocks: [
{ id: 'p1', kind: 'paragraph', region: 'main-body', text, linkSpans: [] },
{ id: 'h2', kind: 'heading', region: 'main-body', text: 'Google Analytics alternatives' },
{ id: 'alt', kind: 'image_alt', region: 'main-body', text: 'Google Analytics alternatives' },
] },
ctaCapture: { status: 'not_collected' },
linkCapture: { status: 'complete', targets: ['/already-linked'] },
rejectionCapture: { status: 'complete', items: [{ sourceUrl: '/analytics', targetUrl: '/rejected-before' }] },
};
const candidate = { blockId: 'p1', start: 8, end: 37 };
return { page, candidates: [
{ id: 'valid-paragraph', ...candidate, targetUrl: '/alternatives' },
{ id: 'heading', blockId: 'h2', start: 0, end: 29, targetUrl: '/alternatives' },
{ id: 'alt-text', blockId: 'alt', start: 0, end: 29, targetUrl: '/alternatives' },
{ id: 'existing-target', ...candidate, targetUrl: '/already-linked#details' },
{ id: 'previous-rejection', ...candidate, targetUrl: '/rejected-before' },
{ id: 'repeat-target', ...candidate, targetUrl: '/alternatives/' },
] };
}
if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
const file = process.argv[2];
const input = file ? JSON.parse(readFileSync(file, 'utf8')) : syntheticFixture();
console.log(JSON.stringify({ synthetic_fixture: !file, ...auditInput(input.page, input.candidates) }, null, 2));
}
If you want to use Jev in production
- Decide who owns the final action.Put countable facts in code and reserve Jev for semantic questions code cannot resolve.
- Calibrate on real outcomes.Do not worship confidence; find the signal that separates outcomes in your workflow.
- Give code veto power.Deletion, publication, charges, and other irreversible actions never gain authority from a high model score.
- Inspect what the model actually saw.When the input is empty or fields are missing, stop the question instead of accepting a polished but unsupported answer.