PRODUCTION CASE STUDY · JEV × SEO

The 0.89-confidence suggestion was wrong

Loomaly reports four production lessons from Jev audits and internal-link suggestions. This review adds a downloadable offline preflight that turns headings, existing links and uncaptured CTAs into repeatable counterexamples.

Published 2026-09-26Updated 2026-10-24Loomaly production notes + our offline lab
00 / BASELINE

What changed after moving to Jev

BEFORE · GENERATIVE AUDIT 53–170s

per page; on one day, 163 of 190 audits timed out.

  • Repeated runs produced different issue lists
  • It reported issues unsupported by the supplied evidence
  • Cost limited each run to 25 pages
AFTER · JEV DECISION LAYER 1–2s

per page at about $0.00003, making full-page coverage practical.

  • 150 of 153 decisions matched across two runs
  • A Jev run over the Pop Mart site cost about one cent
  • Internal-link acceptance rose from 14% to 80%

These figures come from Loomaly design documents, calibration logs, and production records from September 2026. They are not universal performance claims for Jev.

01
CONFIDENCE ≠ CORRECTNESS

High confidence is not correctness

Loomaly asked Jev whether a sentence should receive an internal link, then joined the answers with real accept and reject outcomes. Mean confidence barely separated the two groups.

ACCEPTED0.619mean confidence
REJECTED0.623mean confidence
HIGHEST CONFIDENCE0.89but the suggestion was wrong
FAILED CASE
“What are the alternatives to Google Analytics?”

The target article was relevant, but the source sentence was an H2. The model judged semantic relevance without the editorial rule that headings should not become inline links.

In the same reconciliation, the anchor-text Score separated outcomes better: 1.63 for accepted suggestions versus 1.39 for rejected suggestions. Loomaly therefore shows only suggestions above 1.4 in its free check. This does not mean Score always beats confidence; it means keeping the signal that actually predicts outcomes in your workflow.

02
INPUT BOUNDARY

It can judge only the evidence you provide

On the first Pop Mart audit, Jev marked 168 pages as thin and 56 live product pages as lacking a next step. The conclusions sounded plausible, but the input was incomplete: body content loaded through JavaScript and the crawl was nearly blank; button data was never supplied.

CODE

Decide whether the question is valid

The article uses 150 characters as a collection safeguard for nearly empty crawls. That is Loomaly’s engineering choice, not a content-quality or AdSense threshold. Our lab below checks evidence completeness without labeling short content low value. When buttons were not collected, CTA quality is not evaluable.

JEV

Judge the remaining semantic question

Only after evidence passes deterministic prerequisites should Jev judge specificity, CTA clarity, or intent fit.

The open-source Jev SEO project uses the same boundary: code owns crawling, rules, and scoring, while Jev handles semantic judgments such as page type, intent, helpfulness, and specificity.

03
QUESTION DESIGN

Question design changes the answer

NEGATION

Avoid negative questions

“Is this page missing author information?” increased directional errors. Loomaly blocks negative phrasing before the call and asks “Does this page have author information?” instead.

NUMBERS

Translate numbers in code first

Calculate page length, link counts, and other numeric facts in code, convert them into explicit categories, and pass the semantic state to Jev.

SCOPE

Judge one page at a time

Do not pack many page rows into one cross-row ranking decision. The Loomaly article reports an external test dropping from 0.97 to 0.51 but does not link the primary benchmark, so this site preserves it as an attributed observation rather than an independently verified result.

04
PRODUCTION PARITY

On launch, 40 of 50 suggestions already existed

Internal-link acceptance improved from 14% to 78% in testing. Yet among the first 50 production suggestions, 40 already existed on the customer site. The test script and production system read existing links differently, so offline evaluation never exposed the gap.

TEST14% → 78%iterative suggestion tuning
→
FIRST RELEASE40 / 50duplicates; the batch was discarded
→
AFTER FIX80%production acceptance

The largest gain did not come from another model tweak. Code rejected candidates that should never appear: existing links, previously rejected suggestions, image captions, table-of-contents items, headings, and anchors cut from the middle of a phrase. Image captions alone accounted for 14 of 26 rejections in one round.

SITE-AUTHORED LAB · NO MODEL CALL

Turn the case into a regression: audit evidence before content

The checker and synthetic inputs below were written by this site. They are not Loomaly source, production logs or a reproduction of its acceptance rate. They exercise collection and publication rules: headings and image alt text are not paragraph anchors; existing or rejected targets stay excluded; an uncaptured CTA cannot be called absent. Semantic relevance still requires editorial or model review.

  1. The crawler supplies DOM region, paragraph IDs, exact text offsets, existing inline-link ranges and capture status. The parser must produce kind/region; a model must not relabel an H2 as a paragraph. This example consumes structured evidence; it does not crawl or parse HTML.
  2. Missing or pending evidence returns not_evaluable. Seventy captured characters do not automatically mean thin content: length is diagnostic, not a quality rule. A complete empty CTA list differs from a CTA that was never captured.
  3. Only main-body paragraphs qualify. Exact offsets recover the anchor from source text. Existing targets, recorded rejections, linked text and duplicate or overlapping candidates are excluded. Latin mid-word cuts are rejected; Chinese phrase and grammatical boundaries still need editorial review.
Synthetic caseChecker output
valid-paragrapheligible_for_semantic_review

deterministic_checks_passed

headingrejected

not_a_main_body_paragraph

alt-textrejected

not_a_main_body_paragraph

existing-targetrejected

target_already_linked

previous-rejectionrejected

previously_rejected_target

repeat-targetrejected

duplicate_candidate_target

CTA never capturednot_evaluable

cta_not_collected_or_incomplete

eligible_for_semantic_review only means deterministic preflight passed. It neither establishes a correct recommendation nor authorizes insertion or proves target availability. URL comparison removes fragments and trailing slashes; it does not fetch redirects or infer canonicals, and query parameters remain. Share the collector and rules between test and production, then evaluate semantics against real editorial outcomes.

Download and run offline

node seo_input_audit.mjs
node --test seo_input_audit.test.mjs

Download the complete checker ↓ · Download counterexample tests ↓

Use Node 20+. No dependencies, keys or network are needed; place both files together. To inspect your own collected evidence, save a {page, candidates} JSON file and run node seo_input_audit.mjs input.json. Missing records are never interpreted as no issue.

Inspect the program output for the synthetic fixture
{
  "synthetic_fixture": true,
  "content": {
    "status": "ready_for_semantic_review",
    "bodyCharacters": 70
  },
  "cta": {
    "status": "not_evaluable",
    "reason": "cta_not_collected_or_incomplete"
  },
  "decisions": [
    {
      "id": "valid-paragraph",
      "status": "eligible_for_semantic_review",
      "reason": "deterministic_checks_passed",
      "anchor": "Google Analytics alternatives",
      "targetUrl": "https://example.test/alternatives"
    },
    {
      "id": "heading",
      "status": "rejected",
      "reason": "not_a_main_body_paragraph"
    },
    {
      "id": "alt-text",
      "status": "rejected",
      "reason": "not_a_main_body_paragraph"
    },
    {
      "id": "existing-target",
      "status": "rejected",
      "reason": "target_already_linked"
    },
    {
      "id": "previous-rejection",
      "status": "rejected",
      "reason": "previously_rejected_target"
    },
    {
      "id": "repeat-target",
      "status": "rejected",
      "reason": "duplicate_candidate_target"
    }
  ],
  "action": "review_candidates_only_no_page_changes"
}
Inspect the complete shared implementation
/**
 * Offline preflight for a crawler's structured evidence, not an HTML crawler.
 * No network/model call, quality score, link insertion or page mutation.
 */
import { readFileSync } from 'node:fs';
import { pathToFileURL } from 'node:url';

function normalUrl(value, base) {
  if (typeof value !== 'string' || !value.trim()) throw new Error('Missing URL');
  const url = new URL(value, base);
  if (!['http:', 'https:'].includes(url.protocol) || url.username || url.password) throw new Error('Unsupported URL');
  url.hash = '';
  if (url.pathname !== '/') url.pathname = url.pathname.replace(/\/+$/, '');
  return url;
}

const isComplete = capture => capture?.status === 'complete';
const isBodyParagraph = block => block?.kind === 'paragraph' && block?.region === 'main-body';
const hasText = value => typeof value === 'string' && value.trim().length > 0;
const overlaps = (a, b) => a.start < b.end && b.start < a.end;

export function auditInput(page, candidates = []) {
  const source = normalUrl(page.url);
  const blocks = Array.isArray(page.bodyCapture?.blocks) ? page.bodyCapture.blocks : [];
  const body = blocks.filter(isBodyParagraph).filter(block => hasText(block.text));
  // Length is diagnostic only. A short page is not automatically thin content.
  const bodyCharacters = body.map(block => block.text).join('\n').length;
  const contentReady = isComplete(page.bodyCapture) && body.length > 0;
  const content = contentReady
    ? { status: 'ready_for_semantic_review', bodyCharacters }
    : { status: 'not_evaluable', reason: 'body_evidence_missing_or_incomplete', bodyCharacters };
  const cta = !contentReady
    ? { status: 'not_evaluable', reason: 'body_context_missing_or_incomplete' }
    : isComplete(page.ctaCapture) && Array.isArray(page.ctaCapture.items)
      ? { status: 'ready_for_semantic_review', capturedItems: page.ctaCapture.items.length }
      : { status: 'not_evaluable', reason: 'cta_not_collected_or_incomplete' };
  const linkEvidenceReady = isComplete(page.linkCapture) && Array.isArray(page.linkCapture.targets);
  const rejectionEvidenceReady = isComplete(page.rejectionCapture) && Array.isArray(page.rejectionCapture.items);
  const existing = new Set();
  let malformedEvidence = !linkEvidenceReady || !rejectionEvidenceReady;
  for (const href of Array.isArray(page.linkCapture?.targets) ? page.linkCapture.targets : []) {
    try { existing.add(normalUrl(href, source).href); } catch { malformedEvidence = true; }
  }
  const rejected = new Set();
  for (const item of Array.isArray(page.rejectionCapture?.items) ? page.rejectionCapture.items : []) {
    try {
      if (normalUrl(item.sourceUrl, source).href === source.href) rejected.add(normalUrl(item.targetUrl, source).href);
    } catch { malformedEvidence = true; }
  }
  const selected = [];
  const decisions = candidates.map(candidate => {
    const result = (status, reason, extra = {}) => ({ id: candidate.id, status, reason, ...extra });
    if (!contentReady) return result('not_evaluable', 'body_evidence_missing_or_incomplete');
    if (!linkEvidenceReady || !rejectionEvidenceReady || malformedEvidence) {
      return result('not_evaluable', 'link_or_rejection_evidence_incomplete');
    }
    const matching = blocks.filter(item => item.id === candidate.blockId);
    if (matching.length > 1) return result('not_evaluable', 'ambiguous_block_identity');
    if (matching.length === 0) return result('not_evaluable', 'source_block_not_found');
    const block = matching[0];
    if (!isBodyParagraph(block)) return result('rejected', 'not_a_main_body_paragraph');
    if (!hasText(block.text)) return result('not_evaluable', 'paragraph_text_missing');
    const { start, end } = candidate;
    if (!Number.isInteger(start) || !Number.isInteger(end) || start < 0 || end <= start || end > block.text.length) {
      return result('rejected', 'invalid_anchor_offsets');
    }
    const anchor = block.text.slice(start, end);
    if (!hasText(anchor) || anchor !== anchor.trim()) return result('rejected', 'invalid_anchor_text');
    // This catches Latin mid-word cuts only, not grammatical phrase boundaries.
    const word = /[A-Za-z0-9_]/;
    if ((start > 0 && word.test(block.text[start - 1]) && word.test(anchor[0]))
      || (end < block.text.length && word.test(block.text[end]) && word.test(anchor.at(-1)))) {
      return result('rejected', 'anchor_cuts_a_word');
    }
    let target;
    try { target = normalUrl(candidate.targetUrl, source); }
    catch { return result('rejected', 'invalid_target_url'); }
    if (target.origin !== source.origin) return result('rejected', 'external_target');
    if (target.href === source.href) return result('rejected', 'self_link');
    if (existing.has(target.href)) return result('rejected', 'target_already_linked');
    if (rejected.has(target.href)) return result('rejected', 'previously_rejected_target');
    if (!Array.isArray(block.linkSpans)) return result('not_evaluable', 'inline_link_ranges_not_collected');
    if (block.linkSpans.some(span => !Number.isInteger(span.start) || !Number.isInteger(span.end)
      || span.start < 0 || span.end <= span.start || span.end > block.text.length)) {
      return result('not_evaluable', 'invalid_inline_link_ranges');
    }
    if (block.linkSpans.some(span => overlaps(candidate, span))) return result('rejected', 'anchor_already_linked');
    if (selected.some(item => item.target === target.href)) return result('rejected', 'duplicate_candidate_target');
    if (selected.some(item => item.blockId === block.id && overlaps(candidate, item))) return result('rejected', 'overlapping_candidate');
    selected.push({ blockId: block.id, start, end, target: target.href });
    return result('eligible_for_semantic_review', 'deterministic_checks_passed', { anchor, targetUrl: target.href });
  });
  return { content, cta, decisions, action: 'review_candidates_only_no_page_changes' };
}

export function syntheticFixture() {
  const text = 'Compare Google Analytics alternatives and review privacy requirements.';
  const page = {
    url: 'https://example.test/analytics',
    bodyCapture: { status: 'complete', blocks: [
      { id: 'p1', kind: 'paragraph', region: 'main-body', text, linkSpans: [] },
      { id: 'h2', kind: 'heading', region: 'main-body', text: 'Google Analytics alternatives' },
      { id: 'alt', kind: 'image_alt', region: 'main-body', text: 'Google Analytics alternatives' },
    ] },
    ctaCapture: { status: 'not_collected' },
    linkCapture: { status: 'complete', targets: ['/already-linked'] },
    rejectionCapture: { status: 'complete', items: [{ sourceUrl: '/analytics', targetUrl: '/rejected-before' }] },
  };
  const candidate = { blockId: 'p1', start: 8, end: 37 };
  return { page, candidates: [
    { id: 'valid-paragraph', ...candidate, targetUrl: '/alternatives' },
    { id: 'heading', blockId: 'h2', start: 0, end: 29, targetUrl: '/alternatives' },
    { id: 'alt-text', blockId: 'alt', start: 0, end: 29, targetUrl: '/alternatives' },
    { id: 'existing-target', ...candidate, targetUrl: '/already-linked#details' },
    { id: 'previous-rejection', ...candidate, targetUrl: '/rejected-before' },
    { id: 'repeat-target', ...candidate, targetUrl: '/alternatives/' },
  ] };
}

if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
  const file = process.argv[2];
  const input = file ? JSON.parse(readFileSync(file, 'utf8')) : syntheticFixture();
  console.log(JSON.stringify({ synthetic_fixture: !file, ...auditInput(input.page, input.candidates) }, null, 2));
}
SHIP CHECKLIST

If you want to use Jev in production

  1. Decide who owns the final action.Put countable facts in code and reserve Jev for semantic questions code cannot resolve.
  2. Calibrate on real outcomes.Do not worship confidence; find the signal that separates outcomes in your workflow.
  3. Give code veto power.Deletion, publication, charges, and other irreversible actions never gain authority from a high model score.
  4. Inspect what the model actually saw.When the input is empty or fields are missing, stop the question instead of accepting a polished but unsupported answer.

Sources and boundaries

This page is a structured editorial summary and engineering interpretation of the original Loomaly article, published by @limitlessdeng on September 25, 2026. The original attributes the production figures to Loomaly design documents, calibration logs, and run records. They remain attributed case-study data, not universal performance promises for Jev. The added checker, synthetic fixtures and regression tests are this site’s work. They validate deterministic preflight rules, not Loomaly production acceptance or model quality.