AI visibility audit methodology

v3.2 · 26 June 2026 · This page is updated whenever the method or commercial boundary changes.

Historical note: the Day-1 self-audit was a dogfooding artifact run on an older 30-prompt x 9-endpoint experimental matrix. The current production Snapshot contract is documented on this page: 5 user queries x 8 production endpoints = 40 datapoints.

1. The audit scopes and their model endpoint matrix

We hold a fixed per-product endpoint list. If we ever swap an endpoint or change a datapoint count, this page gets a new version entry and the changelog at the bottom of this document.

What this page does not promise: we do not advertise an "all major models" list, or coverage of ERNIE, MetaSo, or Perplexity Sonar as part of the current production matrix. (Earlier Day-1 dogfooding runs on /audit/self-audit-01.html used a broader 9-endpoint experimental setup; that artifact is kept for transparency but does not represent the current production Snapshot contract.)

2. The 8 current production model endpoints

The current production list is the 8 endpoints below, using each vendor's currently configured API model. All tested models are multilingual and all are measured through vendor-hosted APIs. Public-weight availability is disclosed only where it applies to specific model families or checkpoints; it does not mean Clarivy self-hosts the model, and it does not mean a vendor API model is identical to a public checkpoint unless the recorded model id proves it. We do not claim "the most advanced model per vendor": we record the exact model id that ran for each datapoint, in the raw response payload and in the customer report's Data Provenance block.

3. The 4-column methodology matrix (every datapoint carries it)

Every datapoint in every audit carries these 4 columns. If any cell is empty, the datapoint is rejected before the report is drafted: the audit does not ship with a blank method row.

4. Delivery package (every audit)

Every Clarivy audit ships with three core files plus an Agent Bundle Manifest:

1. human-report.pdf: the customer-facing report. Snapshot ships a 1-page PDF.
2. ai-employee-brief.md: an AI-readable Markdown brief with stable finding_id / evidence_id / action_id identifiers, designed for ingestion by your internal AI agents or your Obsidian vault. An AI can summarize the audit without paraphrasing it, and every recommendation is traceable back to the underlying raw LLM response.
3. raw-data-index.md: a Markdown index listing every (endpoint, prompt) datapoint, with a raw JSON reference in the delivered evidence package or approved evidence store. You (or any auditor) can verify every claim in the report by inspecting the referenced raw response.
4. agent-bundle/manifest.json: the machine-readable entry point for automated workflows. It points agents to findings, evidence, recommended actions, verification plans, memory delta, handoff notes, and adapter instructions.

For Professional and Enterprise scopes, the human-report.pdf expands the same evidence structure: Professional can use the 5-notebook bound PDF (Index / Intent / Content / Quotables / Strategy); Enterprise can extend it with additional markets, languages, and refresh windows when agreed in the quote.

4.1 AI / LLM agent consumption contract

Clarivy reports are designed to be read by humans and by AI agents. The human report explains the judgment; the AI-readable Markdown brief carries the execution graph; the raw-data index lets an agent verify the evidence before it acts.

This is why we describe Clarivy as an AI/LLM-friendly GEO audit platform: the claim is grounded in delivered files, stable identifiers, and raw evidence, not in a broad superlative.

4.2 Deterministic GEO score pair

Clarivy's structured findings include two bounded GEO scores. They are computed by the audit runner before the analysis node writes narrative, so the LLM explains the scores but does not invent or recompute them.

The scoring boundary is deliberate: a Snapshot can say what the tested AI endpoints observed and cited. It cannot honestly claim full SEO authority or domain trust without external backlink, traffic, knowledge-graph, and site-crawl data.

4.3 Repeat-audit customer memory

For repeat audits on the same customer subject, the audit runner can load a small sanitized memory context before the analysis node writes narrative. This memory helps the report avoid repeating old advice blindly and helps the customer's AI agent compare the current run with prior runs. The same continuity layer supports Enterprise Monitor: monthly or recurring GEO deltas should compare like-for-like prompt sets without forcing the customer to re-explain prior decisions every month.

The privacy boundary is explicit: customer memory excludes email, contact name, address, tax id, billing data, raw response text, and raw JSON payloads. It is historical context, not evidence for the current run. Current-run claims must still cite the current structured findings and raw JSON evidence. Customers can request deletion or opt out of repeat-audit memory; the legal boundary is documented in Privacy Policy v1.2, DPA v1.1, and Terms v1.1.

5. Snapshot is a quick diagnostic; Professional / Enterprise are deeper audit scopes

The Snapshot product is a quick diagnostic / risk screening, not a complete commercial-decision-grade audit. Concretely:

• Snapshot baseline (40 datapoints, contact-based access): 5 queries you choose × 8 production model endpoints. Use it to risk-screen a known query ("are we being defamed?"), spot-check a category entry, or baseline a brand before a larger engagement. It is not a substitute for a category-wide audit and should not be the sole basis for a major GEO budget decision.
• Professional Audit and Enterprise Audit are deeper scoped audits. They are quoted after scope review, and the final prompt count, markets, languages, delivery window, refund boundary, and invoice terms are confirmed in the quote or order form.

A Snapshot report's cover page says so explicitly: "Quick diagnostic, not a complete audit": so you do not over-read 40 datapoints into a category verdict.

6. API vs product surface: what the audit does and does not measure

This page is the place we make a hard distinction that other GEO vendors blur. The Clarivy audit is an API audit, not a product-surface audit.

• What we measure. API responses from each endpoint, with controlled prompts, controlled temperature / seed, and a reproducible request payload. The data is therefore a controlled, reproducible datapoint suitable for trend analysis and benchmarking.
• What we do not measure. The Clarivy audit does not fully replicate:
  • the ChatGPT or Gemini in-app chat UI, including personalization, memory, custom GPTs, or tool-use state;
  • Google AI Overviews as they actually appear in a real Google SERP (the audit uses the Gemini API endpoint closest to that surface, but SERP layout, source mix, and ranking are Google's, not ours);
  • Perplexity product UI behavior, including its ranking algorithm, source-selection logic, and personalization;
  • any per-user / per-session personalization, location bias, account-state, or signed-in preferences.
• What this means for the customer. The audit measures "what the underlying model says, on a controlled prompt, today." It is the most reproducible signal available, but it is not a 1:1 mirror of what a real user sees in a real product UI. We disclose this on every report's cover page.

7. Model freshness policy

We do not promise that every endpoint is always running its vendor's "most advanced" model. Three concrete things instead:

1. Monthly model-list review. On the first business day of every month we re-check each provider's official catalog, and we update the configured model id for any endpoint where the vendor has a clearly-better / clearly-newer default for our use case. The change is recorded as a dated methodology or evidence-store snapshot before it is used in customer delivery.
2. Per-datapoint model id disclosure. The exact model id that produced each datapoint is recorded in the raw response payload and surfaced in the customer report's Data Provenance block. You can always see which model produced which answer; we do not hide this behind a generic "ChatGPT" or "Claude" label.
3. No "always the newest" claim. Some vendors retire models, change pricing, or change behaviour in ways that are not yet stable. We will not silently chase a "most advanced" label if doing so would change pricing, break reproducibility, or compromise data-control guarantees. The Data Provenance block is the source of truth: if you care which model produced a specific number, that is where the answer lives.

8. Delivery QA: what runs before a report is emailed to the customer

Every audit must pass the Delivery QA gates below before the email is sent. A draft that fails any of them is fail-closed: it is held for human review and you are informed of the delay, rather than shipped with a known defect.

1. PDF render check. The rendered human-report.pdf is opened and visually verified against the Markdown source (no overflow, no missing pages, no encoding glitches). Snapshot must be exactly 1 page.
2. AI-readable Markdown brief structure check. The ai-employee-brief.md is schema-validated: every finding_id has a matching evidence_id, every action_id references a finding_id, no broken cross-references.
3. Raw data index check. The raw-data-index.md lists exactly N entries (N = expected datapoints for the product: 40 for Snapshot). Every entry must have a resolvable evidence reference inside the delivered package or approved evidence store. A missing or broken reference is a fail-closed defect.
4. Evidence-store reference check. The storage path for the order is verified: the report.pdf, the AI Brief, the raw-data index, and every per-endpoint JSON are present and readable under the approved delivery scope. A single missing file is a fail-closed defect.
5. Fail-closed empty / error response gate. If any endpoint returned an empty response, a 4xx/5xx error, or a refusal, that datapoint is marked partial=true in the raw index and the report's per-endpoint score table shows the partial result (e.g. "0 / 5" not "0 / 5 ✓"). A run with > 20% of datapoints in partial state is held for human review before delivery; the customer is told which endpoints were affected.

9. The 6 anti-hallucination rules (report content)

These six rules gate the report content from going to the customer. A draft that violates any of them is rewritten before delivery.

1. No absolute numbers in headlines. "5 of 30 prompts mention you": not "You are mentioned 16.7% of the time." Numbers are exact; percentages imply false precision.
2. Every claim cites a raw JSON line. "Gemini cited tryprofound.com on prompt #14" must link to snapshots/2026-06-15/gemini_aio-014.json.
3. No "best" / "only" / "guaranteed." These are superlatives. We do not assert them about our own product, and we do not assert them about competitors in customer reports.
4. Disclose endpoint stochasticity. Every report includes a "this is a snapshot" disclaimer that model responses can vary by endpoint, model, prompt, and sampling configuration.
5. Distinguish "no result" from "no data." "Gemini returned no result for prompt #18" is different from "We did not call Gemini for prompt #18." We never silently conflate them.
6. No automated decisions. We score; humans (us) interpret. We do not auto-reject a brand from being audited because of low mention counts; we report the data and let the customer decide.

10. What the audit does not claim

• It does not predict future rankings. AI search products and model endpoints change behaviour monthly.
• It does not claim that acting on the recommendations will produce a specific lift in revenue, traffic, or AI-search ranking. The Princeton 2023 paper reports an aggregate +40% visibility in a controlled setting; individual results vary widely, and the audit is an analytics instrument, not a guarantee of business outcomes.
• It does not cover every AI model that exists. We use the 8 endpoints listed in §2 for the current production matrix; we will not advertise "12+" or "all major models."
• It does not score on ranking position within a single response (yet). v2 will introduce position-weighted scoring.
• It does not replicate the in-app product surface of ChatGPT, Gemini, or Perplexity (see §6).

11. Confidence, accuracy, and limitations: what a buyer can rely on

This section is for procurement and finance leads who need to answer "how accurate is the report, and what is it safe to base a decision on?" The short version: the report is an analytical instrument with auditable raw evidence and a deterministic confidence score: it is not a forecast, and it does not guarantee business outcomes. The details:

11.1 What we mean by "accuracy"

For every audit we publish three independent things, and each has its own accuracy claim:

1. Raw response fidelity: every (endpoint, prompt) datapoint in the raw-data-index.md is the verbatim response we received, with a timestamp and a raw JSON reference. A buyer (or any auditor) can inspect the referenced JSON and confirm that the report's claim about that prompt matches the raw response. This is verifiable, not estimated.
2. Mention / coverage math: the per-endpoint and per-category counts in the report are computed deterministically from the raw responses, not regenerated by an LLM. The same raw JSONs fed into the report today and in 6 months will produce the same numbers. This is reproducible.
3. Interpretation and recommended actions: these are drafted by the analysis node and reviewed under the 6 anti-hallucination rules in §9. The interpretation is not a fact about the world; it is a senior-analyst read of facts (1) and (2). The Data Provenance block on every report tells the reader which inputs produced which paragraph, so a buyer can separate "what the LLM said" from "what we recommend doing about it."

We do not claim guaranteed accuracy in the sense of "our report is 100% right about your future rankings." AI search products are stochastic, models change monthly, and the report measures a single point in time. The accuracy we do commit to is documented in the next two subsections.

11.2 Confidence score (deterministic, not LLM-generated)

Every audit ships with a confidence: high | medium | low field on the AI-readable Markdown brief's YAML frontmatter. It is computed by the computeConfidence() helper in audit-runner/src/render/ai-employee-brief-render.js against endpoint completion, with these thresholds:

A medium or low confidence does not invalidate the audit; it just means the buyer should treat the missing endpoint rows as a "data gap" rather than a zero. The per-endpoint score table in the human-report PDF surfaces this explicitly (e.g. "0 / 5 (partial)" instead of "0 / 5 ✓").

11.3 Limitations: what the audit does not measure

The audit is a measurement instrument with a deliberately bounded scope. The five honest limits every buyer should know:

1. Binary mention only (v1). The current scoring counts "mentioned" or "not mentioned" per (endpoint, prompt). Position-weighted scoring (a mention in the first sentence is more important than a mention in the fifth paragraph) is on the v2 roadmap. Buyers comparing this audit to a vendor that publishes "mention-rank score" should know we are not yet doing that.
2. API surface, not product surface (see §6). The datapoint is the underlying model's response on a controlled prompt, not what a real user sees in ChatGPT, Gemini, or Perplexity with personalization, memory, and tool-use state. A buyer who needs a "what does a real signed-in user see?" measurement needs a different instrument (and probably a different budget).
3. Stochasticity is honest, not hidden. A re-run within 24 hours can vary by endpoint, model, prompt, and sampling configuration. The Data Provenance block + §9 Guardrail 4 + the AI-readable Markdown brief §11 tell the buyer which endpoint class and model id produced each datapoint; the report does not collapse all endpoints into a single "we measured it" claim.
4. No ranking prediction. The audit reports observed evidence for a point in time or agreed cadence. It does not predict what ChatGPT will say in 3 months, and it does not guarantee that acting on the recommendations will produce a specific lift in revenue, traffic, or share-of-voice. Enterprise Monitor compares month-over-month runs where scoped, but each run remains its own evidence snapshot.
5. No customer business-inference. The report never asserts the customer's revenue, headcount, or market share. If a buyer needs that context, the customer briefing form is where it is supplied; the analysis node is forbidden from inventing it.

11.4 How "professional" is enforced

The audit is treated as a regulated analytics deliverable, not a marketing artifact. Three concrete gates enforce that:

• Analysis node is locked: audit-runner/prompts/system-prompt.md contains 6 non-negotiable guardrails (no "best", no "guaranteed", no invented citation URLs, no competitor rankings, no percentage headlines, no omitted Data Provenance footer). The guardrails are the literal system prompt; a violation forces a re-draft before delivery.
• Delivery QA is fail-closed: §8 lists 5 gates (PDF render, AI Brief schema, raw-index N-check, R2 file check, partial-rate gate). A draft that fails any gate is held; the buyer is told, not silently sent a broken deliverable.
• Reproducibility is verifiable: every report's auditLogsUrl or raw-data base points to the delivered evidence package or approved evidence store for that order. The buyer (or any auditor) can inspect the raw JSON references, replay the allowed prompt inputs, and diff the result.

11.5 What "decision-grade" means for a buyer

A Snapshot audit is safe to use as the analytical input for:

• A CMO / Head-of-Growth budget allocation conversation (where to spend the next GEO dollar, which publisher to pitch first, which category to defend).
• A PR / content team's quarterly editorial calendar (which competitor mentions to disrupt, which own page to harden with structured data and llms.txt).
• A monthly trend baseline via Enterprise Monitor. Where scoped, trend reporting remains honest about endpoint-level stochastic noise so you do not chase phantom weekly swings.

It is not a substitute for any of: user-research interviews, clickstream / conversion analytics, an SEO backlink audit, or a brand-tracker survey. It is one input; you decide how to combine it with the rest of your evidence.

For a buyer-facing Q&A: including "how accurate is the report, and why this price?": see the companion page /audit/faq.html (Buyer-Objection FAQ). The two pages cross-link so a procurement lead never has to read the methodology to find the answer to a commercial question.

12. Versioning & change log

• v1.0 (2026-06-11): Initial 7-endpoint matrix, 30-prompt set, 5-notebook structure, 6 anti-hallucination rules.
• v2.0 (2026-06-15): Historical product matrix aligned to the then-current production surface: Snapshot 5 user queries × 7 endpoints = 35 datapoints; the deeper audit product was still named Standard at that time; Enterprise was described as a planned multilingual refresh product. Added §1 (product table), §4 (3-file delivery package), §5 (Snapshot quick diagnostic vs deeper products), §6 (API vs product surface limitation), §7 (Model freshness policy), §8 (Delivery QA gate), §11 (Confidence / accuracy / limitations). Endpoint list pruned to current production (no ERNIE / MetaSo / Perplexity Sonar as part of the current production matrix); historical Day-1 dogfooding references remain in self-audit-01.html with a "historical artifact" banner. Cross-link to companion Buyer-Objection FAQ added in §11.5.
• v2.1 (2026-06-16): Historical P0 product-status reset. Snapshot was the lightweight baseline at that time, while Professional and Enterprise were described as planned deeper audit products. This commercial status was superseded by the 2026-06-22 Enterprise-first scope-review model; the technical contract on this page (endpoint matrix, methodology, delivery QA, anti-hallucination rules, confidence / accuracy / limitations) is unchanged.
• v2.2 (2026-06-16): P1 copy polish. Naturalised phrasing across §1 / §2 / §3 / §4 / §5 / §7 / §8 / §9 / §10 / §11 / §11.5. Renamed the 3-file delivery to "human PDF + AI-readable Markdown brief + raw-data index" (the brief's full name, replacing the earlier internal shorthand in the customer surface; the underlying ai-employee-brief.md filename and §9 anti-hallucination guardrails are unchanged). Removed "we commit to" stacking in §1 / §2 / §7 / §11. No technical change: endpoint list, math, QA gates, anti-hallucination rules, and confidence thresholds are unchanged.
• v2.3 (2026-06-17): Reframed the current production list as model endpoints rather than language-labelled coverage. Clarified that all tested models are multilingual.
• v2.4 (2026-06-18): P0 SEO/GEO stopgap. Title and H1 updated to surface "AI visibility audit methodology" so the page better answers AI-search-intent queries ("AI search visibility audit methodology", "GEO audit methodology"). Added a machine-readable facts file at /llms.txt and a canonical entity page at /about/; both link back to this methodology page as the technical contract. No technical change: endpoint matrix, math, QA gates, anti-hallucination rules, and confidence thresholds are unchanged.
• v2.5 (2026-06-18): Historical changelog: reframed the 7 production endpoints from regional provider grouping to model openness surface. This framing was superseded in v3.3 by the stricter vendor-hosted API + public-weight availability wording.
• v2.6 (2026-06-19): Added the AI / LLM agent consumption contract for the delivery package. The public method now states how ai-employee-brief.md, raw-data-index.md, raw JSON evidence, and llms.txt should be consumed by autonomous agents. No endpoint count or datapoint math changed.
• v2.7 (2026-06-19): Added the deterministic GEO score pair: AI Citation Readiness and Source Trust Baseline. Clarified that these are bounded GEO scores derived from observed LLM responses and citation fields, not SEO, CORE-EEAT, CITE, backlink, traffic, or ranking scores.
• v2.8 (2026-06-19): Added repeat-audit customer memory. Clarified the HOT/WARM/COLD lifecycle, the privacy boundary, and the rule that memory is historical context rather than current-run evidence.
• v2.9 (2026-06-19): Historical boundary. Aligned repeat-audit memory with Privacy Policy v1.1, DPA v1.1, and Terms v1.1. At that time, recurring monitoring was documented as a future continuity concept that would reuse the AI-readable evidence package plus sanitized same-customer memory. Superseded by v3.1.
• v3.0 (2026-06-21): Published the P0-P3 agent-ready infrastructure record. Added Updates as the public product/methodology changelog and Agent Integrations as the public AI agent consumption contract. Clarified that agent ecosystem notes are file-workflow compatibility, not vendor certification. No endpoint count, Snapshot price, datapoint math, or roadmap subscription boundary changed.
• v3.1 (2026-06-23): Superseded the v2.9/v3.0 recurring-monitoring commercial boundary. Clarivy now leads with Enterprise Monitor: monthly full audits by default, optional weekly pulse checks, human reports, and dedicated AI-agent deliverables. Enterprise Monitor can be prepaid or subscription-based after scope review; no subscription or recurring billing starts without explicit approval in the quote or order form. Endpoint count, Snapshot datapoint math, Delivery QA, and anti-hallucination rules were unchanged.
• v3.2 (2026-06-26): Added Grok / xAI as the 8th production endpoint. Snapshot now runs 5 user queries × 8 endpoints = 40 datapoints; Professional and Enterprise quotes use the same 8-endpoint production matrix unless scoped otherwise.
• v3.3 (2026-06-26): Tightened model classification language: all 8 measured endpoints are vendor-hosted APIs; public-weight availability is disclosed only where it applies to specific model families or checkpoints.

Substantive product and methodology changes are announced in Updates and reflected here. Long-form measured evidence articles remain on the blog.