--- title: "The AI Visibility Holdout Test: Fix Half Your Pages, Then Check What Changed" description: "A six-week test protocol for comparing AI-search fixes against untouched pages using verified bot fetches and live-answer citations, not vendor visibility scores." url: "https://groas.com/post/the-holdout-test-for-ai-visibility-fix-h" image: "https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/c326fb85-2eb8-4458-8368-8e85e2063ad7.png" published: "2026-10-14T05:39:34.286Z" modified: "2026-10-14T05:39:34.361Z" --- October 14, 2026 · 10 min read # The AI Visibility Holdout Test: Fix Half Your Pages, Then Check What Changed [Alexander PerelmanHead Of Product @ groas](https://groas.com/author/alexander-perelman)[LinkedIn](https://www.linkedin.com/in/alexander-433793253/) ![Cartoon: a salesman credits his fresh paint job for lifting one rowboat, while the tide has lifted its unpainted twin just as high.](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/c326fb85-2eb8-4458-8368-8e85e2063ad7.png) In this article 1. [The question: did the fix earn citations?](#the-question-did-the-fix-earn-citations) 2. [Why a before-and-after chart cannot answer the question](#why-a-before-and-after-chart-cannot-answer-the-question) 3. [Setup: match pages before touching them](#setup-match-pages-before-touching-them) 4. [Measurements: keep fetches and citations separate](#measurements-keep-fetches-and-citations-separate) 5. [Run the six-week protocol](#run-the-six-week-protocol) 6. [Read the result against the holdout](#read-the-result-against-the-holdout) 7. [Make the tool earn its place](#make-the-tool-earn-its-place) A vendor’s AI-visibility chart goes up two weeks after a schema rewrite. I want to see the pages they didn’t touch. Without a holdout, that rising line could just as easily belong to an engine update, a fresh crawl, or a change in how the vendor calculates its score. I would not buy a bid-management tool because conversions rose during Black Week after someone switched it on. I would ask what happened to comparable campaigns left alone. **AI-visibility fixes deserve the same test.** ## The question: did the fix earn citations? This protocol asks whether a technical change increases verified crawler retrieval and citations in live answers **relative to comparable, untouched pages**. That is a narrower and more useful question than whether a dashboard’s visibility score went up. I use a six-week window: two weeks to establish a baseline, then four weeks with one fix applied to half the pages. The other half stays untouched. At the end, compare changes in verified bot fetches and citations against the holdout, not against a screenshot of where the treated pages started. That distinction matters when evaluating [which tools help businesses fix technical SEO issues that hurt AI visibility](https://groas.com/post/tools-to-get-featured-in-google-ai-overviews). A tool should get credit for a change that beats the holdout, not for being installed when Google next refreshes its index. ## Why a before-and-after chart cannot answer the question ### Search engines change while your test is running The familiar vendor sequence goes like this: crawl a site, flag 400 “AI-readability errors,” change the schema, and present a citation chart two weeks later. The chart may be accurate. The explanation may not be. Search engines do not hold their retrieval behavior steady for your campaign calendar. In an analysis of Google citation behavior, [Ahrefs reported that 37.1% of URLs cited in Google AI Overviews came from the top 10 organic results](https://ahrefs.com/blog/ai-overviews-top-10/), down from 76% in an earlier analysis. Query fan-out can bring different sources into an answer as an engine pursues background sub-queries. A change in that process can move citations even when you change nothing. ![A presentation chart rises while a crane tilts the room around it.](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/6ee00e60-8348-41d8-bb72-f423cff0ed75.png) ### Borrow the PPC control, not the PPC vocabulary Paid search operators know this problem. Switch bidding strategies in November, see conversions rise in December, and you have not yet separated your work from the shopping calendar. You need a simultaneous baseline. The same logic applies here. As [SearchPilot explains in its SEO testing framework](https://www.searchpilot.com/resources/guides/what-is-seo-ab-testing/), comparable URLs can be grouped using historical traffic, volatility, and topical intent. When both groups face the same external shift, the difference between them becomes more informative than either group’s before-and-after chart. AI retrieval adds noise of its own. A page can be fetched without being cited; a citation can appear or disappear across repeated answers. That is why this test records both events separately. **A fetch is evidence of retrieval, not a citation.** Keep that distinction intact all the way to the decision. ## Setup: match pages before touching them ### Choose 10–20 URLs, then make lookalike pairs Take 10–20 pages with similar topical depth, template structure, and publication age. Pull the last 60 days of organic clicks from Google Search Console and pair pages whose traffic falls within roughly 15% of each other. Check intent as well as volume. Two pages with similar click totals but entirely different jobs are not a useful pair. For example, pair a page about “best CRM for commercial contractors” drawing 800 monthly visits with a page about “best ERP for electrical contractors” drawing 750, rather than with your homepage. Do not put five revenue-driving pages on one side and five forgotten posts on the other and call it an experiment. The spreadsheet will look tidy. The comparison will not. ![A split-test ledger places matched blog URLs in variant and holdout columns.](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/2714ece2-fd4e-4c82-a501-8865ebbe29aa.png) ### Assign one page in each pair to the holdout Once the pairs are set, flip a coin for each one. Put one URL in the **treated group** and the other in the **holdout**. Record the assignment before making changes, and do not reshuffle pages after you see an early result. [SearchPilot’s explanation of counterfactual modeling](https://searchpilot.com/resources/blog/how-searchpilot-analyses-test-data/) describes the purpose of matched controls: estimating what the treated group might have done without the intervention. Freeze holdout pages for the full test. That means no copy or metadata edits, template changes, internal-link additions, or quiet CMS cleanups. Tell the people who can publish changes which URLs are off limits. An accidental edit in week four is not a footnote you can wave away later; it changes the comparison you set out to make. ### Test one fix category, not an entire wishlist Pick **one hypothesis** before deployment. These are three separate tests, not a package to install on the same afternoon: - **Structural clarity:** Rewrite the first 120 words into a direct answer to the page’s primary query, without a long preamble. - **Entity and schema depth:** Add precise JSON-LD, such as `TechArticle`, `ItemList`, or relevant `about` declarations, to clarify the page’s topic. - **DOM and rendering simplicity:** Replace client-side accordions or heavy tabs with semantic HTML that exposes the relevant text on initial load. If you rewrite the introduction, add schema, remove tabs, and rebuild the template together, a positive result cannot tell you which change deserves the next rollout. Nor can a negative result tell you what to remove. Choose one intervention and document exactly which treated URLs receive it. ## Measurements: keep fetches and citations separate ### Count verified bots, not claimed user agents Filtering server logs for `GPTBot` or `PerplexityBot` user-agent strings is not enough. A scraper can claim either name. A [crawler-traffic analysis citing HUMAN Security](https://wrenda.com/ai-traffic-data-overcounting-undercounting-2026/) reports spoofed AI crawler requests, and [our 30-day bot-traffic log](https://groas.com/post/we-logged-30-days-of-ai-bots-hitting-our-2) found 3,482 genuine requests among 13,774 claimed bot hits. The exact mix on your site may differ. The practical point is that a raw header is not verification. Set up the checks before the baseline begins. As [Centinel Analytica’s crawler-verification breakdown](https://centinelanalytica.com/verify-ai-crawlers-agents-fake-gptbot/) explains, Googlebot and Applebot use forward-confirmed reverse DNS, while OpenAI and Perplexity publish IP-range files for checking their crawlers. Exclude requests that fail the relevant check. Otherwise, a noisy or spoofed request stream can masquerade as the result of your fix. ![A flow diagram separates claimed bot hits from requests checked against DNS or published IP ranges.](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/a14787a6-f0e0-4e85-99af-1fc0ee453571.png) ### Separate background crawling from live-answer fetches Not every verified fetch has the same meaning. [OpenAI distinguishes its bots](https://developers.openai.com/bots): `GPTBot` crawls for model-related purposes, `OAI-SearchBot` serves search discovery, and `ChatGPT-User` fetches content in response to a user interaction. Record them separately rather than pouring every request into one “AI traffic” total. A verified `ChatGPT-User` request is a useful **leading indicator of live retrieval**. It still does not show that your URL appeared as a citation in the answer. [Presenc’s guide to the bot](https://presenc.ai/blog/chatgpt-user-bot-guide/) explains why these fetches matter and why ordinary pageview analytics may miss them. Use server access logs to measure the requests; use answer checks to measure citations. One cannot stand in for the other. ### Check citations against a fixed prompt set Before you change any page, write five specific user-intent prompts for each pair. That gives you 25–50 prompts across a 5–10-pair test. Keep the wording and page-pair assignment fixed throughout. Every Tuesday at 10:00 AM UTC, run that same battery across Google AI Overviews, Perplexity Pro, and ChatGPT Search. For each prompt and engine, record `1` if the specific URL appears as an inline reference, card, or footnote; otherwise record `0`. Keep the answer or a screenshot with the entry so you can inspect a surprising result later. Fixed prompts, timing, and sequence do not eliminate variation between answers. They do stop you from moving the goalposts by choosing new prompts after one group starts looking good. Ignore the proprietary visibility score for this decision. **The outcome is the citation you can see, not the score a dashboard assigns.** ## Run the six-week protocol ### Weeks 1–2: establish the baseline Do not deploy the fix when you set up the groups. For 14 continuous days, collect daily verified fetches by URL and bot type for both buckets. Run the scheduled citation checks with the fixed prompts. If verification fails or a page changes during baseline, fix the setup before treating anything. At the end of day 14, divide treated-bucket fetches by holdout-bucket fetches to get a baseline retrieval ratio. Treat roughly 0.85–1.15 as a practical pairing check, not a certificate that the pages are identical. If the prospective treated pages receive twice the verified crawler volume before any intervention, revisit the pairs. A lopsided starting point will make a later lift hard to read. Keep the baseline citation records too. Fetches and citations can start from different positions, and you will need both starting points when you compare the groups after deployment. ### Day 15: change treated pages only Deploy the one chosen fix to the treated URLs. For an answer-block test, edit only those openings. For a schema test, place the validated JSON-LD on the treated pages only. Purge cached versions of those URLs so crawlers can reach the changed HTML. Record what you deployed and when. Then freeze both buckets for the remainder of the test. Do not slip in a metadata improvement, replace an image, or add links from a newly published post because it seems harmless. Some of those changes might be good work. They are not part of this test. ### Weeks 3–6: watch for drift and contamination Leave the fix in place for the remaining four weeks. Continue daily verified-log collection and the weekly prompt checks. The post-deployment run gives retrieval and answer systems time to revisit pages; it does not guarantee that every URL will be reconsidered on your schedule. Keep a simple change log for the bucket pages. If someone edits a holdout page to repair a pricing table in week four, repair the table if it needs repairing, then **remove that entire pair from the evaluation**. Do not try to explain away the edit with a spreadsheet adjustment. If you began with 10 pairs and lose two, assess the eight uncontaminated pairs and say that is what you did. Apply the same rule when a treated page receives an unrelated change. A smaller clean comparison is more useful than a larger one whose pages no longer reflect the protocol. ## Read the result against the holdout At the end of week six, compare each group’s change from its own baseline. Look at verified bot fetches by type and at the fixed-prompt citation records. Do not treat a rise in one as proof of a rise in the other. | Pattern | Verified fetches | Fixed-prompt citations | What to do | | :--------------------------------- | :--------------------------------------------------------------------------------- | :--------------------------------------------------- | :---------------------------------------------------------------------------------------------------------- | | **Treated pages beat the holdout** | Relevant verified fetches rise relative to the holdout’s baseline-adjusted change. | Citations also improve relative to the holdout. | Consider extending the same fix to comparable pages; keep measuring as you roll it out. | | **No clear difference** | The groups stay near their baseline relationship. | Neither group gains a clear advantage. | Do not fund a wider rollout on this result. Revisit the hypothesis or the measurement before spending more. | | **Both groups move together** | Both rise or fall at roughly the same time. | Citations shift in the same direction across groups. | Treat the shared movement as background drift, not credit for the fix. | A fetch advantage without a citation advantage is its own finding: machines may be reaching the treated pages more often without choosing them for answers. Sparse citations across both groups are also hard to interpret. In either case, do not promote a crawler chart into a claim about earned citations. I would rather get a null result from a clean test than a beautiful graph I cannot explain. If nested schema changes nothing for the treated pages relative to the holdout, that is a reason to stop paying to roll out _that fix_ on similar pages. It is not proof that schema never matters anywhere. The decision is smaller, and much more useful. ## Make the tool earn its place This is the standard I would put in front of any tool claiming to “fix” AI visibility. A platform that tracks prompts, graphs a synthetic index, and exports a monthly PDF is an observer. It may show you something worth investigating. It has not demonstrated that its recommended fix caused a citation. The work is operational: spot a retrieval problem, make a defined change, verify which bots reach the page, check whether live answers cite it, and compare all of that with pages left alone. That is the standard behind [groas](https://groas.com/), which executes search and content work continuously with a named human strategist setting direction and accountability. But I would apply the same holdout rule to its changes. A brand name does not get to be the control group. If the treated pages win on verified retrieval **and** citations, extend the fix to comparable pages and keep checking. If they do not, leave the holdout alone, retire the claim, and spend the next test on a better hypothesis. The slide can wait. ## Frequently Asked Questions ### Why isn't a before-and-after chart enough to prove an AI visibility fix worked? Search engines change their retrieval behavior while your test runs. For example, Ahrefs reported that the share of Google AI Overview citations coming from the top 10 organic results fell from 76% to 37.1% between analyses. Query fan-out can shift sources into answers even when nothing on your site changed, so a rising chart may reflect engine drift rather than your fix. ### How long should an AI visibility holdout test run? Use six weeks: two weeks to establish a baseline with no changes, then four weeks with the fix applied to half the pages while the other half stays untouched. At the end, compare changes in verified bot fetches and citations between the two groups. ### How do I choose and pair pages for an AI visibility test? Take 10 to 20 pages with similar topical depth, template structure, and publication age, then use the last 60 days of Google Search Console clicks to pair pages whose traffic falls within roughly 15% of each other. Check intent as well as volume, and flip a coin for each pair so one URL gets the fix and the other stays in the holdout. ### Should I apply several fixes at once during the test? No, pick one hypothesis before deployment. Rewriting the first 120 words into a direct answer, adding JSON-LD schema, and replacing client-side accordions with semantic HTML are three separate tests. If you apply them together, a positive result cannot tell you which change deserves a rollout and a negative one cannot tell you what to remove. ### How do I verify that AI bot traffic is genuine? A user-agent string like GPTBot or PerplexityBot is not enough, since scrapers can claim those names. Use forward-confirmed reverse DNS, which Googlebot and Applebot support, or the IP-range files OpenAI and Perplexity publish, and exclude requests that fail the check. ### Does a verified ChatGPT-User fetch mean my page got cited? No. A verified ChatGPT-User request is a leading indicator of live retrieval, but it does not show that your URL appeared as a citation in the answer. Use server access logs to measure fetches and separate answer checks to measure citations, recording GPTBot, OAI-SearchBot, and ChatGPT-User requests separately. ### How do I measure citations in live AI answers? Before changing any page, write five specific user-intent prompts per page pair, giving 25 to 50 prompts for a 5 to 10-pair test. Every Tuesday at 10:00 AM UTC, run the same battery across Google AI Overviews, Perplexity Pro, and ChatGPT Search, recording 1 if the URL appears as an inline reference, card, or footnote and 0 otherwise. ### What should I do if a holdout page gets edited during the test? Repair the page if it genuinely needs repairing, then remove that entire pair from the evaluation instead of adjusting for it. Assess only the uncontaminated pairs and state how many you used. A smaller clean comparison is more useful than a larger one whose pages no longer reflect the protocol. ## Pay For Results, Not For Hours Businesses buy the outcome, agencies resell it, and groas answers for it either way. [See If You Qualify](https://groas.typeform.com/to/xC1bQNUT) ## Structured data ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://graph.groas.com/entity/groas","name":"groas","url":"https://groas.com/","logo":{"@type":"ImageObject","url":"https://groas.com/icon-512.png","width":512,"height":512},"sameAs":["https://www.linkedin.com/company/groas/"]},{"@type":"WebSite","@id":"https://groas.com/#website","name":"groas","url":"https://groas.com/","inLanguage":"en-US","publisher":{"@id":"https://graph.groas.com/entity/groas"}},{"@type":"WebPage","@id":"https://groas.com/post/the-holdout-test-for-ai-visibility-fix-h#webpage","url":"https://groas.com/post/the-holdout-test-for-ai-visibility-fix-h","name":"The AI Visibility Holdout Test: Fix Half Your Pages, Then Check What Changed","description":"A six-week test protocol for comparing AI-search fixes against untouched pages using verified bot fetches and live-answer citations, not vendor visibility scores.","inLanguage":"en-US","isPartOf":{"@id":"https://groas.com/#website"},"about":{"@id":"https://graph.groas.com/entity/groas"},"primaryImageOfPage":{"@type":"ImageObject","url":"https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/c326fb85-2eb8-4458-8368-8e85e2063ad7.png"},"breadcrumb":{"@id":"https://groas.com/post/the-holdout-test-for-ai-visibility-fix-h#breadcrumb"},"dateModified":"2026-10-14T05:39:34.361Z","mainEntity":{"@id":"https://groas.com/post/the-holdout-test-for-ai-visibility-fix-h#article"}},{"@type":"BreadcrumbList","@id":"https://groas.com/post/the-holdout-test-for-ai-visibility-fix-h#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://groas.com/"},{"@type":"ListItem","position":2,"name":"Blog","item":"https://groas.com/blog"},{"@type":"ListItem","position":3,"name":"The AI Visibility Holdout Test: Fix Half Your Pages, Then Check What Changed","item":"https://groas.com/post/the-holdout-test-for-ai-visibility-fix-h"}]},{"@type":"BlogPosting","@id":"https://groas.com/post/the-holdout-test-for-ai-visibility-fix-h#article","url":"https://groas.com/post/the-holdout-test-for-ai-visibility-fix-h","mainEntityOfPage":{"@id":"https://groas.com/post/the-holdout-test-for-ai-visibility-fix-h#webpage"},"isPartOf":{"@id":"https://groas.com/blog#blog"},"headline":"The AI Visibility Holdout Test: Fix Half Your Pages, Then Check What Changed","description":"A six-week test protocol for comparing AI-search fixes against untouched pages using verified bot fetches and live-answer citations, not vendor visibility scores.","datePublished":"2026-10-14T05:39:34.286Z","dateModified":"2026-10-14T05:39:34.361Z","image":{"@type":"ImageObject","url":"https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/c326fb85-2eb8-4458-8368-8e85e2063ad7.png"},"author":{"@id":"https://groas.com/author/alexander-perelman#person"},"publisher":{"@id":"https://graph.groas.com/entity/groas"},"keywords":"pages, test, fix, holdout, fetches, visibility, change, verified","wordCount":2134,"inLanguage":"en-US"},{"@type":"Person","@id":"https://groas.com/author/alexander-perelman#person","name":"Alexander Perelman","jobTitle":"Head Of Product @ groas","description":"Ex Goldman Sachs and Ex Stanford Computer Science","image":"https://groas.com/media/blog/4f17cac3a81acc224cc1b2cfab8f94bbc295a3089d13e8421125771ef8d4064f.jpg","worksFor":{"@id":"https://graph.groas.com/entity/groas"},"sameAs":["https://www.linkedin.com/in/alexander-433793253/"]},{"@type":"FAQPage","@id":"https://groas.com/post/the-holdout-test-for-ai-visibility-fix-h#faq","mainEntity":[{"@type":"Question","name":"Why isn't a before-and-after chart enough to prove an AI visibility fix worked?","acceptedAnswer":{"@type":"Answer","text":"Search engines change their retrieval behavior while your test runs. For example, Ahrefs reported that the share of Google AI Overview citations coming from the top 10 organic results fell from 76% to 37.1% between analyses. Query fan-out can shift sources into answers even when nothing on your site changed, so a rising chart may reflect engine drift rather than your fix."}},{"@type":"Question","name":"How long should an AI visibility holdout test run?","acceptedAnswer":{"@type":"Answer","text":"Use six weeks: two weeks to establish a baseline with no changes, then four weeks with the fix applied to half the pages while the other half stays untouched. At the end, compare changes in verified bot fetches and citations between the two groups."}},{"@type":"Question","name":"How do I choose and pair pages for an AI visibility test?","acceptedAnswer":{"@type":"Answer","text":"Take 10 to 20 pages with similar topical depth, template structure, and publication age, then use the last 60 days of Google Search Console clicks to pair pages whose traffic falls within roughly 15% of each other. Check intent as well as volume, and flip a coin for each pair so one URL gets the fix and the other stays in the holdout."}},{"@type":"Question","name":"Should I apply several fixes at once during the test?","acceptedAnswer":{"@type":"Answer","text":"No, pick one hypothesis before deployment. Rewriting the first 120 words into a direct answer, adding JSON-LD schema, and replacing client-side accordions with semantic HTML are three separate tests. If you apply them together, a positive result cannot tell you which change deserves a rollout and a negative one cannot tell you what to remove."}},{"@type":"Question","name":"How do I verify that AI bot traffic is genuine?","acceptedAnswer":{"@type":"Answer","text":"A user-agent string like GPTBot or PerplexityBot is not enough, since scrapers can claim those names. Use forward-confirmed reverse DNS, which Googlebot and Applebot support, or the IP-range files OpenAI and Perplexity publish, and exclude requests that fail the check."}},{"@type":"Question","name":"Does a verified ChatGPT-User fetch mean my page got cited?","acceptedAnswer":{"@type":"Answer","text":"No. A verified ChatGPT-User request is a leading indicator of live retrieval, but it does not show that your URL appeared as a citation in the answer. Use server access logs to measure fetches and separate answer checks to measure citations, recording GPTBot, OAI-SearchBot, and ChatGPT-User requests separately."}},{"@type":"Question","name":"How do I measure citations in live AI answers?","acceptedAnswer":{"@type":"Answer","text":"Before changing any page, write five specific user-intent prompts per page pair, giving 25 to 50 prompts for a 5 to 10-pair test. Every Tuesday at 10:00 AM UTC, run the same battery across Google AI Overviews, Perplexity Pro, and ChatGPT Search, recording 1 if the URL appears as an inline reference, card, or footnote and 0 otherwise."}},{"@type":"Question","name":"What should I do if a holdout page gets edited during the test?","acceptedAnswer":{"@type":"Answer","text":"Repair the page if it genuinely needs repairing, then remove that entire pair from the evaluation instead of adjusting for it. Assess only the uncontaminated pairs and state how many you used. A smaller clean comparison is more useful than a larger one whose pages no longer reflect the protocol."}}]}]} ```