LinkedIn
I opened our edge crawler log expecting a traffic story. I got a selection story instead. In the same 30 days, our Google Ads updates post for 2026 drew 651 verified AI-bot fetches, including 402 live-answer fetches. Our AI Overviews and ads explainer drew 405 total hits, only 18 verified fetches, and zero live-answer fetches.
Same site. Same month. Roughly the same effort to write. The gap was not word count. It came down to decisions about query fit, freshness and page shape that a standard content-gap report would not have flagged. But before we could see those decisions, we had to stop believing the wrong number.
The first sign: 405 hits that looked like attention
For two weeks, the Overviews post appeared to be getting picked up. The log showed 405 hits. If you stop at that number, the page looks healthy enough to leave alone. We nearly did.
Then we filtered for verified fetches on sitemap paths. The count collapsed to 18 verified fetches, with none in live answers. The rest of the hits were not evidence that an answer engine was using the page. A user agent is a claim, not proof: scrapers and credential scans can show up in the same raw traffic report as legitimate crawlers.
Our log separates the jobs those crawlers do. OAI-SearchBot surfaces sites in ChatGPT search, ChatGPT-User handles user-triggered fetches, and GPTBot is used for training. A GPTBot hit is not a search-answer fetch. Nor is a raw hit from a bot claiming to be OAI-SearchBot necessarily the real thing.
That distinction is not pedantry. In one 8-day log re-cut, only 13% of hits claiming to be GPTBot verified as OpenAI; the figures were 32% for OAI-SearchBot and 39% for ChatGPT-User. The same cut found 468 real fetches against 991 credential scans. That is why we count verified fetches per page, not raw bot hits.
The wrong call was treating those 405 hits as attention. Once we corrected it, the damage was plain: the neighbour post had spent the month looking active in a report while drawing no live-answer fetches. If your report shows hits but not verified fetches per page, it can make an untouched page look busy.
The split the verified log exposed
The updates post had 651 verified fetches. Of those, 402 were live-answer fetches: the page was pulled while an answer was being composed. The Overviews explainer had 18 verified fetches and zero live-answer fetches. Our category pages were crawled but never quoted.
A live-answer fetch is still not a citation. An Ahrefs analysis of 1.4M ChatGPT prompts found that ChatGPT cited only 49.98% of the roughly 47M URLs it retrieved. The log tells us which pages were reached for during answers; it does not let me call every one of those pulls a published citation.
It does give us a useful question. Why did the answer process repeatedly reach for one post and not its neighbour? I went back through the pages in the order a model would encounter them: the title and query, the date, then the material it could actually lift. The answer was less glamorous than a secret ranking factor.
The first page decision: a topic instead of a query
The Overviews post was a clear explainer with a vague job. Its title described a topic, AI Overviews and ads, but did not resemble the question a buyer would ask ChatGPT when they needed an answer that week. The updates post promised dated changes for 2026. That lines up with prompts about what changed, what is new and what to do now.
This matters before the body gets a chance. ChatGPT judges a title, URL and snippet before deciding whether to open a page. In that analysis, natural-language slugs were cited at 89.78% versus 81.11% for slugs without them. That does not prove a title caused our split. It does show why I would not dismiss the title as packaging.
I used to tell clients a clever title was fine if the content was strong. I was wrong about how much work a title can be asked to do here. If the page is meant to answer a current question, a title that names only the subject makes the model work out the question for itself. The updates post did not ask for that favour.
Put the question the page answers in the title. If I cannot hear a buyer asking it, I rewrite it before I polish another paragraph.
The second decision: explaining well, but leaving little to lift
The updates post was shaped for extraction. Its headings named specific 2026 changes; each section carried a date and a fact the model could use in a sentence. The Overviews explainer read well as continuous prose. It had longer blocks, a timeless frame and no date in the title path. To use it in a short answer, a model had more summarising to do.
Freshness also changes the field the page competes in. An analysis of 7,683 pages and 47,097 citations found that 75% of cited pages had been updated within the past year, with consistently cited pages averaging under six months. For commercial queries, more than 60% of citations went to content updated in the preceding six months. A dated updates post made its currency visible. The explainer made the reader, or the model, infer it.
The same goes for evidence. In a Princeton GEO test across 10k queries, adding statistics lifted visibility 41%, citing sources lifted visibility for lower-ranked pages 115%, and quotations added 30 to 40%. Keyword repetition cut visibility 10% against baseline. Those results do not assign a percentage of our traffic gap to any single edit. They do explain why a short, sourced fact is easier to use than three paragraphs that circle a point without stating it cleanly.
Our ignored page explained. The updates page supplied dated facts, named sources and lines that could stand on their own. That is a page-shape decision, not a writing-effort contest. Give each section one fact an answer could lift without rewriting the whole page.
The category pages showed the same failure in another form
Our category pages were fetched, then dropped. They were not direct answers to the sorts of questions the updates post addressed, and being crawled did not make them quotable.
There is a technical limit here, too. Most AI crawlers read raw HTML without executing JavaScript; in a 41-day test, GPTBot, ClaudeBot and PerplexityBot discovered zero pages linked only through JavaScript. If a useful fact or a route to it is buried behind client-side navigation, some crawlers will not find it in the first place. That is a different problem from a readable page that simply gives the model no direct answer.
Then there is the competition for space. ChatGPT cites around five sources per response, versus ten for Google AI Overviews and Perplexity; only 30% of brands stay visible from answer to answer. A broad category page has a harder job in that small field than a dated post that already contains a sentence suited to the answer.
The category pages kept us honest about the diagnosis. A crawl is the start of the selection process, not its finish. Fix access where access is broken; fix the answer where the page is readable but gives the model nothing specific to use.
The late fix: make the explainer easier to select
The updates post was not necessarily better writing. It gave the model a clearer path: a title close to the question, visible 2026 framing, and short facts beneath specific headings. The neighbour post asked the model to infer the question, infer the freshness and compress long explanations. Each request added friction.
I have seen the same pattern in ad accounts: the ad with the cleaner landing page wins, not because the product is better, but because the path has less friction. Here, the updates post cost less work to use. That is my reading of the split, not a claim that the log can isolate a single cause.
We made the fix late. We retitled the Overviews page around the question buyers actually ask, added the date to the title path and introduction, broke two long sections into dated subheads, and inserted the missing numbers with sources attached. The page started drawing verified OAI-SearchBot fetches within days. Not live-answer fetches yet.
That order matters. No fetch means no chance to be used in an answer. A fetch without an extractable answer still leaves the model with work to do. We cannot credit the rewrite with a citation we have not seen, but we can stop mistaking the old raw-hit total for progress.
Why the content-gap report gave us no warning
A conventional content-gap report would have treated both pages as covered. Same domain, overlapping keywords, no broken links and passing Core Web Vitals. None of that tells us whether a title fits a prompt, whether a page shows its currency, or whether an answer can be extracted from it.
Keyword-overlap tools miss prompt fit, extractability and freshness drift. Our ignored page did not need another keyword added to a paragraph. It needed a clearer question at the gate and a dated, sourced answer inside. That is why the gap did not show up as a content gap in the usual sense.
I ran technical audits for years and trusted the green scores. They are useful scores for the things they measure. They could not tell me why one readable post appeared in 402 live-answer fetches while another appeared in none. A crawlability pass is not an answer-selection pass.
How I would run this postmortem on another page
If you want to know which content gaps prevent your business from appearing in AI answers, start with URL-level evidence rather than a missing-keyword list. You need verified bot fetches, live-answer fetches, and the tracked prompts where your page was cited or absent. Inside groas, we look at tracked prompts, AI answers and citations, share of voice, AI crawler visits and referral traffic against the same pages.
I work through those checks in order because each one changes what I do next:
- Verified fetches per page over 30 days. Filter to reverse-verified OAI-SearchBot and ChatGPT-User on real paths. Ignore the raw hit total. If verified fetches are near zero, check access and internal links before rewriting the copy.
- Live-answer fetches. These show whether the page was pulled while an answer was composed. Verified fetches without live-answer pulls are a reason to inspect query fit and freshness, not proof that either one alone is broken.
- Prompt fit and the answer itself. Put the title beside the buyer prompts you track. If the page would not help answer them, rewrite the title and make the relevant facts easy to find, with their dates and sources attached. Then check whether the page is cited, rather than assuming a live fetch did the job.
For the technical side, I run the fetch test first: can OAI-SearchBot reach the URL, read the raw HTML and see the useful fact without JavaScript? A failed test calls for an access fix. A passed test sends me back to the question and the page shape. A keyword report treats those as the same problem; the log does not.
The rule above my monitor is still the shortest version of this postmortem: no page ships without a prompt in the title and a fact in every section. If I cannot point to the question it answers and the sentence an answer could lift, I do not call the page finished.