How AI Search Discovers, Understands and Cites a Brand

When someone asks an assistant “which agency in Tallinn can build an AI product?”, the answer is not pulled from a database of agencies. It is assembled, at that moment, from documents the system can reach and trust. Understanding the assembly line is the difference between GEO as superstition and GEO as engineering. This piece describes the pipeline as it is publicly documented and as we observe it in our own benchmarks. Where something is inferred rather than documented, we say so.

Stage 1: crawling. Can the engine read the page at all?

Each system has its own crawlers, and they are documented. OpenAI operates three: GPTBot (training data), OAI-SearchBot (the index behind ChatGPT search) and ChatGPT-User (fetches triggered by a user in a conversation). Perplexity uses PerplexityBot. Anthropic uses ClaudeBot and the older anthropic-ai token. Google’s AI features use the ordinary Googlebot; Google-Extended is a separate control that affects Gemini training, not whether a page appears in AI Overviews or AI Mode. Microsoft Copilot sits on Bing’s index, crawled by Bingbot. Common Crawl’s CCBot feeds many models indirectly.

This stage is where most accidental invisibility happens. Bot-management presets on CDNs and firewalls often block “AI crawlers” as a category. A robots.txt copied from a template disallows GPTBot. A site that renders its service descriptions from a client-side API returns an empty shell to a crawler that does not execute JavaScript. None of these show up in Google Search Console, because Googlebot is fine.

Stage 2: retrieval. Which passages are candidates for this question?

For a given question, the engine does not read the whole web. It runs one or more searches (its own index, a partner index, or both), takes the top results, splits them into passages, and scores each passage for relevance to the question. Some systems embed passages and compare vectors; some rely on the underlying search engine’s ranking; most do both. The details are proprietary, but the shape is consistent: a page enters the answer only if a passage from it ranks well for the specific question.

Two consequences follow. First, ranking in classic search still matters enormously, because it decides the candidate pool for several engines. Second, the passage has to be self-contained. A paragraph that says “we offer these services” followed by a bulleted list of nouns scores worse than a paragraph that states who the company is, where it is, and what it does in one breath. This is why our commercial pages open with a direct answer of forty to eighty words rather than a hero slogan.

Stage 3: entity resolution. Who is this passage about?

The model now holds a handful of passages from different sites and has to turn them into a coherent answer. If three passages describe the same company under three names, the model may treat them as three weak candidates instead of one strong one. If the website says Tallinn and a directory says Tartu, the model may hedge on location or skip the brand. Language models are good at reconciling obvious variants, but they reward sources that make reconciliation unnecessary.

Structured data helps here in a specific way. Organization, Person, Service and Article data connected by @id references give a crawler an unambiguous statement of the entity graph: this company, this legal name, this founder, these services, these projects. It does not make a model cite you. It removes the reasons a model would not.

• ▸One canonical brand name, with variants declared as alternateName rather than used interchangeably.
• ▸One legal entity and one public address, identical on the site, in structured data, on LinkedIn and in every directory.
• ▸A founder or team page that connects person to company to expertise, with sameAs links to real profiles.
• ▸Services and projects that link to each other, so the graph has edges, not just nodes.

Stage 4: citation selection. Which sources get named?

Engines cite differently. Perplexity cites inline and generously. ChatGPT search shows a small set of sources per answer. Google AI Mode and AI Overviews link a handful of pages, often weighted toward sources Google already ranks highly. Copilot follows Bing. What they share, as far as we can observe, is a preference for sources that are specific, dated, attributed and consistent with each other. A page with an author, a review date and a claim backed by a case study is easier to cite than an anonymous page with the same words.

Third-party corroboration matters most at this stage. If the only source saying a company is an AI agency is the company itself, the model has one data point. If a directory, a partner site and an interview say the same thing, the model has agreement, and agreement is what these systems treat as reliability. This is the honest reason to spend time on directory profiles and partner mentions: not link equity, but independent confirmation.

Where brands get dropped, in our experience

1. 01Blocked or empty at crawl time. The most common and the cheapest to fix. Ten minutes in robots.txt and a firewall rule.
2. 02Present but not retrievable. The page ranks on Google for the brand name but has no passage that answers a category question, so it never enters the candidate pool for “best AI agency in Tallinn”.
3. 03Retrieved but ambiguous. Naming and location inconsistencies make the model hedge. Fixed with entity work, slowly.
4. 04Understood but not cited. The brand is mentioned generically (“several Tallinn agencies offer this”) but not named, because no independent source confirms it. Fixed with evidence and corroboration, slower still.

How we test this

The pipeline above is the model behind our monthly benchmark: a fixed set of brand, commercial and conversational prompts run across ChatGPT, Gemini, Perplexity and Google AI Mode, recording mention, citation, position, competitors and the source URL the engine used. The source URL is the most informative field, because it tells you which stage worked: if the engine cites a directory rather than your site, the site failed at retrieval or resolution while the directory succeeded. The full method, the prompt set and our own results are published in the AI Search Lab as they come in.

For the engineering side of stage 1 and stage 2, see how we structure Next.js websites for search and AI crawlers. For a step-by-step checklist, see how to improve brand visibility in ChatGPT and AI search.