How We Structure Next.js Websites for Search and AI Crawlers

Most advice about “AI-ready websites” is abstract. This is the opposite: a list of the decisions we made building poi369.com on Next.js 15, why we made them, and what they cost. The same patterns run in POITIM and Faltastik. Nothing below requires a plugin or a paid service.

1. Meaningful HTML first, JavaScript for interaction

Every page is a React Server Component by default. Headings, intros, service descriptions, case studies, author information and question-and-answer sections are rendered on the server and arrive in the first response. Client components are limited to what needs the browser: page transitions, the sticker animations, the cookie banner, the package builder, forms.

We considered the opposite approach, a small HTML shell that fetches content from an API, to shrink the document. We rejected it for anything commercial or editorial. Google can execute JavaScript; OAI-SearchBot, PerplexityBot and most snippet extractors are less patient, and a page whose answer is in a fetch response is invisible to them. The cost of the choice is a larger HTML document; the benefit is that every reader, human or machine, gets the same content.

2. Markdown for agents, negotiated by Accept header

A request to any page with `Accept: text/markdown` is rewritten in middleware to an internal route that renders the same page as Markdown. The URL does not change, browsers never send that header so the HTML path is untouched, and the Markdown is generated from the same content modules and i18n messages the HTML uses. There is no second copy to drift. The response carries `Vary: Accept`, a cache header, and `X-Robots-Tag: noindex` so the text version is never indexed as a duplicate.

This exists because a growing number of agents and retrieval pipelines prefer clean text over a DOM. It is cheap to maintain precisely because it is derived, and it gave us a side benefit: `llms-full.txt` is the concatenation of those Markdown renders for the key English pages.

3. robots.txt with explicit crawler groups and Content-Signal

Next.js can generate robots.txt from a typed object, but it cannot emit non-standard directives. We serve the file from a route handler instead, with one group for `*` and one for each AI crawler we want to address by name (GPTBot, OAI-SearchBot, ChatGPT-User, CCBot, PerplexityBot, anthropic-ai, ClaudeBot, Google-Extended). Each group carries `Content-Signal: search=yes, ai-input=yes, ai-train=yes`. We chose all three deliberately: the business goal is to be found and cited, and declaring `ai-train=no` while allowing training crawlers would have been a contradiction.

4. Sitemap dates derived from content signatures

A hand-maintained `lastmod` table lies within weeks: pages change and dates do not. Ours is generated before every build. For each route, a script hashes the source files and the relevant i18n message subtree; if the signature differs from the committed one, the date becomes today, otherwise it is preserved. The result is committed, so a shallow clone on the build server produces the same dates as a local checkout. Google is told a page changed only when it did.

The sitemap itself is also a route handler, because the built-in generator cannot attach the XSL stylesheet we use for a human-readable view. Every route emits one `<url>` per language, each with the full hreflang alternate set, and the language list is read from the routing configuration. We learned that lesson the hard way: when three languages were added, a hand-written locale list meant the new pages never entered the sitemap.

5. Six languages, each self-canonical

• ▸Turkish is the default and has no URL prefix; the other five are prefixed. Route names are localised per language (`/hizmetler`, `/en/services`, `/et/teenused`) from one routing table that also drives links, canonical URLs, hreflang and the sitemap.
• ▸Arabic and Kurdish paths use ASCII transliteration, because percent-encoded script in shared URLs is unreadable.
• ▸Automatic locale redirection is off. With it on, the Turkish home page was never served at its own URL to clients sending `Accept-Language: en`; even performance tools measured the English version.
• ▸Where a body exists in fewer languages than the site (the blog is written in three), `inLanguage` reports the real language of the text, and hreflang for the untranslated locales points at the English URL instead of inventing pages that would 404.
• ▸Pages that exist only in English, like our commercial cluster, declare hreflang for the languages they actually have, with x-default pointing at English. The metadata helper takes an explicit list of available locales for this reason.

6. Structured data as one graph

The root layout emits Organization and WebSite nodes with stable identifiers (`/#organization`, `/#website`). The founder is a Person node with its own identifier (`/#founder`). Every other node refers to these by @id: articles have `author` and `publisher` references, services have a `provider` reference, the profile page wraps the same Person in a ProfilePage, the about page is an AboutPage whose `mainEntity` is the organisation. Commercial pages emit a small `@graph` with WebPage, Service and BreadcrumbList.

We put in only what is visible and verified: legal name, address, founding month, social profiles that exist, services that are sold, projects that are live. No aggregate ratings, no partner badges. FAQ structured data is no longer added to new pages since Google retired the rich result; the question-and-answer sections stay as plain HTML.

7. llms.txt generated, not written

Our first `llms.txt` was a static file, and within a month it described the wrong number of languages and pages. It is now a route handler that reads the same registries as the sitemap (services, commercial pages, case studies, insights, team) and prints canonical URLs with one-line descriptions. It is a courtesy index for agents, not a ranking lever, and it can never disagree with the site again.

8. IndexNow after deploy, from CI

There is no CMS; publishing is a git push that Vercel builds. So the notification step lives in a GitHub Actions workflow: on push it waits until the live site reports the new commit hash in a small build-info file, computes which routes changed by comparing content signatures between the previous and current commit, expands them to every language URL, and posts the list to IndexNow. Bing, Copilot and partner engines learn about a change within minutes; Google still relies on sitemap lastmod, which the same signatures keep honest. The key is served from a route that returns 404 when the environment variable is missing, so local and preview builds never leak or fail.

9. Security and performance are not traded for crawlers

A strict Content Security Policy, HSTS with preload, frame and referrer policies ship on every route. Production has no `unsafe-eval`. Crawlers do not need weaker headers; they need HTML. On performance, the critical CSS is inlined to remove a render-blocking request, analytics loads after `window.load`, hero assets are WebP at the size they are displayed, and reduced-motion users get a static page. The Lighthouse target of 90+ is tested, not assumed.

What we would still change

The blog is written in three languages and served in six with English fallback; we would rather translate the pieces that earn traffic than machine-translate all of them. The manual monthly visibility benchmark should eventually be assisted by tooling, within each engine’s terms. And the commercial cluster exists in English only; the Turkish and Estonian versions are the next content decision, not a routing change, because the model already allows any page to gain a language without touching configuration.

The mechanics above are what our AI search optimization audit checks on other sites, and what the web development work ships by default.