Choose how crawlers discover your pages
Keep your sitemap, crawler instructions and indexing policy tied to the site configuration. Public launches and preview environments can make different publication choices without maintaining separate hand-written policy files.
Example — Prepare a site before opening it to search
A preview stays marked noindex while the team reviews its pages. The production configuration deliberately permits indexing when the site is ready.
For engineers
The site’s seo.robots.policy controls the index/noindex instruction. Current head rendering emits a noindex meta directive; the robots renderer allows crawling so compliant crawlers can read it. Explicit path exclusions and bot-specific policies govern crawling separately. These are not access-control mechanisms.
Wonder Todos exposes the builder through an Astro endpoint. This is its selected response handler; the file imports APIRoute, renderRobotsTxt from the website Astro entrypoint, and its site context:
export const GET: APIRoute = () =>
new Response(renderRobotsTxt(WEBSITE_SITE_CONTEXT), {
headers: {
'Content-Type': 'text/plain; charset=utf-8',
'Cache-Control': 'public, max-age=3600',
},
});
Publish the sitemap from the same routes
The application’s Astro sitemap integration reads the canonical origin and locale table from the website configuration. Page manifests give the expected route inventory; compare that inventory with the emitted sitemap, including non-default locale paths.
Choose either an allowed-crawler list or a disallowed-crawler list; the configuration rejects both together. Validate the deployed robots response and page head, because a local configuration value is not a delivered policy. Keep private material behind authorization rather than relying on crawler instructions.