# Search engines: sitemap, robots.txt, llms.txt

URL: https://docs.xodia.ai/seo/search-engines
Last updated: 2026-11-10

Search engines settings decide how search engines and AI bots see your site: its address, which crawlers may read it, its sitemap.xml, robots.txt and llms.txt, and the codes that prove to Google and Bing that the site is yours. They are free on every plan.

Open Settings in the rail and choose Search engines, or choose Sitemap and robots.txt in the SEO tab. Each card has its own Save. The settings are saved in your site's code, in a file called `xodia.seo.json`, so like every edit they stay a draft until you save and publish, show in Activity and can be restored. Because they live in the code, they work on [Xodia hosting](https://docs.xodia.ai/hosting) and on your own host, such as Vercel, Netlify or Cloudflare.

## Site address

The site address is the full address of your live site, such as `https://example.com`. sitemap.xml and llms.txt list full addresses on it, and robots.txt names the sitemap with it.

- On Xodia hosting it is set for you: your custom domain, or your xodia.site address. Change it in Domains.
- On your own host, type it in. If your host is connected, Xodia fills in its live address for you to save. Without an address, a site on your own host gets no sitemap.xml.
- It is the same address as the one in [Languages](https://docs.xodia.ai/languages#seo) settings: change it in one place and it changes in both.

## Crawlers

Three switches, all on to start with:

- **Search engines**: on, Google, Bing and other search engines can find and list this site. Off, robots.txt asks every crawler to stay away, every page tells search engines not to index it, and on Xodia hosting the live site says so in its answers too.
- **AI training bots**: on, bots that collect pages to train AI models, such as GPTBot, ClaudeBot and Google-Extended, can read this site. Off, robots.txt asks them to stay away.
- **AI search bots**: on, AI search tools can read this site and link to it in their answers. Off, robots.txt asks them to stay away, so the site stays out of their answers.

Turning search engines off asks you to confirm first. While it is off, both AI switches are blocked with every other crawler.

> **Warning:** Pages already in Google and Bing drop out over the next days or weeks after you turn search engines off. You can turn it back on any time; coming back takes a while too.

## Sitemap

sitemap.xml lists your site's addresses for search engines. Choose how it is made:

- **Automatic** (recommended): Xodia lists every page and every CMS item page on each build. Pages hidden from search and drafts are left out, and a site with [languages](https://docs.xodia.ai/languages) lists each page in every language.
- **My own sitemap.xml**: you write the file and Xodia ships it as it is. Xodia checks it as you type: that it is well-formed, and that its addresses are on your site's own address.
- **No sitemap**: no sitemap.xml, and robots.txt doesn't name one. Search engines find pages through links only.

With Automatic, the card shows the addresses sitemap.xml will hold as you work, CMS items marked with their collection. Below them, Left out lists the pages that aren't in it and why: Hidden from search, Draft, Redirect, No fixed address or Needs a site address, with a link to the page's settings or the CMS item where there is one. Open sitemap.xml opens the live file, and Copy address copies its address.

To keep a page out of the sitemap, hide it from search in its [page settings](https://docs.xodia.ai/seo#page-settings): a page hidden from search is never in the sitemap. Page settings show a line under that switch, In sitemap.xml or left out and why, and the Plan tab's sitemap marks pages that aren't in sitemap.xml.

If your site already has its own sitemap.xml, it is used instead of the automatic one. Use the automatic one removes your file, after you confirm.

## robots.txt

robots.txt tells crawlers what they may read. The card shows the file Xodia writes from your Crawlers switches, as crawlers will read it at /robots.txt.

- List the sitemap in robots.txt adds a Sitemap line, so crawlers find every page without you submitting it.
- Your own robots.txt rules are added as written, after Xodia's rules, for example to keep crawlers out of one folder.
- On Xodia hosting, publish previews and a xodia.site address next to a custom domain always ask crawlers to stay away.

### A site with its own robots.txt

An imported site often has its own robots.txt. It wins: it is used as it is, and the crawler switches don't change it. The card lets you edit that file in place, or choose Let Xodia write it: your rules move into Your own rules, your file goes, and the switches work from then on. Turning search engines off still keeps the site out of search on Xodia hosting.

## llms.txt

llms.txt is a short guide to your site for AI tools, at /llms.txt. Choose how it is made:

- **Automatic** (recommended): on each build Xodia writes the site's name, one summary line, then each public page with its title and description, and CMS item pages grouped by their collection. A site with languages names its other languages in one line. When AI training bots are blocked, the file says the site's content may not be used to train AI models.
- **My own llms.txt**: you write the file and Xodia ships it as it is.
- **Off**: no llms.txt. AI tools read the pages themselves.

The summary is one line under the site name. Leave it empty to use your home page's description. Pages hidden from search and drafts are left out, the same as in the sitemap.

Automatic writes no llms.txt while search engines are off, or while AI search bots are blocked: it is a guide for exactly those readers. The card says so, and the file comes back when you turn them on. If your site already has its own llms.txt, it ships instead of the automatic one. Open llms.txt opens the live file.

## Verification

Google Search Console and Bing Webmaster Tools ask you to prove the site is yours before they show its data.

1. Copy the code

   In Google Search Console or Bing Webmaster Tools, add your site and choose the HTML tag method. Copy the code, or the whole meta tag.
2. Paste it in Verification

   Paste it in the Google or Bing field and save. Xodia keeps the code and adds the tag to every page of your site. It is free on every plan and doesn't need Custom code.
3. Publish, then verify

   Once the site is live with the tag, choose Verify in Google or Bing. Then add your sitemap's address in Google Search Console; the card shows it.

For a new site from start to finish, see [Get found on Google](https://docs.xodia.ai/get-started/get-found-on-google).

## Xodia hosting and your own host

On Xodia hosting, every site gets these files on each build. On your own host, sites made from the Xodia template, Marketplace templates included, write the same robots.txt, llms.txt and verification tags in their own build, and sitemap.xml once the site has its address. For an imported site Xodia can't build them for, the card says so and Ask your agent has your AI add them to the site's code.

## For your AI

Your AI changes these settings by editing `xodia.seo.json` at the root of the site's code. A site without the file uses these defaults:

xodia.seo.json:

```json
{
  "siteUrl": "",
  "sitemap": { "mode": "auto" },
  "robots": {
    "searchEngines": true,
    "aiTraining": true,
    "aiSearch": true,
    "listSitemap": true,
    "extra": ""
  },
  "verification": {
    "google": "",
    "bing": ""
  },
  "llms": {
    "mode": "auto",
    "summary": ""
  }
}
```

The MCP tool `get_seo_report` includes the site's search engine state: its address, the sitemap and llms.txt modes, its own files, blocked AI bots, verification and any problems. The links check flags problems too, such as search engines turned off on a live custom domain or a sitemap that lists a page hidden from search. See the [MCP server reference](https://docs.xodia.ai/reference/mcp).