llms.txt is a short Markdown file at the root of your website that summarises who you are and links to your most useful pages, so AI tools can read your site quickly. AI crawlers are the bots that AI companies send to read the web. To be the answer when customers ask an AI, let the right crawlers in, give them clear pages, and treat llms.txt as a helpful extra rather than a magic switch.
This article is part of our starter guide to SEO for small businesses in the UAE. For the wider picture of being cited by assistants, see getting your business cited by AI search.
What llms.txt is
The llms.txt proposal was published by Jeremy Howard in September 2024 and updated to a second version in August 2026. It suggests a Markdown file at /llms.txt that gives AI agents a concise, curated overview of a site, because ordinary web pages are wrapped in menus, scripts and adverts that are hard for a model to read.
The format is simple. In order:
- An H1 with the name of the business or site (the only required part).
- A blockquote with a short summary of what you do.
- Optional paragraphs or lists with more detail.
- Sections under H2 headings, each a list of links with a short note.
A minimal example for a clinic might look like this:
# Example Family Clinic
> Family medicine and physiotherapy clinic in Jumeirah, Dubai. Open Saturday to Thursday. Appointment requests by WhatsApp or phone; our team confirms every booking.
## Services
- [Physiotherapy](https://example.ae/physiotherapy/): assessment, treatment plans, follow-up sessions
- [Family medicine](https://example.ae/family-medicine/): check-ups and referrals
## Practical
- [Location and hours](https://example.ae/contact/): address, parking, opening hours
- [Insurance](https://example.ae/insurance/): accepted insurance providers
Keep it factual. The file is a summary of your site, so every line should match what your pages say.
Does llms.txt help you rank or get cited?
Be realistic here. Google states on its page about AI features and your website that you don’t need new machine-readable files, AI text files or markup to appear in AI Overviews or AI Mode. So llms.txt does not change how Google’s AI features treat you.
Other tools may read it. The proposal’s author notes that thousands of sites now publish one and that AI companies publish llms.txt files for their own developer documentation. It costs little to create and keep up to date, so it is a reasonable extra once the basics are in place, not a replacement for them.
Which AI crawlers visit your site
AI companies run several different bots, and each has a different job. Blocking the wrong one can quietly remove you from AI answers. These are the main ones, as described on each company’s own pages:
| Company | Bot (robots.txt name) | What it is for |
|---|---|---|
| OpenAI | OAI-SearchBot | Showing sites in ChatGPT search results |
| OpenAI | GPTBot | Collecting content that may be used to train models |
| OpenAI | ChatGPT-User | Visiting a page when a user asks ChatGPT to |
| Anthropic | Claude-SearchBot | Indexing content to improve Claude’s search answers |
| Anthropic | ClaudeBot | Collecting content that may be used for model training |
| Anthropic | Claude-User | Fetching a page for a Claude user’s request |
| Perplexity | PerplexityBot | Surfacing and linking sites in Perplexity search results |
| Perplexity | Perplexity-User | Visiting a page to answer a user’s question |
| Google-Extended | A control token for Gemini training and grounding; it does not affect Google Search |
The pattern is the same across companies: search bots decide whether you can appear and be linked in answers, training bots collect content for models, and user bots fetch a page because someone asked. OpenAI’s documentation, for example, says a site can allow OAI-SearchBot to appear in search results while disallowing GPTBot to opt out of training.
How to set your robots.txt
Decide two things separately: do you want to appear in AI answers, and are you comfortable with your content being used for training? Most small businesses want the first. The second is a business choice.
An example that welcomes AI search bots but opts out of training:
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
Three checks before you change anything:
- Look for accidental blocks. Some security plugins, hosting firewalls and CDN bot settings block AI bots by default. A rule in your firewall can override a friendly robots.txt.
- Never block Googlebot while trying to block AI tools. Google-Extended is the separate token for Gemini; Googlebot is your search listing.
- Allow time. Companies say changes can take a while to be picked up; Perplexity, for example, mentions up to 24 hours.
Measure AI crawler visits and referrals
Two different things are worth tracking:
- Crawler visits: your server logs or CDN analytics show which bots fetch which pages. If OAI-SearchBot or Claude-SearchBot never appear, check for blocks.
- Human visits from AI tools: in Google Analytics, look for referrers such as chatgpt.com, perplexity.ai and claude.ai, and track WhatsApp, phone and form clicks as events so you can see which visits turn into enquiries.
Then test by asking. Every few months, ask several assistants the questions your customers ask, such as “Where can I get weekend physiotherapy in Jumeirah?”, and note whether you appear and whether the facts are right. If an assistant gets a fact wrong, find where that wrong fact lives (an old listing, an outdated page) and fix it at the source.
Make your pages easy to quote
Letting crawlers in only helps if what they find is clear. Pages that tend to be cited:
- Answer the question in the first two sentences, then add the detail.
- State facts as text: services, areas, hours, languages spoken, how to book.
- Include short FAQs written in your customers’ words.
- Use structured data that matches the visible text.
- Keep the same name, address and phone number everywhere online.
For clinics and other regulated businesses, stay factual and cautious. Assistants repeat what you publish, so avoid claims you could not support.
How Dub Ai helps
The Dub Ai team’s SEO and content service includes a full website audit covering technical health, speed, content, structured data and local signals, with a prioritised list of fixes; where the team built and runs your site, many fixes can be applied directly. The SEO agent then drafts articles around the questions customers ask, with titles and meta descriptions in English and Arabic, and you confirm each one on WhatsApp before it is published.
Frequently asked questions
Do I need an llms.txt file to appear in AI answers?
No. Google says no special AI text files are needed for its AI features, and other assistants mainly rely on what their search bots find on your pages. An llms.txt file is a useful summary that some AI tools may read, so it is worth adding once your pages are clear and crawlable, but it is an extra, not a requirement.
Will blocking GPTBot remove me from ChatGPT?
According to OpenAI’s documentation, GPTBot is for collecting content that may be used in training, while OAI-SearchBot controls whether your site can appear in ChatGPT search results. You can disallow GPTBot and still allow OAI-SearchBot. Check your firewall or CDN settings too, because a block there can override robots.txt.
How often should I update llms.txt?
Update it whenever the facts it lists change: a new service, a new branch, new opening hours or a changed contact number. A quick check every quarter is sensible. An out-of-date summary is worse than none, because assistants may repeat the old details to customers.
Book your demo on WhatsApp: +971 56 552 6662
Sources
- The /llms.txt file, v2 (Jeremy Howard, published 3 September 2024, modified 10 August 2026)
- AI features and your website (Google Search Central, updated 10 December 2025)
- Overview of OpenAI crawlers (OpenAI)
- Does Anthropic crawl data from the web, and how can site owners block the crawler? (Anthropic)
- Perplexity Crawlers (Perplexity)
- Google’s common crawlers (Google Search Central, updated 14 July 2026)