llms.txt is a short Markdown file at the root of your website that summarises who you are and links to your most useful pages, so AI tools can read your site quickly. AI crawlers are the bots that AI companies send to read the web. To be the answer when customers ask an AI, let the right crawlers in, give them clear pages, and treat llms.txt as a helpful extra rather than a magic switch.

This article is part of our starter guide to SEO for small businesses in the UAE. For the wider picture of being cited by assistants, see getting your business cited by AI search.

What llms.txt is

The llms.txt proposal was published by Jeremy Howard in September 2024 and updated to a second version in August 2026. It suggests a Markdown file at /llms.txt that gives AI agents a concise, curated overview of a site, because ordinary web pages are wrapped in menus, scripts and adverts that are hard for a model to read.

The format is simple. In order:

  1. An H1 with the name of the business or site (the only required part).
  2. A blockquote with a short summary of what you do.
  3. Optional paragraphs or lists with more detail.
  4. Sections under H2 headings, each a list of links with a short note.

A minimal example for a clinic might look like this:

# Example Family Clinic

> Family medicine and physiotherapy clinic in Jumeirah, Dubai. Open Saturday to Thursday. Appointment requests by WhatsApp or phone; our team confirms every booking.

## Services
- [Physiotherapy](https://example.ae/physiotherapy/): assessment, treatment plans, follow-up sessions
- [Family medicine](https://example.ae/family-medicine/): check-ups and referrals

## Practical
- [Location and hours](https://example.ae/contact/): address, parking, opening hours
- [Insurance](https://example.ae/insurance/): accepted insurance providers

Keep it factual. The file is a summary of your site, so every line should match what your pages say.

Does llms.txt help you rank or get cited?

Be realistic here. Google states on its page about AI features and your website that you don’t need new machine-readable files, AI text files or markup to appear in AI Overviews or AI Mode. So llms.txt does not change how Google’s AI features treat you.

Other tools may read it. The proposal’s author notes that thousands of sites now publish one and that AI companies publish llms.txt files for their own developer documentation. It costs little to create and keep up to date, so it is a reasonable extra once the basics are in place, not a replacement for them.

Which AI crawlers visit your site

AI companies run several different bots, and each has a different job. Blocking the wrong one can quietly remove you from AI answers. These are the main ones, as described on each company’s own pages:

Company Bot (robots.txt name) What it is for
OpenAI OAI-SearchBot Showing sites in ChatGPT search results
OpenAI GPTBot Collecting content that may be used to train models
OpenAI ChatGPT-User Visiting a page when a user asks ChatGPT to
Anthropic Claude-SearchBot Indexing content to improve Claude’s search answers
Anthropic ClaudeBot Collecting content that may be used for model training
Anthropic Claude-User Fetching a page for a Claude user’s request
Perplexity PerplexityBot Surfacing and linking sites in Perplexity search results
Perplexity Perplexity-User Visiting a page to answer a user’s question
Google Google-Extended A control token for Gemini training and grounding; it does not affect Google Search

The pattern is the same across companies: search bots decide whether you can appear and be linked in answers, training bots collect content for models, and user bots fetch a page because someone asked. OpenAI’s documentation, for example, says a site can allow OAI-SearchBot to appear in search results while disallowing GPTBot to opt out of training.

How to set your robots.txt

Decide two things separately: do you want to appear in AI answers, and are you comfortable with your content being used for training? Most small businesses want the first. The second is a business choice.

An example that welcomes AI search bots but opts out of training:

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Three checks before you change anything:

Measure AI crawler visits and referrals

Two different things are worth tracking:

  1. Crawler visits: your server logs or CDN analytics show which bots fetch which pages. If OAI-SearchBot or Claude-SearchBot never appear, check for blocks.
  2. Human visits from AI tools: in Google Analytics, look for referrers such as chatgpt.com, perplexity.ai and claude.ai, and track WhatsApp, phone and form clicks as events so you can see which visits turn into enquiries.

Then test by asking. Every few months, ask several assistants the questions your customers ask, such as “Where can I get weekend physiotherapy in Jumeirah?”, and note whether you appear and whether the facts are right. If an assistant gets a fact wrong, find where that wrong fact lives (an old listing, an outdated page) and fix it at the source.

Make your pages easy to quote

Letting crawlers in only helps if what they find is clear. Pages that tend to be cited:

For clinics and other regulated businesses, stay factual and cautious. Assistants repeat what you publish, so avoid claims you could not support.

How Dub Ai helps

The Dub Ai team’s SEO and content service includes a full website audit covering technical health, speed, content, structured data and local signals, with a prioritised list of fixes; where the team built and runs your site, many fixes can be applied directly. The SEO agent then drafts articles around the questions customers ask, with titles and meta descriptions in English and Arabic, and you confirm each one on WhatsApp before it is published.

Frequently asked questions

Do I need an llms.txt file to appear in AI answers?

No. Google says no special AI text files are needed for its AI features, and other assistants mainly rely on what their search bots find on your pages. An llms.txt file is a useful summary that some AI tools may read, so it is worth adding once your pages are clear and crawlable, but it is an extra, not a requirement.

Will blocking GPTBot remove me from ChatGPT?

According to OpenAI’s documentation, GPTBot is for collecting content that may be used in training, while OAI-SearchBot controls whether your site can appear in ChatGPT search results. You can disallow GPTBot and still allow OAI-SearchBot. Check your firewall or CDN settings too, because a block there can override robots.txt.

How often should I update llms.txt?

Update it whenever the facts it lists change: a new service, a new branch, new opening hours or a changed contact number. A quick check every quarter is sensible. An out-of-date summary is worse than none, because assistants may repeat the old details to customers.

Book your demo on WhatsApp: +971 56 552 6662

Sources

Leave a Reply

Your email address will not be published. Required fields are marked *