AI Crawler & Search Bot Access
Last updated: 1st July 2026
This page explains which automated crawlers are permitted to access thehosemaster.co.uk, and why.
We allow access to the major AI and search crawlers so that our product data, technical guides, and specifications stay visible in AI-generated answers, which is where a growing share of trade research now happens. Full technical rules are published in our robots.txt file. This page is a plain-English summary.
Bots we allow:
| Provider | Bot | Purpose |
|---|---|---|
| OpenAI | GPTBot | Training data collection for OpenAI's models |
| OpenAI | OAI-SearchBot | Indexing for ChatGPT Search |
| OpenAI | ChatGPT-User | Fetches a page when a ChatGPT user asks about it directly |
| OpenAI | OAI-AdsBot | Validates ad landing pages submitted to ChatGPT |
| Anthropic | ClaudeBot | Training data collection for Claude |
| Anthropic | Claude-User | Fetches pages when a Claude user asks a direct question |
| Anthropic | Claude-SearchBot | Indexing to improve Claude's search results |
| Perplexity | PerplexityBot | Indexing for Perplexity's search engine |
| Perplexity | Perplexity-User | Real-time fetch when a Perplexity user asks a question |
| Google-Extended | Training data for Gemini and Vertex AI, separate from standard Google Search indexing | |
| Apple | Applebot-Extended | Training data for Apple's AI features |
| Amazon | Amazonbot | Indexing for Amazon's AI and search products |
| Meta | meta-externalagent | Indexing for Meta's AI products |
Standard search engines, including Google and Bing, are handled separately and are always permitted to crawl and index our content.
This list is reviewed periodically as providers introduce new crawlers or retire old ones.
Questions about our crawler policy? Contact our team today.