Microtraining SearchBrand #1
Host: Hussam Sufan
Guest Expert: Francisco Kemeny
Date: October 8, 2025
Duration: ~55 minutes
TL;DR — The Essentials
LLMs don't "search" like Google: they use base training, RAG, and tool calling to generate responses
The llms.txt file is an emerging convention that improves AI response precision by up to 70%
You must configure robots.txt to enable specific crawlers: GPTBot (training) and ChatGPT-User (real-time searches)
GPTBot indexes for future models (6-12 months), ChatGPT-User for immediate searches (days/weeks)
llms.txt removes HTML and code, leaving only structured content in Markdown for LLMs
Implementing llms.txt is simple: create the file at the domain root and reference it in HTML
You can measure impact by creating segments in Analytics for traffic from AI crawlers
The Context: 60% of Consumers Already Use AI for Shopping
We are at an inflection point. 60% of consumers in the United States already use artificial intelligence in their purchasing process, and 46% trust ChatGPT’s recommendations more than those from friends or acquaintances.
Although these numbers are from the United States, the trend is rapidly reaching Latin America. If your site is not technically prepared for AI models to read it, extract information, and cite it correctly, you are invisible to a growing portion of your potential audience.
The big question: How do we make ChatGPT, Gemini, Perplexity, and Claude understand our content and cite us correctly?
The answer is not only strategic (as we saw in the launch webinar) but also technical. And that's exactly what we cover in this microtraining.
How LLMs Really "Search"
Important disclaimer: LLMs technically do not search. Unlike Google, which indexes content in a directory and ranks it, language models predict tokens (words) based on patterns learned from massive data.
However, for practical purposes of this conversation, we will talk about "searching" in LLMs, understanding that it works completely differently from traditional search engines.
Three ways LLMs access information:
1. Model Base Training
All the information collected to train the model. When we talk about "parameters" (the hundreds of billions of parameters of GPT-4, for example), we refer to this knowledge base.
Practical example: If you ask ChatGPT "Who was the president of El Salvador in 2020?" it can answer directly without searching because that information is in its training.
The cut-off date: Each model has a knowledge cutoff date. GPT-4 knows up to April 2023, for example. Everything subsequent to that date is NOT in its base training.
2. RAG (Retrieval Augmented Generation)
The model brings external information and places it in its "context" to generate a response. It is not browsing or indexing: it is taking blocks of information and using them temporarily.
It's like passing a complete PDF in the prompt and telling it "based on this, answer X". The model doesn't save that PDF; it only uses it for that conversation.
3. Tool Calling (use of tools)
Models and agents can use external tools. For example:
Google Search API (SERP): ChatGPT, Gemini, and Perplexity can call Google, get the first 10 results, and use that context
Web browsing: They can enter specific sites, read the full HTML, and extract information
Web scraping: Some models browse pages and extract structured data in real time
Key concept: Accessibility
If your site blocks crawlers, your content will NOT be:
In the base training of future models
Available for real-time browsing
Accessible for extraction via RAG or tool calling
That's why Wikipedia appears so much: It is completely public, with no access restrictions. Crawlers have indexed all its information millions of times.
What is the llms.txt file?
The llms.txt is a convention (still unofficial) that emerged approximately a year ago to give language models a simpler way to read your content.
The problem it solves
Imagine an LLM entering your website. It encounters:
Thousands of lines of HTML
JavaScript for interactivity
CSS for styles
Tracking tags (Google Analytics, Facebook Pixel, etc.)
Navigation, footers, sidebars
Popups, modals, forms
In the midst of all that code, there are a few paragraphs with the actual content you want the model to read.
The problem: The model spends tokens (computational capacity) cleaning and filtering HTML to find the relevant text. This reduces the quality of the response and the accuracy of citations.
The solution: The llms.txt delivers a clean document to the model, in Markdown, with only the content that matters.
# Kemeny Studio We are a consulting firm specializing in generative AI implementation and software development. ## Services - Implementation of generative AI solutions - Software Factory with core AI components - Digital transformation consulting ## Use Cases ### Document Indexing 20% improvement in mentions in AI responses ### Product Descriptions Higher click-through rate from ChatGPT, Gemini, and Perplexity to the site ## Contact - Web: kemeny.studio - LinkedIn: /in/kemeny
Useful analogy: It's like passing all this context directly into the ChatGPT prompt. Instead of the model having to "guess" what's important in your HTML, you explicitly tell it.
Effectiveness data
70%
Improvement in response quality and precision
Sites with well-implemented llms.txt show up to 70% more accuracy in how LLMs describe their content.
20%
More mentions in document indexing
In real implementations with clients, a 20% increase in mentions in AI responses has been observed when llms.txt is combined with good content structure.
Critical Configuration: robots.txt for AI Crawlers
Before creating your llms.txt, you need to ensure that AI crawlers can access your site. This is done in the robots.txt file.
The most important crawlers
Crawler Company Purpose GPTBot OpenAI Training of future models (GPT-5, GPT-6, etc.) ChatGPT-User OpenAI Real-time browsing when users use web search Google-Extended Google Training of Gemini and other Google AI models ClaudeBot Anthropic Training of Claude Claude-User Anthropic Real-time searches by Claude users PerplexityBot Perplexity Indexing for searches in Perplexity
Critical difference: GPTBot vs ChatGPT-User
GPTBot is for training. If you block it, your content will NOT be in GPT-6, GPT-7, etc. You will see the impact in 6-12 months when a new version of the model is released.
ChatGPT-User is for real-time browsing. If you block it, when someone uses ChatGPT with web search activated, the model will NOT be able to access your site to read content. The impact is immediate (days/weeks).
Example of a well-configured robots.txt
# Enable all traditional crawlers User-agent: * Allow: / # OpenAI - Training of future models User-agent: GPTBot Allow: / # OpenAI - Real-time searches User-agent: ChatGPT-User Allow: / # Google - Training of Gemini User-agent: Google-Extended Allow: / # Anthropic - Training of Claude User-agent: ClaudeBot Allow: / # Anthropic - Real-time searches User-agent: Claude-User Allow: / # Perplexity User-agent: PerplexityBot Allow: / # Meta (Facebook, Instagram) User-agent: Meta-ExternalAgent Allow: / # TikTok User-agent: Bytespider Allow: / # Block private content Disallow: /admin/ Disallow: /private/ Disallow: /internal/ # Sitemap (important for discovery) Sitemap: https://yoursite.com/sitemap.xml # Reference to llms.txt (next section) # See: https://yoursite.com/llms.txt
Pro tip: If you have private pages, dashboards, or internal content, use Disallow: /path/ to block them specifically. Do not block the entire site by default.
How to Create and Implement Your llms.txt
Step 1: Structure Your Content in Markdown
The llms.txt must be a plain text file (.txt) with Markdown format. Include:
Clear description of your company/brand
Main services or products
Use cases or industries you serve
Key pages of the site
Important resources or documentation
What NOT to include:
HTML, JavaScript, CSS
Repetitive navigation (headers, footers)
Excessive promotional content
Sensitive or private information
Real example of llms.txt
# Your Company - Brief description We are [your company], specialists in [your main industry/service]. ## What we do - Service 1: Concise description of what it is and for whom - Service 2: Main benefit and use cases - Service 3: Key differentiator ## Who we work for We work with companies in: - Industry A (e.g., retail, banking, technology) - Industry B - Industry C ## Main Resources ### Documentation - Implementation guide: /docs/implementation-guide - API Reference: /docs/api ### Success Stories - Case A: Title and key result - Case B: Title and key result ### Blog We publish content on [main topics]: - Main blog: /blog ## Contact - Website: yoursite.com - LinkedIn: /company/your-company - Email: contact@yoursite.com --- Last updated: October 2025
Step 2: Place the file in the root of your domain
The file must be accessible at:
https://yoursite.com/llms.txt
Similar to how you have:
https://yoursite.com/robots.txthttps://yoursite.com/sitemap.xml
Step 3: Reference the file in your HTML
In the <head> of your site, add:
<head> <meta charset="utf-8"> <title>Your Title</title> <!-- Reference to llms.txt --> <link rel="llms-txt" href="/llms.txt" /> <!-- Rest of your head --> </head>
For non-technical teams: You can use ChatGPT to generate your llms.txt. Simply give it your website and ask: "Create an llms.txt for my site in Markdown format, extracting only the key information without HTML."
Use Cases: Where Does This Work?
1. E-commerce with thousands of products
Common question: "Should I include all my products in llms.txt?"
Answer: No. Focus on:
Main categories
Star or best-selling products
Featured collections
Site structure (how to navigate)
Let crawlers discover individual products via sitemap.xml and site navigation.
Context engineering: As the model's context window runs out, its performance decreases. Do not overload the llms.txt with level 8 depth information. Prioritize high-value content.
2. SaaS with technical documentation
Success Story: Zapier
One of the first to implement llms.txt was Zapier. They created a structured file with all their documentation so that LLMs could help users implement integrations correctly.
Result: Better quality in responses when users ask "How do I integrate Zapier with X?" in ChatGPT or Claude.
3. Service companies (consulting firms, agencies)
Use llms.txt to clarify:
Exactly what services you offer
Target industries or clients
Summarized success stories
Key differentiators vs. competitors
Ready to implement but need help?
SearchBrand.ai will soon include automatic llms.txt generation optimized for your site. Meanwhile, you can use the platform to monitor how models are currently citing you.
How to Measure the Impact
1. Create a segment in Google Analytics
Identify traffic from AI crawlers by creating a custom segment that includes:
User-Agent contains "GPTBot"
User-Agent contains "ChatGPT-User"
User-Agent contains "ClaudeBot"
User-Agent contains "PerplexityBot"
User-Agent contains "Google-Extended"
Compare this channel with:
Organic Search
Referral
Direct
Social
2. Monitor with SearchBrand.ai
The platform allows you to:
See your position when someone asks about your industry
Identify which sources the models are citing
Compare yourself with competitors
Detect content gaps you need to cover
3. Test in incognito/temporary mode
Pro tip: Use ChatGPT's "temporary chat" (memoryless mode) to eliminate bias from previous conversations. This shows you how the model responds without your historical context.
Steps:
Open ChatGPT
Activate "Temporary chat" (clock icon)
Ask something about your industry/brand
Observe if you appear and how they describe you
Check which sources are being cited
4. Create an automated alert
Francisco Kemeny shares this advanced trick:
Prompt for ChatGPT: "Monitor the site [yoursite.com] every day and check for changes in how you describe my content when someone asks about [your industry/service]. Compare with yesterday's description and notify me of any changes."
This creates a recurring conversation where the model notifies you if it detects changes in how it is reading your site.
Expected Results Timeline
Action Expected Impact Timeline Enable ChatGPT-User in robots.txt Appear in real-time ChatGPT searches Days to 2 weeks Implement llms.txt Improved precision in citations and descriptions 1-4 weeks Enable GPTBot for training Appear in base knowledge of future models 6-12 months (next GPT-5, 6, etc.) Optimize content + llms.txt 20% more mentions, 70% better precision 1-3 months
Realistic expectations: Model base training takes time. If you enable GPTBot today, don't expect GPT-4 to "know" about you tomorrow. You will see that impact when GPT-5, GPT-6, etc., are launched.
However, for real-time searches (ChatGPT-User, web search), the impact can be visible in weeks.
Resources and Tools
📚 Official Documentation
llmstxt.org — Specification and convention guides
Project GitHub — Issues, discussions, and examples
🔍 Explore real examples
llmstxthub.com — Hundreds of examples of sites that have already implemented llms.txt
Anthropic llms.txt — Example implementation by Anthropic (creators of Claude)
🛠️ Tools
ChatGPT or Claude: Ask it to generate your llms.txt from your current site
SearchBrand.ai: Monitor how models cite you (coming soon: automatic llms.txt generation)
Google Analytics: Create segments to measure traffic from AI crawlers
Frequently Asked Questions (FAQ)
What is the llms.txt file?
The llms.txt is a convention file (not yet official) that provides language models (LLMs) with a simplified version of your site's content in Markdown format. It removes all HTML and code, leaving only the relevant information you want the models to extract and use. This improves the quality and precision of the responses LLMs generate about your content by up to 70%.
What is the difference between GPTBot and ChatGPT-User?
GPTBot is the crawler that OpenAI uses to collect information and train future models (GPT-5, GPT-6, etc.). If you block it, your content will NOT be in the model's base training. ChatGPT-User is the crawler used when a user performs real-time searches with ChatGPT. If you block it, ChatGPT will not be able to access your site when someone asks something at that moment. Both are important but have different purposes and distinct impact timelines.
How do LLMs actually search for information?
LLMs don't "search" like Google. They work in three ways: 1) Base training: Using model parameters with information up to a cutoff date, 2) RAG (Retrieval Augmented Generation): Bringing information into context from external sources, and 3) Tool Calling: Using tools like Google search APIs to get updated results, browse websites, and extract content in real-time.
How soon will I see results after implementing llms.txt?
There are two different timelines: For real-time searches (ChatGPT-User, web search), effects can be seen in days or weeks. For model base training (GPTBot), you might not see changes until a new version of the model is launched (6 months to 1 year). Current studies show up to 70% more precision in responses and 20% more mentions when implemented correctly along with good content structure.
Where do I place the llms.txt file on my site?
The llms.txt file should be placed at the root of your domain, accessible at yoursite.com/llms.txt (just like robot.txt and sitemap.xml). Additionally, you can reference it in the <head> of your HTML with a link tag: <link rel="llms-txt" href="/llms.txt" />. You don't need any special plugins or tools, just create the file in Markdown format and upload it to your server.
Should llms.txt be in Spanish or English?
Language models are multilingual and work well in any language. You can create your llms.txt in your audience's primary language. Models can process content in Spanish, Portuguese, English, etc., and generate accurate responses without issue. What is important is the clarity and structure of the content, not the specific language.
How do I measure if LLMs are accessing my site?
You can create a custom segment in Google Analytics that identifies traffic from AI crawlers (GPTBot, ChatGPT-User, ClaudeBot, GoogleExtended, PerplexityBot). You can also use tools like SearchBrand.ai to monitor how models cite you in their responses. Another advanced option is to create an automated alert in ChatGPT that monitors changes in how the model reads your site day by day.
Implementation Checklist
Review and update robots.txt with AI crawlers (GPTBot, ChatGPT-User, etc.)
Verify that sitemap.xml is updated and accessible
Create llms.txt in Markdown format with key site content
Upload llms.txt to the domain root (yoursite.com/llms.txt)
Add reference to llms.txt in the <head> of the HTML
Verify accessibility by navigating directly to the llms.txt URL
Create a segment in Google Analytics for AI crawler traffic
Test in ChatGPT temporary mode (memoryless) how it describes you
Monitor your positioning across multiple models with SearchBrand.ai
Document changes and measure impact month by month
Conclusion: Prepare Now, Reap Later
Implementing llms.txt and correctly configuring your crawlers is not the distant future: it is critical infrastructure today.
Unlike traditional SEO, which can take 6-12 months to show significant results, here there are two speeds:
Immediate impact (2-4 weeks): Better precision in real-time searches, more mentions when users ask in ChatGPT, Gemini, Perplexity
Long-term impact (6-12 months): Your content will become part of the base training of future models. When GPT-5, GPT-6, Gemini 3.0 are launched, your brand will already be in their core knowledge
The parallel with traditional SEO: Just like in 2003-2005 when SEO started, we are in the early days of optimizing for LLMs. Companies that move now will have a significant competitive advantage.
We are not reinventing the wheel, but we are adapting it to a new type of engine. The fundamentals are similar: accessibility, structured content, authority. Only the tools and conventions change.
Remember: 60% of consumers already use AI for shopping. If your site is not technically prepared, you are invisible to them. And that percentage will only grow.
Want to implement but need guidance?
Join the SearchBrand.ai community. Receive micro-trainings like this, share best practices, and access tools to monitor your positioning across all AI models.
About the participants:
Hussam Sufan is the founder of SearchBrand.ai