Skip to main content
TechnicalSEO & Discovery

How to implement and verify robots.txt

Create a robots.txt file in the /public folder specifying which crawlers can access which parts of your site. Include Sitemap: https://yourdomain.com/sitemap.xml. For AI crawlers, allow them explicitly if you want indexing by Perplexity, ChatGPT, etc. Test via the Robots Tester in Google Search Console.

Related questions

Can my site rank well on search using Lovable?How can I get the best SEO and GEO results with Lovable?How can I improve SEO for my project?How long does it take Google to index my page?How often should I update SEO settings, content, and sitemap?