robots.txt
A file that tells search engine crawlers which pages they can or cannot access.
robots.txt is a text file at the root of a website (yoursite.com/robots.txt) that gives instructions to search engine crawlers about which pages they should or shouldn't crawl. It's a request, not an enforcement — well-behaved crawlers respect it; malicious crawlers ignore it.
Common uses: blocking crawlers from admin/staging pages, internal search result pages, and other non-public content. Don't use robots.txt to hide sensitive content — use authentication instead.
A site blocks /admin and /internal-search-results from crawling in robots.txt. Google indexes only public pages.
Related terms
Need help applying robots.txt to your business?
Book a free 30-minute strategy call. I'll show you how robots.txt fits into a real growth strategy for your business.
Book a free strategy call