Robots.txt in 2026: how to create, test, and avoid breaking indexation
A practical robots.txt guide: what to block, what not to block, why robots does not remove pages from Google, and how to connect it with sitemap.
Robots.txt decision tree
Robots manages crawling. It is not a privacy or deindexing tool.
Robots.txt is a small file with large risk. One line can block important pages from crawling, while another can leave filters, parameters, and technical folders open for crawl waste.
The main rule: robots.txt controls crawling, not guaranteed removal from search. Google warns that robots.txt should not be used as a way to hide web pages from search results.
What robots.txt is good for
| Job | Use robots.txt? | Note |
|---|---|---|
| Reduce crawl waste | yes | filters, parameters, technical folders |
| Block CSS/JS | usually no | Google should understand the page |
| Remove a page from index | no | use noindex or remove the URL |
| Point to sitemap | yes | useful for search engines |
| Hide private data | no | use authentication, not robots |
Basic example
User-agent: *
Disallow: /wp-admin/
Disallow: /*?sort=
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/sitemap.xml
Before publishing, check that you are not blocking service pages, blog pages, images, CSS, or JavaScript needed for rendering.
Common competitor mistakes
- blocking the entire site with
Disallow: /; - blocking theme resources and hurting rendering;
- blocking URLs and expecting them to disappear from the index;
- forgetting sitemap;
- not checking robots after redesign or migration.
Next step: if you are unsure, start with a technical audit, check the site in UNmiss, and add sem.chat so users do not get stuck after search visits.
Sources
How to choose an SEO agency in 2026: a verification procedure
Not another ranking. Ten criteria with a specific check for each, how to tell a number you can trust from one you can't, what the market actually charges with links to the agencies' own price pages, what belongs in the contract, and when you don't need an agency at all.
Read →
A/B Testing in 2026: How to Boost Site Conversion with a Split Test (Step-by-Step Guide)
What A/B testing is and how to run a split test correctly in 2026: hypotheses, statistical significance, sample size, tools after Google Optimize sunset, AI experiments, and the link between CRO and SEO.
Read →
Google Tag Manager Setup from Scratch: 2026 Guide
How to install Google Tag Manager, connect it to GA4, and set up Consent Mode v2, tags, triggers, and variables. A step-by-step 2026 guide with a checklist.
Read →Want to apply this to your site?
We will review the current situation, find the first growth levers, and suggest a practical working format.
