GeneratorsLesson 54Free preview
robots.txt Generator
This lesson is part of Web Developer Tools · All levels
Summary
The generator helps create text files in the root domain that give clues to the search engine crawler. This file contains the user-agent, Allow, Disallow, and sitemap location rules. robots.txt is not secure: it is only followed by a compliant crawler, so the private pages still need authentication protection.
Key points
- robots.txt is placed in the root domain, for example https://contoh.com/robots.txt (https://contoh.com/robots.txt)
- User-agent determines the crawler that is the target of the rule
- Disallow tells the crawler not to crawl a certain path
- Sitemap tells you the location of the site's URL list
- It does not replace server authentication or protection
Full lesson
Code examples and quizzes
Full lessons, code examples, and quizzes are available to subscribers. 5 questions.