Technical SEO
Robots.txt
A text file that instructs search engine crawlers which pages or sections of a site to crawl or ignore
Robots.txt is a file placed in a website’s root directory that provides instructions to search engine crawlers about which pages or directories they should or should not access. For link building, understanding robots.txt is important when evaluating potential link partners. If a page where your link is placed is blocked by robots.txt, search engines cannot crawl it, meaning your backlink provides no SEO value. When auditing link opportunities, checking the target site’s robots.txt ensures your links will be discoverable by crawlers. Additionally, your own robots.txt should not accidentally block important pages that receive backlinks.