🤖 SEO Robots.txt Testing Project

This project tests whether robots.txt actually blocks crawler access or just prevents indexing. Choose a test page below:

✅ Dynamic Crawlable Page

Fetches content via API calls. Tests crawler behavior with dynamic content.

Visit Dynamic Page →

URL: /crawl

✅ Static Crawlable Page

Same content as dynamic page but embedded statically. Better for SEO.

Visit Static Page →

URL: /static-crawl

🚫 Blocked Page

Blocked by robots.txt. Tests if crawlers respect the directive.

Visit Blocked Page →

URL: /no-crawl

🧪 How to Test

  1. Deploy this site to a public URL
  2. Check robots.txt at /robots.txt
  3. Submit your site to Google Search Console
  4. Monitor server logs for bot requests to both pages
  5. Check which pages appear in search results
View robots.txt →Test API Endpoint →