User-agent: * Allow: / Allow: /$ #Disallow: / #User-agent: * Disallow: / Crawl-delay: 10 # Paths (clean URLs)