# # robots.txt # # This file will be ignored unless it is at the root of your host: # Used: http://example.com/robots.txt # Ignored: http://example.com/site/robots.txt # # For more information about the robots.txt standard, see: # http://www.robotstxt.org/wc/robots.html # # For syntax checking, see: # http://www.sxw.org.uk/computing/robots/check.html # Unwanted robots # Allow baidu to have access, as of 2024-01-05 #User-agent: baiduspider #Disallow: / User-agent: Yandex Disallow: / User-agent: PiplBot Disallow: / User-agent: AhrefsBot Disallow: / User-agent: SemrushBot Disallow: / User-agent: DataForSeoBot Disallow: / User-agent: MJ12bot Disallow: / User-agent: SEOkicks Disallow: / User-agent: BLEXBot Disallow: / User-agent: Bytespider Disallow: / # Disallow AI training bots for various LLMs # See https://www.cyberciti.biz/web-developer/block-openai-bard-bing-ai-crawler-bots-using-robots-txt-file/ User-agent: GPTBot Disallow: / User-agent: ChatGPT-User Disallow: / User-agent: Google-Extended Disallow: / User-agent: CCBot Disallow: / User-agent: TurnitinBot Crawl-delay:60 # Allow msn/bing to crawl at full speed, as of 2024-01-04. Let's see what happens. #User-agent: msnbot #Crawl-delay: 5 # #User-agent: bingbot #Crawl-delay: 5 User-agent: * # Ajax items Disallow: /22/299/generate-shareable-links* # Ad links Disallow: /handler/listing/click/ Disallow: /de_DE/handler/listing/click/ Disallow: /es_ES/handler/listing/click/ Disallow: /fr_FR/handler/listing/click/ Disallow: /22/275/click/ Disallow: /de_DE/22/275/click Disallow: /es_ES/22/275/click Disallow: /fr_FR/22/275/click # Robot tracker Disallow:/robocop Disallow:/?q=robocop Sitemap: http://bachtrack.com/sitemap.xml User-agent: A1 Sitemap Generator* Crawl-delay: 4 User-agent: Mediapartners-Google* Disallow: /