User-agent: MJ12bot Disallow: / #Baiduspider User-agent: Baiduspider Disallow: / #Yandex User-agent: Yandex Disallow: / #Adding a crawl delay for Semrushbot 8th April 2018 due to massive use of server. User-agent: SemrushBot Crawl-delay:10