# # Please note, we do NOT allow nonauthorized robots. # Some of crawlers in this robots.txt do not respect directives, so we use internal trap functions to block. # # Copyright owners have the legal right under the DMCA to reserve the right to view content only to website visitors. # Webmasters have the legal right under DMCA to block access to anyone who wants to store or copy website content. # It is a crime to use any trick or false information to gain access to a computer system. # It is a crime running a robot that pretends to be a user by faking its useragent. # It is a crime using false information to gain access to a computer system. # So please respect this and get in touch before running crawler on this site as you might get blocked. # # Thank you very much # User-Agent: bingbot Crawl-delay: 10 User-agent: msnbot Crawl-delay: 10 User-agent: findlinks Disallow: / User-agent: www.integromedb.org/Crawler Disallow: / User-agent: Butterfly Disallow: / User-agent: TurnitinBot Disallow: / User-agent: SlySearch Disallow: / User-agent: findlinks Disallow: / User-agent: magpie-crawler Disallow: / User-agent: Pixray-Seeker Disallow: / User-agent: MJ12bot Disallow: / User-agent: 008 Disallow: / User-agent: Ezooms Disallow: / User-agent: AhrefsBot Disallow: / User-agent: lb-spider Disallow: / User-agent: WBSearchBot Disallow: / User-agent: psbot Disallow: / User-agent: HuaweiSymantecSpider Disallow: / User-agent: sistrix Disallow: / User-agent: EC2LinkFinder Disallow: / User-agent: htdig Disallow: / User-agent: SemrushBot Disallow: / User-agent: discobot Disallow: / User-agent: linkdex.com Disallow: / User-agent: SeznamBot Disallow: / User-agent: EdisterBot Disallow: / User-agent: SWEBot Disallow: / User-agent: picmole Disallow: / User-agent: Yeti Disallow: / User-agent: Yeti-Mobile Disallow: / User-agent: PagePeeker Disallow: / User-agent: CatchBot Disallow: / User-agent: yacybot Disallow: / User-agent: netEstate NE Crawler Disallow: / User-agent: SurveyBot Disallow: / User-agent: COMODO SSL Checker Disallow: / User-agent: Comodo-Certificates-Spider Disallow: / User-agent: gonzo Disallow: / User-agent: schrein Disallow: / User-agent: BacklinkCrawler Disallow: / User-agent: Afilias Web Mining Tool Disallow: / User-agent: SEOkicks Disallow: / User-agent: SEOkicks-Robot Disallow: / User-agent: suggybot Disallow: / User-agent: bdbrandprotect Disallow: / User-agent: BPImageWalker Disallow: / User-agent: Updownerbot Disallow: / User-agent: lex Disallow: / User-agent: Content Crawler Disallow: / User-agent: DCPbot Disallow: / User-agent: KaloogaBot Disallow: / User-agent: MLBot Disallow: / User-agent: Wget Disallow: / User-agent: libwww-perl Disallow: / User-agent: curl Disallow: / User-agent: Java Disallow: / User-agent: iCjobs Disallow: / User-agent: oBot Disallow: / User-agent: WebmasterCoffee Disallow: / User-agent: Qualidator* Disallow: / User-agent: Webinator Disallow: / User-agent: Scooter Disallow: / User-agent: *thunderstone* Disallow: / User-agent: larbin Disallow: / User-agent: OpidooBOT Disallow: / User-agent: ips-agent Disallow: / User-agent: TinEye Disallow: / User-agent: Unister* Disallow: / User-agent: ReverseGet Disallow: / User-agent: Twitterbot Disallow: / User-agent: UnwindFetchor Disallow: / User-agent: ShopWiki Disallow: / User-agent: CompSpyBot disallow: / User-Agent: Slurp disallow: / User-Agent: discoverybot disallow: / User-agent: Exabot disallow: / user-agent: BoardReader disallow: / User-agent: ia_archiver Disallow: / User-agent: JikeSpider Disallow: / User-Agent: MSIECrawler Disallow: / User-agent: magpie-crawler Disallow: / User-agent: MJ12bot Disallow: / User-agent: Yandex Disallow: / User-agent: YandexBot Disallow: / User-agent: Baiduspider Disallow: / User-Agent: * Allow: / Sitemap: https://www.osullivanlegal.com.au/sitemap.xml