About AussieSearchBot
AussieSearchBot is the small, polite crawler behind the “Australian websites” section of Aussie Search.
- It identifies itself as
AussieSearchBot/1.0 (+https://aussiesearch.app/about/crawler/). - It always obeys
robots.txtandnoindexmeta tags. - It only fetches a site’s home page, about one request per second, and revisits about once a month.
- It stores only the page title, a short description (up to 160 characters), the address and the date. It never stores images or full page text.
- Every result links straight back to the website.
- It starts from websites linked in our own listings and from public government sites.
Opt out
To block it, add this to your robots.txt:
User-agent: AussieSearchBot Disallow: /
To have your site removed now, email [email protected] with the web address.