About the CivicBot

by

If you are here, you probably saw this link in a server log from my bot hitting your web server.
 
The goal of this bot is to crawl public content related to the governance of public government entities in the USA in order to build out a data platform that will allow for improved collaboration with the public and between organizations.
 
If that is not the content I am crawling, or not what your site serves, something went awry.  Please let me know what URLs we are hitting that don't fit our goal. 
 
The bot is coded to respect robots.txt and throttle its requests. If it is doing something that is not respectful of your site, let me know. 
 
It is currently in quite an early state. It runs from my home PC as I am developing the first version. I am testing on a small selection of sites, testing out how it works against different underlying platforms.  If you are a vendor who provides governance SaaS apps to many organizations, you are more likely to see this bot more often. (And depending on which vendor, also more likely to already know me - feel free to email and say 'Hi')
 
I am aiming to find the balance between capturing timely content updates while still respecting the performance of your servers. If I have missed that mark, let me know.
 
For any questions or concerns, please contact me at hikingdave @ gmail.com