alt Hacker NewsHow do you get access get all the local government meetings? Do you have a crawler that looks up every city in the country then visits each website and pull down the info? A public listing site?
Replies
Hi! I've written crawlers for about a dozen municipal hosting platforms, and you can learn the bare-bones of it from our "How" page: https://civic.band/how.html
I also gave a talk on this concept that walks through the whole process: https://www.youtube.com/watch?v=MtWzNnZvQ6w
The short answer is: there's no common API for any of these sites, and even the ones that do have an API are sometimes misconfigured. It's why I wrote all the scrapers by hand.
There isn’t any sort of standard for recording public meetings. I’ve seen everything mic less live streams with obstructed cameras to well curated flawless back and forth with great audio and transcripts. Meeting to meeting it can vary.
Perhaps something like https://www.perigon.io?
This video that was posted here yesterday shows some details: https://youtu.be/W420BOqga_s?t=93
Apparently there is scraping of public data + keyword matching + moderators filtering the matches.
An example that he shows a bit earlier in the video comes from this page, which has an RSS feed: https://www.cityofsanbenito.com/AgendaCenter/City-Commission...
The video says it's open source but I can't find the source.