logoalt Hacker News

phildinilast Tuesday at 9:52 PM0 repliesview on HN

Hi! I've written crawlers for about a dozen municipal hosting platforms, and you can learn the bare-bones of it from our "How" page: https://civic.band/how.html

I also gave a talk on this concept that walks through the whole process: https://www.youtube.com/watch?v=MtWzNnZvQ6w

The short answer is: there's no common API for any of these sites, and even the ones that do have an API are sometimes misconfigured. It's why I wrote all the scrapers by hand.