logoalt Hacker News

marklittoday at 5:51 AM0 repliesview on HN

There are ~15 GB of SAR imagery at the bottom being rendered as is from GeoTIFF files. On my 2020 MBP rendering that amount of data in QGIS would lag without building mosaics and tiles.

The Parquet pattern I'm promoting makes working across a wide variety of datasets much easier. Not every dataset is huge but being in Parquet makes it much easier to analyse across a wide variety of tooling.

In the web world, you might only have a handful of datasets that your systems produce so you can pick the format and schemes ahead of time. In the GIS world, you are forever sourcing new datasets from strangers. There are 80+ vector GIS formats supported in GDAL. Getting more people to publish to Parquet first removes a lot of ETL tasks for everyone else down the line.