alt Hacker NewsThe author writes a significant amount about their workstation, and how cleaning up the geometry can be helpful when accessing it remotely over S3.
The dataset is a 8,000 row spreadsheet.
My advice when working with such a small dataset is not to overthink it.
Replies
3eb7988a1663 • today at 4:58 AM
He posts that same preface on all of his blog posts. Seemingly more pertinent when it is one of the 1TB datasets he pulls in from time to time.
There are ~15 GB of SAR imagery at the bottom being rendered as is from GeoTIFF files. On my 2020 MBP rendering that amount of data in QGIS would lag without building mosaics and tiles.
The Parquet pattern I'm promoting makes working across a wide variety of datasets much easier. Not every dataset is huge but being in Parquet makes it much easier to analyse across a wide variety of tooling.
In the web world, you might only have a handful of datasets that your systems produce so you can pick the format and schemes ahead of time. In the GIS world, you are forever sourcing new datasets from strangers. There are 80+ vector GIS formats supported in GDAL. Getting more people to publish to Parquet first removes a lot of ETL tasks for everyone else down the line.