logoalt Hacker News

kentmtoday at 6:20 PM0 repliesview on HN

> But I won't follow you as far as "expecting developers to have expertise in how and where their software runs is unreasonable".

In my professional experience, management tends to treat this as a dichotomy rather than a range. And its lead to situations where we're choosing to route every decision through a small number of 'experts' and expecting that to scale, or have people make very bad decisions about things like indices, data layout, etc.

I align with you on this. If you are using some sort of system, there is a sort-of minimum bar of expertise one should acquire. When using analytical platforms, having a basic understanding of how data is stored and the levers it gives you to optimize data processing should be standard.

Some examples:

MySQL - I would expect developers to be able to come up with a list of columns to index and why, understand why UUIDv4 is a bad choice for primary keys, know how to write performant SQL by hand and understand when their ORM is going to do the wrong thing, etc.

Redshift - Understand dist and sort keys and have some mental models on how data shuffles and is pruned based on each choice.

Iceberg + Spark/Trino - Understand that data tends to be stored/loaded as batch files. Understand how deletes work. Understand partitions and sort as options. Understand generally how bloom filters work and when you should decide to place a bloom filter. Understand why UUIDv7 is better than UUIDv4 when its part of a partition key.