alt Hacker NewsHere’s the thing though, most companies work with small data. The distribution of data set size follows a power law which means that few engineers get to work with petabyte sized datasets. That said, the job market definitely incentivizes people to have that kind of experience on their resume if they want to keep progressing in salary. This incentivizes over engineering.
Replies
groundzeros2015 • yesterday at 8:40 PM
yes, but engineers also suck at communicating costs and benefits (and understanding them).
> the job market definitely incentivizes people to have that kind of experience on their resume
Yeah, this is sadly often true, but it's also another trap that people don't have to fall into.
I've been involved with hiring data engineers, and I see experience with distributed computing way more often than I see knowledge of simple profiling and debugging tools. But I'd personally value the latter a lot more when interviewing.
Companies that hire for skills they don't need are of course perpetuating this problem, but they're also paying a big "tax" in the sense that they're not hiring for the skills they actually do need.