alt Hacker NewsIt used to be the case that a "data engineer" was just a software engineer that was good at data-related things (maybe they were good at Kafka, airflow, DBT, knew why you'd pick postgres for one job and Clickhouse for another, etc).
Then the BI people (who had never written a line of code except maybe some SQL) got jealous, started calling themselves "data engineers", broke everything ("what do you mean 'we need tests???'"), and ruined the title.
Replies
Maxion • today at 10:08 AM
Once upon a time they were called DBAs.
I have a different take on this. I have a friend who used to work at a bank as an analyst, and he had a hardcore finance/math background, and was a self-taught R/Python/SQL coder before this whole 'big data' thing became popular a decade ago.
Back then his bank started building a data science/engineering dept, and he jumped ship, and essentially did the same work for 2x the salary. Let me tell you, understanding Black-Scholes and what the data actually means and how to source it, as well as what finance people actually are looking for is a skill that's 1000x as difficult to get as learning R/Python/Pandas etc.