alt Hacker NewsMany years ago I was working at a firm that investigated stock pump and dump schemes on Yahoo Finance message boards.
We had to scrape the boards and then analyze messages to see if we could identify the identity of the people running the schemes.
This was 2002 and I wasn't aware that the LWP::Simple existed in Perl (aka the requests library in Python).
I ended up using basic TCP socket libraries to connect to port 80 and do http requests. It was, to put it mildly, a major pain in the ass.
That being said, I learned a TON about how http, tcp, html, etc all work together. 20+ years later, I still use some of that knowledge when analyzing network protocols at work.
I mention this b/c nothing is stopping people from doing similar projects now. e.g. Linux from scratch [0] is one great example of learning fundamentals even though we now have 1 click "launch me a Linux VPS" options.
Joel Spolsky make this point too. Even if you are using Java, it's still helpful to understand how CPUs interact with cache because the highest performance comes from optimizing the whole stack.
Replies
I have a tendency to prefer writing my own library for something instead of relying on someone else's. Not always, obviously, but quite often existing libraries fall short or are too complex to use. And solving the problem myself is almost always an educational experience.
For example, I'm currently writing my own graph layout library because I'm not happy with Dagre and ELK.
My side projects are all hand coded for this reason. There's no pressure to ship fast, so why not take the time?
Terry Tao has a term for this I liked quite a bit, "productive struggle"
My work is more data science related, but remember those same struggles where I spent hours, days, weeks, months, even years crashing my head against a problem. A lot of daily hours researching, talking to colleagues, implementing tangential papers that lead to nothing. Most of the time I arrived to a satisfactory solution, but others just have to gave up.
Maybe because I experienced all those struggles I'm confident to delegate a relevant portion of my work to some LLM, being able to validate the results, knowing what to ask and detect easily when something was bad implemented or where I gave ambiguous instructions.
The catch is that I'm starting to do things where I haven't experienced those struggles in prior times, for example, web app development for my data work. I don't feel confident that I'm doing a good work, I'm just vibing, don't know the implications of some decisions. It doesn't makes me feel comfortable shipping things I don't really understand, but also is nice to be able to do things that previously required years of studying and practice.
We are on strange times.