Both things can be true. I’ve noticed both the same thing the parent posted and what Linus posted and my vibe on the split (I haven’t kept detailed notes) is that on greenfield code they tend to maybe over-verify and on brownfield code or data analysis they sometimes give up too early or… I’m not sure, need a bit of encouragement to keep pulling at threads.

On the data analysis side, something specific I’ve noticed is an (understandable) bias towards computing numerical statistics, which they do very well and reading the post-analysis report has significantly improved my own “statistical thinking” approach overall. Numerical statistics are cool and understandably what a text-based LLM is going to want to work with, but asking the model to produce time-domain and frequency-domain plots of, say, specific events has multiple times resulted in “trying to plot this out has shown the opposite of what I concluded numerically… recalculating…” There’s still a pretty significant review and critically assess step for me, especially since the actions I take as a result of the analysis are pretty expensive, especially if they steer the next data collection run in a useless or harmful direction.