I try these things a couple times a month. They're always underwhelming. Earlier this week I had the thing work tells me to use (claude code sonnet 4? something like that) generate some unit tests for a new function I wrote. I had a number of objections about the utility of the test cases it chose to write, but the largest problem was that it assigned the expected value to a test case struct field and then... didn't actually validate the retrieved value against it. If you didn't review the code, you wouldn't know that the test it wrote did literally nothing of value.

Another time I asked it to rename a struct field across a the whole codebase. It missed 2 instances. A simple sed & grep command would've taken me 15 seconds to write and do the job correctly and cost $~0.00 compute, but I was curious to see if the AI could do it. Nope.

Trillions of dollars for this? Sigh... try again next week, I guess.