alt Hacker NewsHammer is not a perfect analogy because of how simple it is, but sure let's go with it.
Imagine that occasionally when getting in contact with the nail it shatters to bits, or goes through the nail as it were liquid, or blows up, or does something else completely unexpected. Wouldn't you want to fix it? And sure, it might require deep understanding of the nature of the materials and forces involved.
That's what I'd do.
Replies
A better analogy might be something like medicine. There are many drugs prescribed that are known to help with certain conditions, but their mechanism of action is not known. While there may be research trying to uncover those mechanisms, that doesn't stop or slow down rolling out of the medicine for use. Research goes at its own pace, and very often cannot be sped up by throwing money at it, while the market dictates adoption. I see the same with LLMs. I'm sure this has attracted the attention of more researchers than anything else in this field, but I would expect any progress to be relatively slow.
Use the human brain as an example then. We don't really know how it works. I mean, we know there's neurotransmitters and neural pathways etc (much like nodes in a transformer), but we don't know how exactly intelligence or our thinking process works.
We're also pretty good at working around human 'hallucinations' and other inaccuracies. Whether it be someone having a bad day, a brain fart, or individual clumsiness. eg in a (bad) organisation, sometimes we do it with layers of reviews and committees, much like layers of LLMs judging each other.
I think too much is attached to the notion of "we don't understand how the LLM works". We don't understand how any complicated intelligence works, and potentially won't for the forseeable future.
More generally, a lot of society is built up from empirical understanding of black box systems. I'd claim the field of physics is a prime example. And we've built reliable systems from unreliable components (see the field of distributed systems).