You can get use out of a hammer without understanding how the strong force works.
You can get use out of an LLM without understanding how every node works.
You can get injured by using a hammer without understanding how it works.
You can damage a company by using a spreadsheet and not understanding how it works.
In your personal opinion, what are the things you should know before using an LLM?
Hammer is not a perfect analogy because of how simple it is, but sure let's go with it.
Imagine that occasionally when getting in contact with the nail it shatters to bits, or goes through the nail as it were liquid, or blows up, or does something else completely unexpected. Wouldn't you want to fix it? And sure, it might require deep understanding of the nature of the materials and forces involved.
That's what I'd do.