logoalt Hacker News

EagnaIonatyesterday at 7:47 AM0 repliesview on HN

Anything relatively new in a technology LLMs struggle with, especially if the documentation is lacking.

Godot for example in ChatGPT.

It may no longer still be the case, but the documentation for GoDot was lacking and often samples written by others didn't have a version number associated with it. So samples it would suggest would never work, and even when you told it the version number it failed to generate workable code.

The other stuff I've noticed is custom systems. One I work with is a variation of Java, but LLMs were treating it as javascript. I had to create a LoRA just to get the model from not trying to write javascript answer. Even then it could never work, because it had never been trained on real world examples.