logoalt Hacker News

ronsorlast Tuesday at 9:32 PM2 repliesview on HN

One of my first tests with it was "Write a Python 3 interpreter in JavaScript."

It produced tests, then wrote the interpreter, then ran the tests and worked until all of them passed. I was genuinely surprised that it worked.


Replies

Calavarlast Tuesday at 10:45 PM

There are multiple Python 3 interpreters written in JavaScript that were very likely included in the training data. For example [1] [2] [3]

I once gave Claude (Opus 3.5) a problem that I thought was for sure too difficult for an LLM, and much to my surprise it spat out a very convincing solution. The surprising part was I was already familiar with the solution - because it was almost a direct copy/paste (uncredited) from a blog post that I read only a few hours earlier. If I hadn't read that blog post, I would have been none the wiser that copy/pasting Claude's output would be potential IP theft. I would have to imagine that LLMs solve a lot of in-training-set problems this way and people never realize they are dealing with a copyright/licensing minefield.

A more interesting and convincing task would be to write a Python 3 interpeter in JavaScript that uses register based bytecode instead of stack based, supports optimizing the bytecode by inlining procedures and constant folding, and never allocates memory (all work is done in a single user provided preallocated buffer). This would require integrating multiple disparate coding concepts and not regurgitating prior art from the training data

[1] https://github.com/skulpt/skulpt

[2] https://github.com/brython-dev/brython

[3] https://github.com/yzyzsun/PyJS

wubrrlast Tuesday at 9:38 PM

It's ability to test/iterate and debug issues is pretty impressive.

Though it seems to work best when context is minimized. Once the code passes a certain complexity/size it starts making very silly errors quite often - the same exact code it wrote in a smaller context will come out with random obvious typos like missing spaces between tokens. At one point it started writing the code backwards (first line at the bottom of the file, last line at the top) :O.