All of the links show "Error: Gist API returned 403".

Is there a reason these pelicans always have roughly the same composition (side-view, 2d, biking right, flat ground beneath, etc)? I don't see any of that detailed in the prompt, yet they all seem to generate roughly the same image of differing quality.

Is there any point anymore regarding this svg test? I would not be surprised if in the training they're fine tuned for this task too

Did any LLM so far draw pelican knees correctly and have them bend in opposite direction from human knees? Knees of many animals bend opposite to humans.

Did any LLM draw the front bicycle wheel correctly? ie. center of front wheel slightly AHEAD of steering wheel axis. This is done for bicycle stability.

Simon, at this point I really wonder if teams aren’t gaming this. You should pick a random animal doing a random thing every time.

If you have a grading rubric, huge points off for adding arms instead of using the wings as arms!

Has any ab tried to game this yet and just made the most amazing pelican by hand and always reply with that?

I wonder, given Simons reputation in AI benchmarking, whether model providers try to train or tweak their models to perform better at drawing bicycles and pelicans?

Absolutely BRUTAL! :)

Thank you for doing this, I love your benchmark the most!

For all the comments of "I'm sure they're fine-tuning for pelicans": https://dylancastillo.co/posts/pelicanmaxxing.html

  "Sorry, HN haters, but there’s little evidence that AI labs are pelicanmaxxing.
   Or at least they’re not doing it in a plainly obvious manner."

What does the mean pelican look like at this point?

Also 3X token use vs. 1.2

lol

Definitely an upgrade over 1.2

[flagged]