alt Hacker NewsUnsurprisingly, the pelicans are excellent. Google are the only lab who have publicly shared that they have specifically trained for SVG output - https://twitter.com/sunjiao123sun_/status/202455551655137292... - and they've had great pelican results in the past too (which they have shown off themselves - https://twitter.com/JeffDean/status/2024525132266688757).
Here's low, medium, high. Even low is better than pelicans from any other lab: https://tools.simonwillison.net/markdown-svg-renderer?url=ht...
low cost 10.185 cents, medium was 11.098 cents, high was 17.166 cents.
The costs are similar because it used more output tokens than reasoning tokens for all three pelicans, even for high.
Replies
As a cyclist, I tend to pay more attention to the bike shape than most people. Everyone knows what a bike looks like, but as with LLMs, most people would struggle to draw one - correctly connecting all the different tubes at correct angles etc.
I note that even these bikes are not actually rideable. For a bike to be stable, the front wheel axle needs to extend forward a little ahead of the fork - the fork must ”bend forwards”. This is what actually keeps the bike upright, not gyroscopic effects or some such. Given LLMs are generally good at basic maths, im surprised they get this wrong.
Maybe a bit less so in this case though. As per the announcements, the model was actually trained for this prompt, so was fed some training data. I can well imagine the data there was just not so good.
Recently, I’ve discovered that Claude can render great SVG versions of user interface screenshots. I ask it to substitute simple SVG friendly landscapes for any photographic images that happen to be part of the screenshots. The results are much better than they should be, especially with Opus. Usually good enough to be used on a website as a way to get an infinitely scalable, much smaller version of a screenshot. But clearly, I should try the Gemini models on this real world task. That’s where we’ll find out if all of this training to beat the benchmark actually pays off.
You can really see the effort in the details.
Subtle highlights on the metal frame, sweat beads, jaunty little hat…it has style. At least compared to the rather literal ChatGPT outputs. https://static.simonwillison.net/static/2026/gpt-pelicans-gr...
It does very good at Pac Man. I would say as good as Opus 5.5, it was slightly cheaper, but took twice as long.
Argon: https://jonclegg.github.io/pacman-bakeoff/dev/#gemini-4-argo...
Opus 5.5 https://jonclegg.github.io/pacman-bakeoff/dev/#claude-opus-5...
Simon’s headstone is going to have a pelican on a bicycle, isn’t it
It's obvious OAI hillclimbs the image generation -> SVG loop. I'd put money down on that being why OAI's pelicans have a yellowish tint that matches the yellow tint of OAI's earlier image models.
Why are all these AI's making the pelican's pouch so large though? Unless they have some huge fish in there or something they don't tend to look like that... Is it that it's drawing a "cartoon" pelican or something?
https://en.wikipedia.org/wiki/Pelican#/media/File:Pelikan_Wa...
How many cicles before pelicans become photoreallistic SVGs and them come back to stylistic artistic abstract renderings?
Also, that first tweet contains a commentary that, for me, explain a lot about the public misconception about how current AI works:
> wait, does this mean you guys are actively training svg generation? i was mostly amazed by it thinking the models doing better at it by itself due to becoming better at reasoning.
No, it's not better reasoning that drives current AI, it's better examples we have been giving them. Whole army of people generating content for the frontier labs.
Also, reasoning improving the out of distribution part, like computer use, tool use etc.
At some point Simon needs to move from Pelicans to something more complex. Every time a new model is launched by major Frontier company, I already know with certainty that first HN comment on the news would be about how Model is drawing the Pelicans.
Wow, those are I think the best looking pelicans ever so far
simonw, your pelicans are so famous now that the frontier labs can just RLVR for SVG images explicitly.
It's a simple manner of prompt...generate concept art with nanobanana, measure, llm creates svg with text, measure the distance between rasterized svg embedding to nanobanana art....do post-training RL on a 3 million such images and matching embeddings.
I swear this will be a canonical example of Goodhardt's Law in action.
You have to switch to a new benchmark that isn't so easy to run a verifiable rewards pipeline on...Perhaps a longer horizon task where it's easy to mess up midway.
That pelican left leg stays so firmly on the left side of the bike. Superintelligence ftw
Front bicycle wheel in all Argon examples is still wrong. This bicycle will not be stable. Axis of rotation of front wheel needs to be in-front of axis of rotation of steering wheel. Frontier VLMs still do not understand how bicycles work.
It starts to be so good that the model doesn't really matter. Each model starts to have its own personality, and it's the personality of the drawing, like a real artist, that defines the style of the pelican, and that makes it harder and harder to decide if it's good or bad.
You gotta fix your tool on mobile browser, both firefox and chrome don't work.
Since we know jagged intelligence is a thing, how useful of an indicator is the pelican test, regardless of if labs are optimizing for it or not?
Excellent, but not perfect -- if google really is training to pelicans, then clearly they are not capable of perfect pelicans.
As one point, the high pelican's arm is see-through/messed up.
They do look worlds better than previously, to be fair.
doesn't this mean that you should change the pelican test to something else?
At least you should make sure this post is entirely generated by agent, otherwise I'll not grant my agent the credibility score on this comment.
Regarding the pelican test. Maybe someone will know this.
Modern models can make nearly photorealistic games in threejs. So why do they struggle with these pelicans?
The fish in the basket on high is a lovely touch.
The fish in the basket is a nice touch
> Google are the only lab who have publicly shared that they have specifically trained for SVG output
wasn't the whole point of pelicans is that labs *wouldn't* train for that?
so now the benchmark is ruined
Honestly surprising if other labs aren’t training for svg generation. Feel like SVG’s are going to be really useful for dynamic UI generation stuff that goes beyond cobbling Html together
Nobody cares about the AI heckling pelicans.
Give me reverse engineering bench and agentic coding.
Leave making visuals to humans who actually have aesthetic sense unlike whatever made this mess
I've been dismissive of the SVG-of-pelicans benchmark, but last night ChatGPT (on the website, toggle set to GPT-6 High) generated some SVGs and included them in our chat. I'm not sure that this feature was that valuable in this case, but it seems that textually generating SVGs is now a product feature at least for ChatGPT. The SVGs had some clear issues and I found myself wishing that it had done a better job: the nose of the rocket is overlapping with the rocket body, and in the second image I think there's only supposed to be one orange line but it tried to add styling or something that went wrong.
<https://tools.simonwillison.net/svg-render#%3Csvg%20font-fam...> <https://tools.simonwillison.net/svg-render#%3Csvg%20font-fam...>
- I was having trouble embedding the SVGs as links, and ChatGPT recommended Simon Willison's SVG Renderer as a workaround.
Just my 2 cents on the pelicans benchmark: I think it might help if you were a bit more explicit about not taking this one benchmark too seriously. I know you say that at times, but at other times (such as the comment I'm replying to) you elide it which can read as a seriousness about the benchmark.
Holy shit, these are like art. Especially the bikes.
still radial spokes on both wheels of the bike which is disappointing. no real bicycle is built this way
[flagged]
[dead]
The models are solving Millennium Prize questions and making progress on the Riemann Hypothesis, yet the top comment is still looking at pelicans somehow. To me, it feels even more superficial and pointless than arguing about the presidential candidates' tie choices in a debate.
what is even the point of these now? labs are most definitely training on this question now. I feel like this test scratches the surface of a "real" test for these new models
Me in 2025 https://news.ycombinator.com/item?id=44759781
> If it's on HN and is a meme at this point, it will end up in the training set.
> It's kind of fun to imagine that there is an intern in every AI company furiously trying to get nice looking svg pelicans on bicycles.
Google listened and assigned a few interns, I guess, for the task