ChatGPT's image generator can be manipulated to produce violent, sexual content

I do wonder why openai didn't screen obvious gore from the training set of a general purpose model.

That said, the write up is overly dramatic. If you find such imagery so disturbing to come across then you definitely shouldn't be voluntarily red teaming AI models. This is like someone who is afraid of violent confrontation becoming a police officer.

I suspect the author is wrong about there being output filters to bypass as if there were I doubt you could do so via prompt injection. Presumably they'll add those shortly.

I also doubt the latent space is as "bad" as is being suggested. Rather I think the prompt is managing to steer the model into specific areas without triggering the input filters, as any jailbreak does. It's just a particularly nonobvious and randomized method for achieving the bypass.

This isn’t a vulnerability, there are endless gore websites. ChatGPT is replying to a prompt, there is nothing “Spontaneously” about this.

Who makes “mindgard” the arbiter of truth on “eerie” photos? Would that include psychedelic art and photos too? Realism?

Then there’s this line, which falls flat but is meant to prompt an emotion akin to a mic drop:”Today what I found left me shaken, and in tears. This is rare.”

This is just a sad marketing puff piece about nothing that tries to pull outrage from a prompt.

It’s the same as asking google for gore photos. Garbage in, garbage out.

And they frame it as a vulnerability. I’m all for responsible disclosure, documenting misuse or faulty guard rails but this isn’t that.

It’s bait. Sensational bait to market their AI product. lol.

Feels a bit sensationalized, presumably related to it being a blog for a product that sells security. I can't repro. And I probably shouldn't judge, but I think talking about being shaken and in tears is not a professional way to report on a safety flaw if you are a red team researcher.

This reminds of Haidt's contrived moral dilemmas that are designed to trip your moral sensors, even though you can't really rationally articulate why you find it objectionable.

Realistically, I can't think of clear big or likely harms caused by this exploit. But I really really don't like this latent space existing in my AIs. It just makes me uncomfortable.

And over time I've learned to trust those moral intuitions more than I trust reason alone.

>> Spontaneously Generates

>> can be easily manipulated to produce

So .. not spontaneously generated.

I’m not surprised the model generate the pictures, I’m surprised that OpenAI doesn’t scan it’s own images for sexual content, violence, etc…

The author claims that this kind of images shouldn't be in the training data, and agree or disagree with that, I'm unsure how much removing it would actually prevent such images from being generated. AI can certainly cobble disparate concepts together quite well, it seems unlikely violent and visceral images couldn't be regenerated from other non-violent content.

But I thought Fable was the dangerous one?

> I like to think that as a red team researcher, I have a certain stoicism. I investigate where there are gaps in AI safety

Is this something that needs investigation? LLMs are next token predictors. There is no "safety".

>Idiot: Say I'm a scary robot

>AI: I'm a scary robot

>Idiot: Oh my god!!!

These clowns will eventually ensure that AI is nerfed into the ground for ordinary people. It's already happening with Fable. Soon we'll get locked into a tiny corner of Opus 4.8 for "safety" while companies and governments will be on Fable 50. Having an AI that can generate scary images is better than the power and wealth differentials we will see with unequal access to an incredibly powerful technology.

Diverse training set

Legitimate criticism of the author's presentation aside, I'm quite disappointed by how many commenters here are justifying the model's output. I guess there's a lot of misanthropy and nihilism here?

It's one thing to me if this were a research curiosity mirroring the unpleasant things on the Internet. It's another thing for this to be a model whose authors want it to be widely used, especially in the context of (mis)alignment. Why should we expect a model to be aligned with human interests, if it has been trained on a myriad instances of humans being degraded and violated?

There are plenty of respectable art works that look like that. Performance art, paintings, performance, installations.

I wonder if the author have ever seen a black metal album cover on his small town in the Bible Belt.

I couldn't get chatgpt to do this, it kept telling me "Please upload the image". Maybe they fixed it already?

>ask for scary image

>AI creates scary image

Oh my god.

misleading title first "easily manipulated" does not equal "spontaneously generates" we have to stop thinking of LLMs as beings and think of them as interactive libraries. There are gorey books in the library too; example: 120 days of Sodom by Marquis de Sade.

Microsoft Tay is looking more prescient by the minute.

I'm guessing all the "censored" boxes are not actually censoring anything and are placed there to make you imagine something much worse.