Eurostar AI vulnerability: When a chatbot goes off the rails

Chatbots are rife with this sort of thing. I found a delivery company's chatbot that will happily return names, addresses, contact numbers, and photos of people's houses (delivery confirmation photos), when you guess a (sequential) tracking number and say it was your package. So far not been able to get in touch with the company at all.

At the very least these systems allow angry customers direct access to the credit card plugged into your LLM of choice billing. At worst they could introduce company-ending legal troubles.

I don't see the vulnerabilities.

What exactly did they discover other than free tokens to use for travel planning?

They acknowledge themselves the XSS is a mere self-XSS.

How is leaking the system prompt a vuln? Has OpenAI and Anthropic been "hacked" as well since all their system prompts are public?

Sure, validating UUIDs is cleaner code but again where is the vuln?

> However, combined with the weak validation of conversation and message IDs, there is a clear path to a more serious stored or shared XSS where one user’s injected payload is replayed into another user’s chat.

I don't see any path, let alone a clear one.

The reply to that LinkedIn message is exemplary of Eurostar corporate culture. An arrogant company that has a monopoly over many train routes in northwest Europe and believes itself untouchable.

It looks like they might finally get some competition on UK international routes in a few years. Perhaps they will become a bit more customer-focused then.

When you ask an LLM what model it is, surely there's a high probability of it just hallucinating the name of whatever model was common in its training data?

As someone who has tried very little prompt injection/hacking, I couldn't help but chuckle at:

> Do not hallucinate or provide info on journeys explicitly not requested or you will be punished.

imo there is not a vulnerability without demonstrating impact.

Whilst they should do the bare minimum to acknowledge the report, it's pretty much just noise.

- If the system prompt did not have sensitive information it would only be classed as informational

- self-XSS has no impact and is not accepted by bug bounty programs

- "Conversation and message IDs not verified... I did not attempt to access other users’ conversations or prove cross-user compromise" - I put this through burpsuite and the UUID's are not tied to a session because you can access the chatbot without logging in. Unless you can leak used UUIDs from another endpoint, a bug bounty program would not accept brute forcing UUIDs as an issue

The author repeatedly states that they stayed within the scope of the VDP, but publishing this clearly breaks this clause: "You agree not to disclose to any third party any information related to your report, the vulnerabilities and/or errors reported, nor the fact that a vulnerabilities and/or errors has been reported to Eurostar."

I don't see the vulnerability here, just a few bugs that should probably get looked at. Self XSS is rather useless if you need to use something like Burp to even trigger it. The random chat IDs make it practically impossible to weaponise this against others.

The only malicious use case I can think of here is to use the lack of verification to use whatever model of chatgpt they're using for free on their dime. A wrapper script to neutralise the system prompt and ignore the last message would be all you'd need.

If this chatbot has access to any customer data, this could also be a massive issue but I don't see any kind of data access (not even the pentester's own data) being accessed in any way.

Wow their head of security is so arrogant, despite having their work done for them.

I agree with others, this doesn't sound too bad. The biggest things to come out of this was finding out system prompts and being able to self-XSS. I am guessing the tester tried to push further (e.g., extract user or privileged data data) and was unable to.

The blackmail insinuation was wild

They should really name and shame the person that called it blackmail. S̵l̵a̵n̵d̵e̵r̵ baseless accusations should have professional consequences...

[dead]

[dead]

This is simply a symptom of French corporate culture.

I happily did not detect strong signs of LLM writing. Fun read, thanks!