Northflank Breached

I hope this is a real wake-up call for teams running anything remotely SaaS-shaped. I count at least six substantial, lightning-speed attacks against multi-tenant as-a-service vendors since September. Not this year--just since last month. PingCAP/TiDB, Fireworks AI, Zeabur, OpenCode Console, Railway, now Northflank. All of them modern "we keep your content safe" players with high hopes and probably pretty good protections.

This is a full infosec emergency unfolding here. No matter how grinding it is to motivate a team to invest in protection or resilience (and it is grinding): We gotta get ready. Stat! The AI-accelerated attacker apocalypse has begun. It's boom, boom, boom, thanks for everything, on to the next. They have us sighted and are firing for effect.

So credential rotation, tripwire secrets, DFIR on standby, solid vulnerability and intrusion detection, real listening to that signal, minimized blast radius, spending caps and time limits and tight scoping on credentials, fast incident alerting... We need to get on a real war footing here.

A trick i've started doing is encrypting my env vars, then hard-code the decryption key in the code

that way if the provider gets popped, they need my binary to decrypt it

it shouldn't be necessary and it sounds stupid, but in cases like this, it proves invaluable (it appears they don't have customer code/binary access, but maybe that wasn't important enough to mention)

Unfortunately for infra startups, I think we're going to see a convergence back to hyperscalers as a result of incidents like this and railway.

There's no amount of "nicer to use" that can replace security, at least certainly when selling to enterprises

This could all go multiple ways but one of them is we end up in a place where technical security is just not possible to implement any more. What does that world look like?

Presumably it means much more behavioral security and inherent friction built in to everything we do. Maybe high value transactions will just inherently have to take multiple days to do, just to slow down things enough that human reaction time can equal what LLMs can execute?

On the technical side, I can see zero trust becoming completely saturated through every layer. The passkey you authenticate with isn't opening a gate to a trusted tier within which components trust each other - even within the same software component. Every internal function call at every layer doing hardware signature verification against continuously minted private keys (since we assume exposure, the lifetime can't be longer than milliseconds now).

It's really interesting to think through what a fundamentally robust security environent does look like if you bake in an assumption that technical security is impossible.

Is the Network Infrastructure Hiding Protocol work at IEEE going to help thwart AI assisted attacks, if adopted?

What is Northflank? I am surprised at the amount of new business out there

i wonder if there is a chain reaction point, when one compromised vendor holds secrets to another one and that becomes the next target. Northflank lists Sentry as their customer. So if Northflank secrets are decrypted and Sentry used them for deployment, you have a starting point to attack Sentry next which then opens up a lot more.

I think all of these companies need to let you separate secrets from other env vars and not let you ever read your secrets once they're written down.

We're observing a worrisome speed up in security breaches. To me it's clear that weaponized AI will cause more harm than some productivity benefits.

It’s the death of homegrown software.

> The runtime platform for autonomous software

Well, at least they were honest I suppose

I'm curious what people think of BYOC (bring your own cloud) as an option offering to customers or buying it as a customer. Looking at Northflanks website they list it as an offering, I presume those customers are not affected by the breach.

What are your opinion on the concept? Growing trend that should be encouraged or a mistake for most startups? Would you choose one provider over the other because of it assuming price was about the same?

I get that this is often a super expensive "call us" option today, but in theory it doesn't have to be, so for the sake of discussion let's assume it can be delivered at a reasonable cost for the provider at reasonable price for the customer.

If people from Northflanks is reading this, I'm sorry for the incident and I hope you recover as soon as possible.

Unfortunately, this is going to be an everyday story. More and more bugs and breaches will be discovered and this move fast world more and more will be continued to be made.

Security is a perception, but right now it seems like it needs good PR.

The Hacker News front page is just becoming a security incidence status page at this point. Everything seems pretty fucked.

The bigger the task, the bigger the surface, the bigger the risk.

How these Paas (Railway before) are being breached? Poor choices or something else? Scary Nonetheless. I thought AI advancement will make the self hosting painful. If PaaS are no good, then what can be done?

i'm surprised at the growth of all these neoclouds given the current security risk climate...

    Run agents and everything they build. Secure execution from the first sandbox

So if you want to judge as a customer for selection and figure out how likely a new(ish) PaaS company going to get breached, how would you evaluate them?

1. Ask them about what percentage of their own resources goes towards cybersecurity, audits etc. Would they even tell you.

2. Technical stack maturity. If the details are available, because they won't reveal any of it to you.

3. Other signals: Do they give talks on platform security, who is incharge of their cyber security.

doppler next?

not a good time to be a PaSS or SaSS ...

[flagged]

[dead]

So sad...

Unfortunately I'm not longer using PaaS tools. Instead I use Azure for work and vps now for all my personal stuff. With AI it's easier than ever to lock a server down tight and mitigate this kind of exposure.

I'm actually pretty pissed at this now. I have a throwaway appstore app that uses northflank for the backend, now I have to go in and rotate revenuecat secrets, apple secrets, ai secrets. For an app with basically 2 users.

Can I even delete the app on apple's app store or whats the process like what a pain in the ass. I'm pretty done tbh.

I feel like these breaches started to accelerate after Anthropic wrote a full post about how easy it is to abliterate glm 5.3 to make a frontier-level cyber model.

Now, to be clear, abliteration was no state secret, but that blog post was pure accelerationism.