alt Hacker News"Hetzner decided to cancel our account and terminate all servers"
Comments
Backups are irrelevent here (yes, backups are important). if hetzner really deleted production data without warning or providing a grace period for their customer to migrate their data, then they are simply not a stable foundation to build on.
I have never been a customer of google cloud for this reason and i sure as hell wont deploy new servers on hetzner until they provide a clear statement on what went wrong and what they will do to make sure they never screw up like this again.
Hetzner, the ball is in your court.
Hang on, Hetzner literally deleted all their data without warning?
That’s actually insane and business killing. Both for Hetzner’s reputation and potentially for their customer.
Most of the times you hear people complaining about Hetzner shutting down someone's servers, it's because they were hosting content going against their ToC or similar.
But this seems to be about Kiwix (which in short is "offline Wikipedia" in various ways) and doesn't seem to be about questionable content in any way.
Eventually I guess we'll get Hetzner's perspective on this, as they tend to start writing publicly about issues once the other side starts writing publicly about it as well.
Personally I've been a happy user of Hetzner for many years, with no issues that weren't my own doing. But reading about people having their servers deleted in the middle of the night on a Sunday (Berlin time) and all data wiped immediately, with no recurse, does sound a bit aggressive. Luckily it seems like both me and Kiwix has mirrors for the data we care about.
For anyone else who needs to hear this,
Hi,
I don’t have a mastodon to reply directly to you.
But i have had some issues with content being taken down by VPS providers as well.
What I’ve found works well is to use a VPS provider that the public is unaware of. And for some time I had used OVH based on the unlimited bandwidth and the reasoning that Wikipedia and Julian assange (who have far more enemies than I ever will) were using OVH.
I don’t know if that’s true any more because I subsequently moved my content to ENS and IPFS.
Anyway regardless of where your content is actually hosted or lives,
What I had done was turn my “real” servers into content origins , which were concealed form the rest of the world and lock it down in the firewall so it could only be reached by disposable squid proxy servers with a 10-liner config file
Then I pointed DNS , cloud flare etc at the squid nodes
And couldn’t care less if they were taken down.
Because I could deploy new ones in minutes elsewhere.
I didn’t have “bad content”, just ruthless business competition that kept coming at me like Tonya Harding.
And I’m sharing because your content didn’t seem too offensive either.
In the front end VPS nodes you’d just put the real address of your content as the remote origin.
And then nobody but you will ever know where it is.
Then generally your hosting company shouldn’t be aware of what it is either unless they’re snooping around in your files, and if they are, hell with them too.
You’re welcome to pass this along as a remark on avoiding censorship, or keep it to yourslelf as proprietary information I don’t mind. Let me know if you want or need an example squid conf. It’s seriously 10 lines at most and many examples found on google.
Not the first time this is happening:
- Ask HN: Hetzner banned me with no explanation. What can I do? (https://news.ycombinator.com/item?id=32318524)
- Hetzner didn't even provide a detailed info on why they deactivated my account (https://news.ycombinator.com/item?id=40781617)Hetzner froze my account because I owed them 0.02 EURO's. It was not possible to pay it with a VISA credit or VISA debit, nor Amex card. They required me to wire transfer the money. However my bank does not allow the wiring of a 0.02 EURO amount, as the amount is too low.
Out of pure spite I built my own data center.
also negative experiences here. if they get a copyright-violation request from someone, they won't contact you about it. they'll just take your server down immediately and ask you to respond. obviously thats not a sane course of action and i cannot recommend using them for any kind of production systems.
i am always angry if i see articles about them here on HN because such a vendor should be blacklisted and not promoted.
I consider myself lucky to never run into anything like this, because Hetzner doesn't even allow me to sign up in the first place. Yep, I went as far as uploading my real US driver's license -- only because I heard good stuff about Hetzner (something I would normally never do). They are like, sorry, still can't tell if you are a robot.
Dodged a bullet.
Been seeing a lot of negative posts surrounding experiences with Hetzner of late. Definitely facing issues and losing reputation.
They do normally send out termination mails. You can see an example of one here (note the full month notice)
https://lowendspirit.com/discussion/comment/191966#Comment_1...
Would definitely be good to hear hetzners side of the story because all the cases I’ve seen thus far turned out to be a case of initial telling being understandably upset but leaving out crucial details.
They definitely are trigger happy with telling customers to find someone else & generally don’t elaborate on why
Heh, this happened to me the other day. I had hooked it up to paypal and didn't realize it wasn't set up to autopay, so I had an outstanding balance of $8 for about a week and they nuked everything. It was just for a hobby project so it was no big deal and I'll provision a new server with them, but I'm not sure I'd use them for a serious project, even though their prices are good. Granted, if it was for a serious project, I would have spent more time and care setting stuff up.
The funny bit was I paid the invoice, and then my account remained suspended. When support finally got back to me a few days later, they said (and I quote)
Dear Client
We want to give you one last chance as a gesture of goodwill, so we revoked the cancellation for you.
Kind regards
I have a small number of very important servers on Hetzner and stories like this make me scared, but I haven't found a cost-effective equivalent for the "Storage Box" product - real block storage. I'm paying €11 a month for 5TB of storage. Is there any competition for that?
I can't stress how important it is to own your own hardware and colocate. Also, if you are paying for a dedicated server, you can often save money by moving to colocation.
After their experience with targeted "deplatforming" rumble started its own cloud: https://www.rumble.cloud/
8TB isn't too bad to restore. At that scale they can backup on a local drive daily for very little money
Been a Hetzner customer for years and have considered using them for a new business project of mine. Will reconsider it partly after reading this. At least use a separate provider for backups so I can quickly recover, just in case.
Seeing it happen to a reputable project such as Kiwix [0] definitely damages my perception of Hetzner. I've read numerous complains on Reddit a few months ago but they mostly boiled down to breaching the ToS in obvious ways. Still, not giving a heads up before cancelling a service and no option to recover data is just bad business practice.
[0] (I've deployed Pi's with Kiwix in remote areas in Africa, it's an amazing project)
Hetzner = Cloud BOFH?
That said, I host on them too. But some stuff is on nearlyfreespeech.net.
This thread is not really interesting because we don’t have the Hetzner’s side of the story.
Would be good to have a fall-back solution, is there something similar in dedicated server price in the EU as Hetzner? Or does no on else come close?
The biggest issue I have had with Hetzner was with a dedicated server. I was constantly (3 times or more a week) getting abuse messages about my MAC address not being correct:
"""" We have detected that your server is using different MAC addresses from those allowed by your Robot account.
Please take all necessary measures to avoid this in the future and to solve the issue. We also request that you send a short response to us. This response should contain information about how this could have happened and what you intend to do about it. In the event that the following steps are not completed successfully, your server can be locked at any time after DATEHERE.
How to proceed: - Solve the issue - Please note, in case you have fixed the problem, please wait at least 10 minutes before rechecking: https://abuse.hetzner.com/retries/?token=TOKENHERE - After successfully testing that the issue is resolved, send us a statement by using the following link: https://abuse.hetzner.com/statements/?token=TOKENHERE
Please visit our FAQ here, if you are unsure how to proceed: https://docs.hetzner.com/robot/dedicated-server/faq/error-fa... """
I was just using standard Docker to host a web app. No proxmox or KVM of any sort. I would just wait the 10 minutes, click their link https://abuse.hetzner.com/retries/?token=TOKENHERE, which would retry and would come back fine and my response would be "I changed nothing and the retry came back solved. I've done tcpdumps over a weeks time to see if any MAC addresses leak from the OS and none have while a similar ticket like this gets opened every couple days." The ticket would close shortly after I submitted.
I inquired to them at least twice about this and they just kept telling me I was leaking a MAC address that I wasn't allowed to even when I had proof of tcpdumps over a week time period. I found someone else who had this issue with them (most issues around this that I found were people hosting Proxmox) and they had Hetzner replace the NIC and it fixed the issue. Well, Hetzner wouldn't replace my NIC because "it was working" even though I referenced these abuse tickets. I ended up getting another dedicated server, migrated my app over there, and I haven't had issues since.
Their support is seriously not very good. Since that experience, I have had backups elsewhere and test restoring those backups regularly. The price to performance I get from them is unbeatable and like I said, I haven't had issues since getting a new machine. But, I'm definitely cautious and don't exactly trust things to not go sideways even though it's been 2 years since that experience.
I don't know why this is a surprise to anyone wrt Hetzner. Users have repeatedly warned that Hetzner terminates accounts of clients that they do not like. Hetzner does this without warning, even having the audacity to send you a bill thereafter.
As an example, you run any crypto related operation, even if it's a mere 5% of your workload, you will have this happen to you. You don't even have to be hosting anything at all.
Huge missed opportunity to use “name-and-fame”.
This is not good.
It does raise an interesting question of how to reliably contact a customer if email is broken?
People don't like hearing this, but Hetzner support is horrible. In the two years we'd had an account with them having used numerous auctioned boxes, we had to reach out to support a handful of times, and every single time they'd started the conversation by telling us it's not their business to help us. They supposedly only help if something's broken, however when we DID run into technical issues, like NVMe's slowing down to a halt, or transient networking issues, they would go out of their way to tell us they don't give a shit.
We cancelled our account last month because of that.
I cannot imagine the world of hurt that we'd be ushered in, had they actually dropped our data wholesale like they did for OP.
hetzner is cheap, but cheap often has hidden costs / risks.
aws, azure, and gcp aren’t cheap , but they offer better stability—both technically and operationally.
I've had many bad experiences with Hetzner, from taking my server offline because someone posted something bad and created an Abuse report, to unwillingness to cooperate to let me keep my ipv6 subnet after a forced move of data centers, to many minor shenanigans. Oh and banning my forum account because I was defending myself against some racist accusations (he was the racist)
I am always recommending to not build on Hetzner.
Ok but on topic, who is this guy and why did they do this to him?
We need to boycott hetzner.
This headline could probably use more context; even from the thread, it's hard to tell who's account was cancelled or what the significance of that might be.
Of course, this is very concerning. I'll wait to see what their response is. I do understand there's many reasons to trash Hetzner as they are much much cheaper than the big 3 hyperscalers and many HN posters are employed by them.
Unsuprising. The crypto validators in the past were the canaries in the coal mine for Hetzner and almost no-one cared when they were cancelled off of Hetzner. Now they terminated your servers and the same has happened to them.
After all, Hetzner is now priotizing shareholder value and is removing smaller customers wasting their compute resources.
That sucks. I was literally trying to download some files into kiwix and it didn't work.
Some of the files they host are pretty big, so maybe Hetzner just decided it wasn't worth hosting any more.
I've been using Hetzner for years though and never had an issue. But I don't get anywhere close to the 20TB traffic limit.
This reminds me that I should set up some backups though.
Three possibilities come to mind:
1) There is some fundamental data aspect Kiwix hasn't mentioned (or is entirely unaware of). I.e. CP or some other super illegal stuff.
2) Hetzner is profoundly incompetent, deleted production servers by accident, and the "But we sent you an email!" thing is a lie to cover up the mistake.
3) There is some kind of interaction that happened prior to this that we aren't privy to. Perhaps a series of late bills, legal threats, or some other inter-personal issue.
Predicted outcome:
I either expect Kiwix to get a knock by federal/national authorities. Or the more likely outcome in my opinion: some frustratingly vague statement by Hetzner PR about its customers being "mistaken" in regards as to why data-go-poof.
I mean seriously, let us assume it's something illegal: Sure, fine, whatever. Wouldn't it make more sense for that material to not be deleted, so whoever the guilty party is would be arrested for/prosecuted by it? Deleting the servers would be like police being informed about a murder weapon and asking the tipster to destroy the weapon before an arrest is even made. It doesn't make any sense to me. Surely if some bad thing were discovered, there would be some method to encrypt/restrict illicit material without destroying it.
Either bad blood, or unpaid bills, or simple incompetence seems like the most likely culprits to me.
Hetzner did something similar a couple of years ago, suddenly disabling 1000 Solana validators that were using their service:
https://www.theblock.co/post/182283/1000-solana-validators-g...
[flagged]
Whenever I ask a CTO if they have a backup (or plan-B) they say we're on AWS, we backup there and they will never go down as a company. And then I ask them what they do when their account gets shut (e.g. because they are selling something bad on Amazon and have the same phone number as the company account?) Or the instance some years ago where GCP closed because someone had wrongly classified image on their drive?
You should have all you backups in a different location and terraform tested with a different cloud provider, otherwise you're risking the company.
[Edit] Where I come from: That doesn't say anything about Hetzner, I have been with them for 20+ years, they have stopped individual servers in that time frame, but haven't cancelled my whole account.