alt Hacker NewsTell HN: HN was down
- HN errored on all authenticated requests with 502 Bad Gateway. It did still respond to a limited amount of unauthenticated requests with presumably cached pages, which did not get updated. The last post on /newest claimed "0 minutes ago", but was actually much older (1:32:57 PM GMT) and not the newest post.
- This status page actually identified the outage: https://hackernews.onlineornot.com/ - Pages by Hund and Statuspal did not show the outage.
- The last post before the outage was https://news.ycombinator.com/item?id=46301823 (1:39:59 PM GMT). The last comment was https://news.ycombinator.com/item?id=46301848 (1:41:54 PM GMT).
- There was an average of ~4 seconds per comment just prior to the outage. Based on this, HN likely went down at 1:41:58 PM GMT.
Comments
I got stuck in an infinite loop.
Try opening HN -> it's down, better check HN to see everyone talking about a major website being down -> Try opening HN -> loop
Yes, and I'm a little ashamed to admit my morning routine wasn't the same without it.
I'm with Hund. Our hn.hund.io page did not catch this because it was requesting the cached, unauthenticated page. It now monitors authentication as well.
Is this still a valid account for HN status? It says it’s the official one, but with the changes at Twitter to no longer show chronological feeds (at least for users that aren’t logged in), it’s rather useless. The top 5 listed post (for me) are seemingly random from 2014 - 2022.
Is there a better place to check, beyond a basic down detector that may provide more insight or signal that the outage is acknowledged?
It would be a nice feature if hackernews had a simple news.ycombinator.com/status page that I could trust. I don't like loading those some-sketchy-domain/status-right-now sites.
When it was down my thought was "damnit, I'll actually have a productive workday now."
A lot of the outage indicators failed. Someone needs to create an outage indicator reliability dashboard.
Looks like 3 hrs
https://hackernews.onlineornot.com/incidents/yaz-eOJeARBL
https://downforeveryoneorjustme.com/hacker-news
Strangely, nothing from the statuspal, which is the first google result
PSA - if you delete your cookies, HN gets it easier. Or just test it in a private window.
It did work without being logged on. The auth service appeared to be down as the log in attempt (just showing the page) failed.
I was fortunate enough to get to watch a bunch of kids racing and bouncing in bouncy castles in our school gym during HN's downtime.
It's not that much different from HN, come to think of it.
(ha, ha)
Yeah I couldn't log in for a bit this morning. It's concerning how often and how many times I tried. Glad it's resolved.
I Still found it funny that we have downdetector, downdetectordowndetector, downdetectordowndetectordowndetector,... (I lost count) but downdetector doesn't detect HN's status.
As soon as I noticed it was down I came to hacker news to post about it, but...
Man if I had a coin every time I email the mods and this thing goes down, I'd strangely have two coins.
Yes, and on one request I saw a message like "Restarting server - this won't take long", and soon after it's back up.
Maybe PG is more involved now and hired the "10,000 lines of AI code a day" person who made a deployment mistake?
I was able to view the site without being signed in (i.e. private window) but any browser I was logged into wouldn't load.
I'm sure it's a coincidence but it started working again shortly after emailing [email protected]
It was the first time since I started using this website (August last year) that it was down.
I'm still impressed nonetheless.
I'd like to know what caused the outage and how it could have been prevented, for learning purposes.
I got irrationally angry when it refused to load the website
In other words, productivity in tech skyrocketed for hours..though it seems some work was flavoured with irrational anger.
In with the rest, yes, and my first thought is always "I have an internet outage" when HN is down. :-P
Yep "having trouble serving my request" or so.
The "site status" apps are all smoke and mirrors, how unreliable.
What was the longest that HN has been down? I feel like this is up there.
Could have been accidental flagging of sorts. Didn't work on PC for few minutes while showing fine on phone.
Will we get a post mortem?
Thought it had something to do with some model updates, like Gemini Flash 3 for a moment.
Yes, I got an error message, but I cleared my cookies and was able to access.
Hi, Got an error an hour ago on phone, not loged.
Anyway, glad to see you back.
Paris 1812.
Cheers from France.
Yes, it' been out for me too, southern hemisphere, GMT -03
I'll admit this ruined my morning
We need a serious post mortem for this
Athens, Greece yes it was down.
Yup, France here , Paris 1809.
HN was down about an hour ago.
Glad to see it back !
Cheers.
Yes, was down for me too this morning
Yes was down for me. Ontario, Canada.
Yes, it was down for a few hours
I got confused by the "minutes ago" thing.
Working with full dates in the HTML and doing a tiny JavaScript that calculates the "minutes ago" would actually be a neat improvement.
I have never before seen this website down.
hn applied ycombinator too strictly
I tried to refresh an embarrassing amount of times
Yes, sorry! We're investigating, but my current theory is we got overloaded because I relaxed some of our anti-crawler protections a few days ago.
(The reason I did that is that the anti-crawler protections also unfortunately hit some legit users, and we don't want to block legit users. However, it seems that I turned the knobs down too far.)
In this case, though, we had a secondary failure: PagerDuty woke me up at 5:24am, I checked HN and it seemed fine, so I told PagerDuty the problem was resolved. But the problem wasn't resolved - at that point I was just sleeping through it.
I'll add more as we find out more, but it probably won't be till later this afternoon PST.