alt Hacker NewsRate limits on GitLab.com are changing
Comments
Providing kickbacks to the repos being scraped would be a good way to help fund open source projects and pay creators like streaming services do. Seems like they're headed in this direction - it would be a massive product differentiator over GH
Damn, we’re even having Claude write important press releases now
60 requests an hour per IP seems low for a school or office network. A few people browsing issues and source files could use that up quite fast.
> You get 429 Too Many Requests with RateLimit-* headers and a Retry-After. Wait the interval it gives you, then retry.
Is there a test endpoint where one can validate the behavior of their ratelimit detection? Basically I do not want to cause excessive load on your servers just to test my implementation.
> A request that arrives with no credentials gets 60 requests per hour per IP address.
One request per minute.
60 requests per hour per IP if you haven't signed in... well that's unfortunately low.
I would spend thousands of dollars for gitlab in terms of: 1) better UX for admin panel, I'm not sure what I've enabled and what not. Several buttons do not disable the rest of the settings, leaving me with some doubts (e.g. if I disabled grafana, why is there a setting that talks about where/how I store?) 2) a minimal version of gitlab without all the AI
Github would hit four nines if they followed suit. no clue why the dont try
Walk back in 10...9...
I think it's because people are building agentic flows, reducing the amount of developer seats needed. It's the first step towards usage based pricing.
Congrats on making the world worse with AI. All this performative data scraping and uploading and no progress at all.
I am honestly surprised they aren't going lower at this point.
Gitlab must pay a fortune to bot traffic, most of which is malicious or garbage at best.
I understand why they're doing this, but the anticausative title kinda rubs me the wrong way.
Worst case, this could be the start of a paywall to learn from, contribute to, or host open source projects.
Hopefully they find some kind of carve out for OSS projects while still blocking the egregious offenders.
> The requested URL was not found on this server.
Getting that so I do not know exactly what they are doing. From the title I am guessing they are restricting or throttling if downloads exceeds some value.
I noticed that recently Github.com has some kind of weird bot detection on public repos. I have a browser extension for switching User Agents for a specific legacy site, sometimes i forget to turn it off and Github will require me to login to view public repos.
All of this is most likely due to mass scraping by LLMs. Welcome to the total shitification of the web.
If you are using LLMs to interact with sites like GitLab and GitHub, and you have the option to use a GraphQL API, you should jump on it immediately.
GraphQL is absolutely terrible for human developers to interact with, but it's like Facebook could see into the future back in 2012. I cannot imagine a more perfect API surface for agents. With the REST API on GitHub, you can consume maybe 10 issue JSON blobs before your context window is blown out. With GraphQL constraining the results you can easily read hundreds in the same token budget.
Additionally, the # of requests your agents need to make can be reduced in many cases since GraphQL can join across types whereas REST APIs cannot. You essentially get savings in two dimensions here. Quota and raw token volume per logical response.