nerd

30 Aug 2026

30 Aug 2026 10:46pm GMT

Slashdot

AI's 'Creepy' Crawlers Criticized by Linux Foundation's IT Infrastructure Director

The Linux Foundation's director of IT infrastructure says they now spend more CPU cycles "rendering commits for scrapers than we spend on all other kinds of legitimate access." At any one time, across 5 geo-distributed nodes, there are 14 CPU cores doing nothing but rendering git commits as html.... [W]hen a source is guaranteed to be LLM-free, like the entire history of kernel commits, it's worth its weight in gold as a source of training data... At the time of writing, linux.git is about 1.48 million commits. Oh, and we have about 922 forks of it on git.kernel.org - but don't worry, it's actually extremely efficient on the backend, since it's mostly the same objects in every fork. Unless, of course, you're a scraper, in which case you have, oh, several BILLION valid URLs you can scrape, only to get 922 duplicates of the same 1.48 million commits - which is exactly what the scrapers are doing. But wait, it's not just commits itself. You can also ask for patches, plain renders, diffs between arbitrary commits - cgit is happy to let you, which was perfect for the times when the Internet was for humans or crawlers who obeyed robots.txt, and is AWFUL right about now, because we can generate 1.2 METRIC BAJILLION valid URLs just for a single fork of linux.git. Initially, this was the solution - look through the logs, find out which IPs are obvious scraper bots, and fail2ban them. At first, this was easy, because the bots helpfully told you who they were via their user-agent. Then, they wised up and started pretending that they were random vanilla browsers. So, we started banning them by IP - after all, it's easy to figure out that an IP that is trying to grab every possible commit in a 8-year-old abandoned fork of linux is not really some lone Chrome on Windows user who is just furiously clicking every link that comes across their screen. The bots then started fanning out to entire subnets, but this was still meh, because obviously an IP coming from Google Compute is just pretending to be a Firefox user... And... that's when things turned really, really ugly. Suddenly, the crawlers were coming from millions of random residential or mobile IPs, all pretending to be random modern browsers. An IP like that would make 4-5 requests and then never show up in the logs again... They descended like swarms of locust, hit hard and fast until the system fell over and then moved on to the next target until you recovered. Then, they returned. Rinse. Repeat. They still do that - welcome to the wonderful world of "proxy SDK monetization." It's big business, and your TV is probably doing it... Today, git.kernel.org receives about 6M daily requests demanding to see random commits. Of these, 66% are still immediately batted away with the Anubis challenge, but 33% are now solving the math and getting through to the main site - because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge... With a bunch of generous assumptions, legitimate requests are only about 2% of git.kernel.org traffic - everything else are scrapers... [W]e're turning off features to reduce the number of crawlable URLs and to gate off actions that are expensive for us to run. Expect to lose some functionality, at least when accessing our resources anonymously. Trust me, we hate it just as much as you, but at this point it's a necessity... [W]e promise to still offer all of our data for download to anyone who asks. You just may have to jump through more hoops to get it. Sorry.

Hacker News

Show HN: Academa – Long-form STEM lecture videos generated by LLMs

Comments

30 Aug 2026 10:22pm GMT

Mark Zuckerberg's Social Reckoning

Comments

30 Aug 2026 9:41pm GMT

Slashdot

Chess.com Launches New Poker Site, Plans More Classic Game Sites with Player Ratings - Thanks to AI Protyping

In May Chess.com "quietly launched" a free educational poker site named Gambit, reports The Verge, where players can learn the game and improve an Elo-like rating without risking real money. And next Chess.com will expand into more classic games, including go, backgammon, and mahjong: Like Gambit, Chess.com is developing Elo-style rating systems for these games, as well. [Chess.com's chief growth officer, Albert Cheng] didn't say when the new sites will launch but that "development is moving quite rapidly" on them, using the same playbook he used for Gambit... He and one other employee took Gambit from prototype to launch within a year, partly with help from AI, mainly for coding tasks. "I think the biggest advantage that it's provided is essentially being able to read our existing codebase and our design system and apply consistent patterns to how we develop Chess.com over to a different platform and do so quickly," Cheng said.... "Just because code is easier to write doesn't mean that you just let it run on its own, but it certainly is a superpower in getting things off the ground." Cheng also said the version of Gambit that's live at the time of writing has "zero or close to zero" AI-created assets, which were used in early prototypes. According to Cheng, without help from AI, getting another site up and running would have required a bigger team or "would have been a lot more disruptive to the status quo of building and growing Chess." Now that a "small, scrappy" team can get a new platform off the ground much faster, it's opened the door for Chess.com to dip into more games, starting with poker. The poker site already has 75,000 members, according to a blog post Friday - and they've already played nearly 20 million hands and completed 100,000 lessons. The site is now holding a daily poker tournament, and on Monday will broadcast the event with live commentary on their Twitch and YouTube channels.

Google Maps Renamed Lake Ontario to 'Lake America' - But Only for U.S. Users

It's already happened - at least on Google Maps. Google's online maps reflect name changes in official government sources, Google posted Saturday. So when the U.S. Geographic Names Information System formally changed the name for "Lake Ontario" to "Lake America," Google also renamed it "Lake America" - for visitors from the U.S. "Those in Canada will continue to see 'Lake Ontario'," Google said. And people from every other country in the world... "will see both names." Though here's how it will be written... Lake Ontario (Lake America) "These updates follow our long-standing policy for bodies of water with names that vary from country to country, and are starting to roll out now." Meanwhile, NPR reports that Ontario's Premier posted a billboard reading "Lake Ontario - Now and Always" on Canada's side of the lake (repeating the message in French). And The Daily Beast reports Google move has also drawn some criticism online: "Hey you gave one of the Great Lakes the wrong name; pretty embarrassing mistake so you might want to fix that," one user wrote on X... MapQuest has said it will not be changing the name, and instead launched a new tool that allows users to rename it themselves to whatever they want.