fidzu : fedora

Inscrutable vHMC

We have a vHMC vm. This is a virtual Hardware Management Console for our power10 servers. You need one of these to do anything reasonably complex on the servers. I had initially set it up on one of our virthosts just as a qemu raw image, since thats the way the appliance is shipped. But that was making the root filesystem on that server be close to full, so I moved it to a logical volume like all our other vm's. However, after I did that, it started getting high packet loss talking to the servers. Nothing at all should have changed network wise, and indeed, it was the only thing seeing this problem. The virthost, all the other vm's on it, they were all fine. I rebooted it a bunch, tried changing things with no luck.

Then, we had our mass update/reboot outage thursday. After rebooting that virthost, everything was back to normal with the vHMC. Very strange. I hate problems that just go away where you don't know what actually caused them, but at least for now the vHMC is back to normal.

Mass update/reboot cycle

We did a mass update/reboot cycle this last week. We wanted to:

• Update all the RHEL9 instances to 9.7 which just came out

• Update all the RHEL10 instances to 10.1 which just came out.

• Update all the fedora builders from f42 to f43

• Update all our proxies from f42 to f43

• Update a few other fedora instances from f42 to f43

This overall went pretty smoothly and everything should be updated and working now. Please do file an issue if you see anything amiss (as always).

AI Scrapers / DDoSers

The new anubis is working I think quite well to keep the ai scrapers at bay now. It is causing some problems for some clients however. It's more likely to find a client that has no user-agent or accept header might be a bot. So, if you are running some client that hits our infra and are seeing anubis challenges, you should adjust your client to send a user-agent and accept header and see if that gets you working again.

The last thing we are seeing thats still anoying is something I thought was ai scraping, but now I am not sure the motivation of it, but here's what I am seeing:

• LOTS of requests from a large amount of ip's

• fetching the same files

• all under forks/$someuser/$popularpackage/ (so forks/kevin/kernel or the like)

• passing anubis challenges

My guess is that these may be some browser add on/botnet where they don't care about the challenge, but why fetch the same commit 400 times? Why hit the same forked project for millions of hits over 8 or so hours?

If this is a scraper, it's a very unfit one, gathering the same content over and over and never moving on. Perhaps it's just broken and looping?

In any case currently the fix seems to be just to block requests to those forks, but of course that means the user who's fork it is cannot access them. ;( Will try and come up with a better solution.

RDU2-CC to RDU3 move

This datacenter move is still planned to happen. :) I was waiting for a new machine to migrate things to, but it's stuck in process, so instead I just repurposed for now a older server that we still had around. I've setup a new stg.pagure.io on it and copied all the staging data to it, it seems to be working as expected, but I haven't moved it in dns yet.

I then setup a new pagure.io there and am copying data to it now.

The current plan if all goes well is to have an outage and move pagure.io over on december 3rd.

Then, on December 8th, the rest of our RDU2-CC hardware will be powered off and moved. The rest of the items we have there shouldn't be very impactful to users and contributors. download-cc-rdu01 will be down, but we have a bunch of other download servers. Some proxies will be down, but we have a bunch of other proxy servers. After stuff comes back up on the 8th or 9th we will bring things back on line.

US Thanksgiving

Next week is the US Thanksgiving holiday (on thursday). We get thursday and friday as holidays at Red Hat, and I am taking the rest of the week off too. So, I might be around some in community spaces, but will not be attending any meetings or doing things I don't want to.

comments? additions? reactions?

As always, comment on mastodon: https://fosstodon.org/@nirik/115595437083693195