fidzu : Fedora

Infra tickets migrated to new forejo forge

On tuesday I migrated our https://pagure.io/fedora-infrastructure (pagure) repo over to https://forge.fedoraproject.org/infra/tickets/ (forgejo).

Things went mostly smoothly, the migration tool is pretty slick and I borrowed a bunch from the checklist that the quality folks put together ( https://forge.fedoraproject.org/quality/tickets/issues/836 ) Thanks Adam and Kamil!

There are still a few outstanding things I need to do:

• We need to update our docs everywhere it mentions the old url, I am working on a pull request for that.

• I cannot seem to get the fedora-messaging hook working right It might well be something I did wrong, but it is just not working

• Of course no private issues migrated, hopefully someday (soon!) we will be able to just migrate them over once there's support in forgejo.

• We could likely tweak the templates a bit more.

Once I sort out the fedora-messaging hook I should be able to look at moving our ansible repo over, which will be nice. forgejo's pull request reviews are much nicer, and we may be able to leverage lots of other fun features there.

Mass rebuild finished

Even thought it started late (was supposed to start last wed, but didn't end up starting really until friday morning) it finished over the weekend pretty easily. There was some cleanup and such and then it was tagged in.

I updated my laptop and everything just kept working. I would like to shout out that openqa caught a mozjs bug landing (again) that would have broken gdm, so that got untagged and sorted and I never hit it here.

Scrapers redux

Wed night I noticed that one of our two network links in the datacenter was topping out (10GB). I looked a bit, but marked it down to the mass rebuild landing and causing everyone to sync all of rawhide.

Thursday morning there were more reports of issues with the master mirrors being very slow. Network was still saturated on that link (the other 10G link was only doing about 2-3GB/sec).

On investigation, it turned out that scrapers were now scraping our master mirrors. This was bad because all the BW used downloading every package ever over http and was saturating the link. These seemed to mostly be what I am calling "type 1" scrapers.

"type 1" are scrapers coming from clouds or known network blocks. These are mostly known in anubis'es list and it can just DENY them without too much trouble. These could also manually be blocked, but you would have to maintain the list(s).

"type 2" are the worse kind. Those are the browser botnets, where the connections are coming from a vast diverse set of consumer ip's and also since they are just using someone elses computer/browser they don't care too much if they have to do a proof of work challenge. These are much harder to deal with, but if they are hitting specific areas, upping the amount of challenge anubis gives those areas helps if only to slow them down.

First order of business was to setup anubis in front of them. There's no epel9 package for anubis, so I went with the method we used for pagure (el8) and just set it up using a container. There was a bit of tweaking around to get everything set, but I got it in place by mid morning and it definitely cut the load a great deal there.

Also, at the same time it seems we had some config on download servers for prefork apache. Which, we have not used in a while. So, I cleaned all that up and updated things so their apache setup could handle lots more connections.

The BW used was still high though, and a bit later I figured out why. The websites had been updated to point downloads of CHECKSUM files to the master mirrors. This was to make sure they were all coming from a known location, etc. However, accidentially _all_ artifact download links were pointing to the master mirrors. Luckly we could handle the load and also luckily there wasn't a release so it was less people downloading. Switching that back to point to mirrors got things happier.

So, hopefully scrapers handled again... for now.

Infra Sprint planning meeting

So, as many folks may know, our Red Hat teams are all trying to use agile and scrum these days. We have various things in case anyone is interested:

• We have daily standup notes from each team member in matrix. They submit with a bot and it posts to a team room. You can find them all in #cle-standups:fedora.im space on matrix. This daily is just a quick 'what did you do', 'what do you plan to do' any notes or blockers.

• We have been doing retro/planning meetings, but those have been in video calls. However, there's no reason they need to be there, so I suggested and we are going to try and just meet on matrix for anyone interested. The first of these will be monday in the #meeting-3:fedoraproject.org room at 15UTC. We will talk about the last 2 weeks and plan for what planned things we want to try and get in the next 2.

The forge projects boards are much nicer than the pagure boards were, and we can use them more effectively. Here's how it will work:

Right now the current sprint is in: https://forge.fedoraproject.org/infra/tickets/projects/325 and the next one is in: https://forge.fedoraproject.org/infra/tickets/projects/326

On monday we will review the first, move everything that wasn't completed over to the second, add/tweak the second one then close the first one, rename the 'next' to 'current' and add a new current one. This will allow us to track what was done in which sprint and be able to populate things for the next one.

Additionally, we are going to label tickets that come in and are just 'day-to-day' requests that we need to do and add those to the current sprint to track. That should help us get an idea of things that we are doing that we cannot plan for.

Mass update/reboot outage =========================o

Next week we are also going to be doing a mass update/reboot cycle with outage on thrusday. This is pretty overdue as we haven't done such since before the holidays.

comments? additions? reactions?

As always, comment on mastodon: https://fosstodon.org/@nirik/115951447954013009