Building a mirror of various Red Hat oriented "stuff"

Building a mirror for rpm-based distributions.

I've already described in brief how I built a mirror that currently mirrors Debian and Ubuntu on a daily basis. That was relatively straightforward given that I know how to install Debian and configure a basic system without a GUI and the ftpsync scripts are well maintained, I can pull some archives and get one pushed to me such that I've always got up to date copies of Debian and Ubuntu.

I wanted to do something similar using Rocky Linux to pull in archives for Almalinux, Rocky Linux, CentOS, CentOS Stream and (optionally) Fedora.

(This was originally set up using Red Hat Enterprise Linux on a developer's subscription and rebuilt using Rocky Linux so that the machine could be passed on to someone else if necessary. Red Hat 9.1 has moved to x86_64v2 - on the machine I have (HP Microserver gen 8) 9.1 it fails immediately. It has been rebuilt to use Rocky 8.8).

This is a minimal install of Rocky as console only - the machine it's on only has 4G of memory so won't run a GUI reliably. It will run Cockpit so can  be remotely administered. One user to run everything - mirror.

Minimal install of Rocky 8.7 from DVD .iso. SELinux is enabled, SSH works for remote access. SELinux had to be tweaked to allow /srv/ the appropriate permissions to be served by nginx. /srv is a large LVM volume rather than a RAID 6 - I didn't have enough disks

Adding nginx, enabling Cockpit and editing the Rocky Linux mirroring scripts resulted in something straightforward to reproduce.

nginx

I cheated and stole large parts of my Debian config. The crucial part to remember is that there is no autoindexing by default and I had to dig to find the correct configuration snippet.

  # Load configuration files for the default server block.

        include /etc/nginx/default.d/*.conf;

        location / {
                autoindex on;
                autoindex_exact_size off;
                autoindex_format html;
                autoindex_localtime off;
                # First attempt to serve request as file, then
                # as directory, then fall back to displaying a 404.
                try_files $uri $uri/ =404;
        }

 Rocky Linux mirroring scripts

Systemd unit file for service

[Unit]
Description=Rocky Linux Mirroring script

[Service]
Type=simple
User=mirror
Group=mirror
ExecStart=/usr/local/bin/rockylinux

[Install]
WantedBy=multi-user.target

 Rocky linux  timer file

[Unit]
Description=Run Rocky Linux mirroring script daily

[Timer]
OnCalendar=*-*-* 08:13:00
OnCalendar=*-*-* 22:13:00
Persistent=true

[Install]
WantedBy=timers.target

Mirror script

#!/bin/env bash
#
# mirrorsync - Synchronize a Rocky Linux mirror
# By: Dennis Koerner <koerner@netzwerge.de>
#
# The latest version of this script can be found at:
# https://github.com/rocky-linux/rocky-tools
#
# Please read https://docs.rockylinux.org/en/rocky/8/guides/add_mirror_manager
# for further information on setting up a Rocky mirror.
#
# Copyright (c) 2021 Rocky Enterprise Software Foundation

This is a very long script in total.

Crucial parts I changed only listed the mirror to pull from and the place to put it.

# A complete list of mirrors can be found at
# https://mirrors.rockylinux.org/mirrormanager/mirrors/Rocky
src="mirrors.vinters.com::rocky"

# Your local path. Change to whatever fits your system.
# $mirrormodule is also used in syslog output.
mirrormodule="rocky-linux"
dst="/srv/${mirrormodule}"

filelistfile="fullfiletimelist-rocky"
lockfile="/home/mirror/rocky.lockfile"
logfile="/home/mirror/rocky.log"

 Logfile looks something like this: the single time spec file is used to check whether another rsync needs to be run

deleting 9.1/plus/x86_64/os/repodata/3585b8b5-90e0-4856-9df2-95f646bc62c7-PRIMARY.xml.gz

sent 606,565 bytes  received 38,808,194,155 bytes  44,839,746.64 bytes/sec
total size is 1,072,593,052,385  speedup is 27.64
End: Fri 27 Jan 2023 08:27:49 GMT
fullfiletimelist-rocky unchanged. Not updating at Fri 27 Jan 2023 22:13:16 GMT
fullfiletimelist-rocky unchanged. Not updating at Sat 28 Jan 2023 08:13:16 GMT

It was essentially easier to store fullfiletimelist-rocky in /home/mirror than anywhere else.

Very similar small modifications to the Rocky mirroring scripts  were used to mirror the other distributions I'm mirroring. (Almalinux, CentOS, CentOS Stream, EPEL and Rocky Linux).

 

 

 

20230828 - OMGWTFBBQ - Breakfast is happening more or less

 And nothing changes: rediscovered from past Andrew at his first Cambridge BBQ and almost the first blog post here:

"House full of people I knew only from email, some very old friends. Wires and leads filling the front room floor - laptops _everywhere_ .

...

Thirty second rule on sofa space - if you left for more than about 30 seconds you had to sit on the floor when you got back (I jammed myself onto a corner of the sofa once I realised I'd barely get through the crush :) )
[Forget students in a mini / UK telephone box - how many DDs can you fit into a very narrow kitchen :) ]

It's a huge, dysfunctional family with its own rules, geeky humour and in-jokes but it's MINE - it's the people I want to hang out with and, as perverse as it sounds, just being there gave me a whole new reaffirmed sense of identity and a large amount of determination to carry on "wasting my time with Linux" and Debian"

The *frightening* thing - this is from August 31st 2009 ... where have the years gone in between.

20230826 - OMGWTFBBQ - BBQ still in full swing

 There's been a very successful barbeque running in the garden: burgers, sausages, beer, vegetarian dishes and then ice cream.

The chance to catch up with people you only meet in IRC. Talking and laughter - and probably a couple of games of Mao.

Thanks also to our sponsors - Collabora, Codethink and RattusRattus for contributions to food and drink.

20230826 OMGWTFBBQ - Cambridge is waking up

 The meat has been fetched: those of us in the house are about to get bacon sandwiches. Pepper the dog is in the garden. Time for the mayhem to start, I think.

Various folk are travelling here so it will soon be crowded: the weather is sunny but cool and it looks good for a three day weekend.

This is a huge effort that falls to Steve and Jo and a huge disruption for them each year - for which many thanks, as ever. [And, as is traditional on this blog, the posts only ever seem to appear from Cambridge].