Sunday, 2 February 2020

Rebuilding a mirror - software mirroring of Linux distributions for fun

... and also because sometimes it's just easier to have a local cache

I've a mirror machine sitting next door: it's a private mirror of CentOS, Debian, Debian CDs, EPEL [extra packages for Enterprise Linux from Fedora], Ubuntu releases, and Ubuntu CDs. I've run this intermittently for a few years for myself: it's grown recently because I added CentOS and EPEL.

A 6TB LVM volume is now at about 62% full, so there's plenty of space.

Layout

My machine has these mirrors placed under /srv. The mirrors directory is root owned, the individual directories are owned by a non-root user which also runs the scripts. So you end up with /srv/mirrors/debian or /srv/mirrors/ubuntureleases, for example, where the final directory is owned by a mirror user and the mirror user owns the scripts and the crontab that runs them.

Package used for mirroring

The ftpsync Debian package is pretty much all you need: each individual download source has its own ftpsync.conf script which sits under /etc/ftpsync

The configuration files are essentially identical for Debian and Ubuntu - the only difference is in the source from which you mirror.

Configuring the ftpsync.conf script

The changes needed from the example configuration script in /usr/share/doc/ftpsync are minimal and straightforward: edit The TO directory and the source from which you mirror.

My ftpsync-debian.conf has the following lines changed from the supplied config file, for example.

TO="/srv/mirrors/debian/"

RSYNC_HOST="debian.hands.com"
RSYNC_PATH="debian"

Calling and usage:


Calling is simple: ftpsync sync:archive:debian which references the ftpsync-debian.conf above. Each ftpsync call must have a corresponding configuration file or the ftpsync script complains.

Usage is from mirror's crontab - excerpt below

13 03 * * * /usr/bin/ftpsync sync:archive:debian
37 03 * * * /usr/bin/ftpsync sync:archive:centos


Hope this helps someone: will add a couple of bits on configuring Apache to serve this in another post.

[EDIT - changed rsync host listed above at request of jcristau]





2 comments:

  1. I have a question on the choice of path.

    I have a server set up mirroring Ubuntu at the moment, and I want to add Debian mirroring to it.

    The drive set up currently is:

    /dev/sda2 is mounted as /media/8TB
    /dev/sda3 is mounted as /media/8.1TB
    /dev/sda4 is mounted as /media/8.2TB

    Within /media/8TB I have ubuntu-cdimage/ and ubuntu-mirror/. The drive is 1/3 full.

    Within /media/8.1TB I have ubuntu-oldimage/. It's 75 percent full, so it's complete.

    Within /media/8.2TB I have nothing. It's empty.

    I'm anticipating debian-cd/ and debian/ (packages) going into /media/8TB, but I'm not clear where to point ftpsync for the TO variable.

    Do I have enough space on /dev/sda1 and /dev/sda3 to mirror *everything* Debian?

    ReplyDelete
  2. The top lines of my config for debian-cd are

    MIRRORNAME=`hostname -f`
    TO="/srv/mirrors/debian-cd/"
    MAILTO="$LOGNAME"
    # HUB=false

    for example. So wherever the mount point is
    that you have for /dev/sda4 so /media/8.2TB/debian-cd , maybe?

    The backup config for apache in Debian has a prebuilt entry to display everything under /srv as discussed.

    debian-cd is 203G

    [That's the latest Debian media images so
    those for Debian 11 Bullseye for all architectures - so 11.0 as released on
    release day].

    debian is 1.4T - All current distributions, all current architectures - so sid -> bookworm -> bullseye -> buster -> stretch -> jessie


    ReplyDelete