Sunday, 2 February 2020

Rebuilding a mirror - software mirroring of Linux distributions for fun

... and also because sometimes it's just easier to have a local cache

I've a mirror machine sitting next door: it's a private mirror of CentOS, Debian, Debian CDs, EPEL [extra packages for Enterprise Linux from Fedora], Ubuntu releases, and Ubuntu CDs. I've run this intermittently for a few years for myself: it's grown recently because I added CentOS and EPEL.

A 6TB LVM volume is now at about 62% full, so there's plenty of space.

Layout

My machine has these mirrors placed under /srv. The mirrors directory is root owned, the individual directories are owned by a non-root user which also runs the scripts. So you end up with /srv/mirrors/debian or /srv/mirrors/ubuntureleases, for example, where the final directory is owned by a mirror user and the mirror user owns the scripts and the crontab that runs them.

Package used for mirroring

The ftpsync Debian package is pretty much all you need: each individual download source has its own ftpsync.conf script which sits under /etc/ftpsync

The configuration files are essentially identical for Debian and Ubuntu - the only difference is in the source from which you mirror.

Configuring the ftpsync.conf script

The changes needed from the example configuration script in /usr/share/doc/ftpsync are minimal and straightforward: edit The TO directory and the source from which you mirror.

My ftpsync-debian.conf has the following lines changed from the supplied config file, for example.

TO="/srv/mirrors/debian/"

RSYNC_HOST="debian.hands.com"
RSYNC_PATH="debian"

Calling and usage:


Calling is simple: ftpsync sync:archive:debian which references the ftpsync-debian.conf above. Each ftpsync call must have a corresponding configuration file or the ftpsync script complains.

Usage is from mirror's crontab - excerpt below

13 03 * * * /usr/bin/ftpsync sync:archive:debian
37 03 * * * /usr/bin/ftpsync sync:archive:centos


Hope this helps someone: will add a couple of bits on configuring Apache to serve this in another post.

[EDIT - changed rsync host listed above at request of jcristau]





No comments:

Post a Comment