Sunday, 9 February 2020

CD release time post 3 20202029 1653 - Release for Oldstable / Squeeze / 9.12 has happened

Just tidying up on the last of my tests. The release to the mirrors for 9.12 has happened.

Much sterling work by  amacater, Isy, RattusRattus, Sledge and Schweer (in alphabetical order). This one seemed to have turned up fewer bugs.

Lots of fun, the usual bouts of problems between chair and keyboard and a feeling for me of something worthwhile to share in.

My private CD mirror is now up to date with 10.3 and a few machines here have already been upgraded :) On a day when the weather is cold, wet, windy and rainy, it's been a good excuse to sit in the warm with a laptop.

Here's to the next one, then.

Saturday, 8 February 2020

CD release time post 2 20200208 2152 - lots of install CD/DVD/BD image testing going on

It's been a busy day. Lots of installs, a couple of bugs found, mostly going very well. The main 10.3 Debian update has been pushed to the mirror network and I've updated two of my machines so far. All good.

Hofstadter's Law states that "It always takes longer than you think, even taking account of Hofstadter's Law" ( the Douglas Hofstadter who wrote Goedel Escher Bach) and these long days always disappear into the memory until the next time.

There's also an update planned for the images for oldstable - so we'll be busy for a fair while yet.

UPDATE: The release looks good: some CDs for the smaller arches remain to be built but it's done for now.

Oldstable testing will happen tomorrow

As has become traditional - another blog post round about CD release time

Just waiting to start: Isy, Sledge, RattusRattus and Schweer are all standing by on IRC

Debian 10.3 update is out - the CDs will begin being built and tested this weekend. Would much rather be in Cambridge, as ever :)

Also dotting between various machines doing general updates and stuff: this is being written from my newer laptop and is the first update from this one.
As ever, it should be fun.

Sunday, 2 February 2020

Rebuilding a mirror - Fedora EPEL mirroring and Apache config

This follows on from the previous post: Fedora does things differently. EPEL is a sub-project: software packaged for CentOS/Red Hat Enterprise Linux by the Fedora Project maintainers.

Mirroring Fedora appears to be unusual if you're not a major (Tier 1) mirror.

The Fedora Wiki page at https://fedoraproject.org/wiki/Infrastructure/Mirroring
is very useful.

Public mirrors are required to have minimum bandwidth and, ideally, to include everything. Fedora is large: fedora-buffet is the whole thing under /pub to download for those who have many TB to spare. fedora-enchilada is everything under /pub/fedora (including the Fedora distribution itself), fedora-epel is EPEL alone.

Public mirrors are required to register with the Fedora Account System [FAS]: those undertaking private mirroring are requested to mirror from Tier1 mirrors or below. The script below relies on mirrors which include fedora-buffet. These are used to achieve rsync speed-ups (by accessing cached bundles of timestamps to speed up differential transfers based on the time files changed).

Since I'm not running a public mirror and not reporting back to the public mirror network stats using mirror-manager, I needed to find a Tier1 mirror that would permit me to mirror privately and my nearest is Hoch Schule Esslingen at ftp-stud.hs-esslingen.de

The suggested mirroring script is quick-fedora-mirror available from Fedora's git repositories at https://pagure.io/quick-fedora-mirror This is licensed under Creative Commons CC0.

The script is written in zsh (with features not included in bash or other shells) and Python3 so I had to do a quick apt-get install zsh. There is a quick-fedora-mirror.conf.dist file included as a template for editing. The suggestion for the completed quick-fedora-mirror.conf file is to move it to /etc

The script requires space somewhere to store a couple of files: timefile and timefile.prev which record the times that the script was last run. I found it easier to just put these into /home/mirror and I copied the quick-fedora-mirror script itself into /usr/local/bin/

One easy mistake to make: this script should be configured to write into the top level directory - so into /srv/mirrors rather than /srv/mirrors/epel or you end up with /srv/mirrors/epel/epel

Apache2 configuration

This was relatively easy - set the document root to /srv/mirrors by uncommenting the relevant lines for /srv in /etc/apache2/apache.conf and amend the /etc/apache2/000-default to show /srv/mirrors as the new DocumentRoot

This results in the six separate directories all being appropriately served under http://localhost as /centos, /debian, /debian-cd, /epel, /ubuntu and /ubuntu-releases.


Rebuilding a mirror - software mirroring of Linux distributions for fun

... and also because sometimes it's just easier to have a local cache

I've a mirror machine sitting next door: it's a private mirror of CentOS, Debian, Debian CDs, EPEL [extra packages for Enterprise Linux from Fedora], Ubuntu releases, and Ubuntu CDs. I've run this intermittently for a few years for myself: it's grown recently because I added CentOS and EPEL.

A 6TB LVM volume is now at about 62% full, so there's plenty of space.

Layout

My machine has these mirrors placed under /srv. The mirrors directory is root owned, the individual directories are owned by a non-root user which also runs the scripts. So you end up with /srv/mirrors/debian or /srv/mirrors/ubuntureleases, for example, where the final directory is owned by a mirror user and the mirror user owns the scripts and the crontab that runs them.

Package used for mirroring

The ftpsync Debian package is pretty much all you need: each individual download source has its own ftpsync.conf script which sits under /etc/ftpsync

The configuration files are essentially identical for Debian and Ubuntu - the only difference is in the source from which you mirror.

Configuring the ftpsync.conf script

The changes needed from the example configuration script in /usr/share/doc/ftpsync are minimal and straightforward: edit The TO directory and the source from which you mirror.

My ftpsync-debian.conf has the following lines changed from the supplied config file, for example.

TO="/srv/mirrors/debian/"

RSYNC_HOST="debian.hands.com"
RSYNC_PATH="debian"

Calling and usage:


Calling is simple: ftpsync sync:archive:debian which references the ftpsync-debian.conf above. Each ftpsync call must have a corresponding configuration file or the ftpsync script complains.

Usage is from mirror's crontab - excerpt below

13 03 * * * /usr/bin/ftpsync sync:archive:debian
37 03 * * * /usr/bin/ftpsync sync:archive:centos


Hope this helps someone: will add a couple of bits on configuring Apache to serve this in another post.

[EDIT - changed rsync host listed above at request of jcristau]





Saturday, 16 November 2019

Seems like I only ever write something at CD release time ... Buster release 2 is happening round about now

Updating CD images again for Buster release 2. I've spent a happy day chasing down various CD images and discovering my ignorance once again :)

Thanks to Sledge for having me around and also to Schweer - who wasn't feeling so well. No RattusRattus or Isy this time round - that meant I got to do a little more but I do miss their company.

All looks set fair for CD, DVD images and so on: the Jigdo16GB and the BD-1 isos built from jigdo files also worked well for me. This is getting to be a slick process and it's good to be a part of it.

And another one in a few months, I suspect :)

Sunday, 8 September 2019

Chasing around installing CD images for Buster 10.1 ...

and having great fun, as ever, making a few mistakes and contributing mayhem and entropy to the CD release process. Buster 10.1 point update just released, thanks to RattusRattus, Sledge and Isy and Schweer (amongst others).

Waiting on the Stretch point release to try all over again.. I'd much rather be in Cambridge, but hey, you can't have everything.