Just tidying up on the last of my tests. The release to the mirrors for 9.12 has happened.
Much sterling work by amacater, Isy, RattusRattus, Sledge and Schweer (in alphabetical order). This one seemed to have turned up fewer bugs.
Lots of fun, the usual bouts of problems between chair and keyboard and a feeling for me of something worthwhile to share in.
My private CD mirror is now up to date with 10.3 and a few machines here have already been upgraded :) On a day when the weather is cold, wet, windy and rainy, it's been a good excuse to sit in the warm with a laptop.
Here's to the next one, then.
Sunday, 9 February 2020
Saturday, 8 February 2020
CD release time post 2 20200208 2152 - lots of install CD/DVD/BD image testing going on
It's been a busy day. Lots of installs, a couple of bugs found, mostly going very well. The main 10.3 Debian update has been pushed to the mirror network and I've updated two of my machines so far. All good.
Hofstadter's Law states that "It always takes longer than you think, even taking account of Hofstadter's Law" ( the Douglas Hofstadter who wrote Goedel Escher Bach) and these long days always disappear into the memory until the next time.
There's also an update planned for the images for oldstable - so we'll be busy for a fair while yet.
UPDATE: The release looks good: some CDs for the smaller arches remain to be built but it's done for now.
Oldstable testing will happen tomorrow
Hofstadter's Law states that "It always takes longer than you think, even taking account of Hofstadter's Law" ( the Douglas Hofstadter who wrote Goedel Escher Bach) and these long days always disappear into the memory until the next time.
There's also an update planned for the images for oldstable - so we'll be busy for a fair while yet.
UPDATE: The release looks good: some CDs for the smaller arches remain to be built but it's done for now.
Oldstable testing will happen tomorrow
As has become traditional - another blog post round about CD release time
Just waiting to start: Isy, Sledge, RattusRattus and Schweer are all standing by on IRC
Debian 10.3 update is out - the CDs will begin being built and tested this weekend. Would much rather be in Cambridge, as ever :)
Also dotting between various machines doing general updates and stuff: this is being written from my newer laptop and is the first update from this one.
As ever, it should be fun.
Debian 10.3 update is out - the CDs will begin being built and tested this weekend. Would much rather be in Cambridge, as ever :)
Also dotting between various machines doing general updates and stuff: this is being written from my newer laptop and is the first update from this one.
As ever, it should be fun.
Sunday, 2 February 2020
Rebuilding a mirror - Fedora EPEL mirroring and Apache config
This follows on from the previous post: Fedora does things differently. EPEL is a sub-project: software packaged for CentOS/Red Hat Enterprise Linux by the Fedora Project maintainers.
Mirroring Fedora appears to be unusual if you're not a major (Tier 1) mirror.
The Fedora Wiki page at https://fedoraproject.org/wiki/Infrastructure/Mirroring
is very useful.
Public mirrors are required to have minimum bandwidth and, ideally, to include everything. Fedora is large: fedora-buffet is the whole thing under /pub to download for those who have many TB to spare. fedora-enchilada is everything under /pub/fedora (including the Fedora distribution itself), fedora-epel is EPEL alone.
Public mirrors are required to register with the Fedora Account System [FAS]: those undertaking private mirroring are requested to mirror from Tier1 mirrors or below. The script below relies on mirrors which include fedora-buffet. These are used to achieve rsync speed-ups (by accessing cached bundles of timestamps to speed up differential transfers based on the time files changed).
Since I'm not running a public mirror and not reporting back to the public mirror network stats using mirror-manager, I needed to find a Tier1 mirror that would permit me to mirror privately and my nearest is Hoch Schule Esslingen at ftp-stud.hs-esslingen.de
The suggested mirroring script is quick-fedora-mirror available from Fedora's git repositories at https://pagure.io/quick-fedora-mirror This is licensed under Creative Commons CC0.
The script is written in zsh (with features not included in bash or other shells) and Python3 so I had to do a quick apt-get install zsh. There is a quick-fedora-mirror.conf.dist file included as a template for editing. The suggestion for the completed quick-fedora-mirror.conf file is to move it to /etc
The script requires space somewhere to store a couple of files: timefile and timefile.prev which record the times that the script was last run. I found it easier to just put these into /home/mirror and I copied the quick-fedora-mirror script itself into /usr/local/bin/
One easy mistake to make: this script should be configured to write into the top level directory - so into /srv/mirrors rather than /srv/mirrors/epel or you end up with /srv/mirrors/epel/epel
Apache2 configuration
This was relatively easy - set the document root to /srv/mirrors by uncommenting the relevant lines for /srv in /etc/apache2/apache.conf and amend the /etc/apache2/000-default to show /srv/mirrors as the new DocumentRoot
This results in the six separate directories all being appropriately served under http://localhost as /centos, /debian, /debian-cd, /epel, /ubuntu and /ubuntu-releases.
Mirroring Fedora appears to be unusual if you're not a major (Tier 1) mirror.
The Fedora Wiki page at https://fedoraproject.org/wiki/Infrastructure/Mirroring
is very useful.
Public mirrors are required to have minimum bandwidth and, ideally, to include everything. Fedora is large: fedora-buffet is the whole thing under /pub to download for those who have many TB to spare. fedora-enchilada is everything under /pub/fedora (including the Fedora distribution itself), fedora-epel is EPEL alone.
Public mirrors are required to register with the Fedora Account System [FAS]: those undertaking private mirroring are requested to mirror from Tier1 mirrors or below. The script below relies on mirrors which include fedora-buffet. These are used to achieve rsync speed-ups (by accessing cached bundles of timestamps to speed up differential transfers based on the time files changed).
Since I'm not running a public mirror and not reporting back to the public mirror network stats using mirror-manager, I needed to find a Tier1 mirror that would permit me to mirror privately and my nearest is Hoch Schule Esslingen at ftp-stud.hs-esslingen.de
The suggested mirroring script is quick-fedora-mirror available from Fedora's git repositories at https://pagure.io/quick-fedora-mirror This is licensed under Creative Commons CC0.
The script is written in zsh (with features not included in bash or other shells) and Python3 so I had to do a quick apt-get install zsh. There is a quick-fedora-mirror.conf.dist file included as a template for editing. The suggestion for the completed quick-fedora-mirror.conf file is to move it to /etc
The script requires space somewhere to store a couple of files: timefile and timefile.prev which record the times that the script was last run. I found it easier to just put these into /home/mirror and I copied the quick-fedora-mirror script itself into /usr/local/bin/
One easy mistake to make: this script should be configured to write into the top level directory - so into /srv/mirrors rather than /srv/mirrors/epel or you end up with /srv/mirrors/epel/epel
Apache2 configuration
This was relatively easy - set the document root to /srv/mirrors by uncommenting the relevant lines for /srv in /etc/apache2/apache.conf and amend the /etc/apache2/000-default to show /srv/mirrors as the new DocumentRoot
This results in the six separate directories all being appropriately served under http://localhost as /centos, /debian, /debian-cd, /epel, /ubuntu and /ubuntu-releases.
Rebuilding a mirror - software mirroring of Linux distributions for fun
... and also because sometimes it's just easier to have a local cache
I've a mirror machine sitting next door: it's a private mirror of CentOS, Debian, Debian CDs, EPEL [extra packages for Enterprise Linux from Fedora], Ubuntu releases, and Ubuntu CDs. I've run this intermittently for a few years for myself: it's grown recently because I added CentOS and EPEL.
A 6TB LVM volume is now at about 62% full, so there's plenty of space.
Layout
My machine has these mirrors placed under /srv. The mirrors directory is root owned, the individual directories are owned by a non-root user which also runs the scripts. So you end up with /srv/mirrors/debian or /srv/mirrors/ubuntureleases, for example, where the final directory is owned by a mirror user and the mirror user owns the scripts and the crontab that runs them.
Package used for mirroring
The ftpsync Debian package is pretty much all you need: each individual download source has its own ftpsync.conf script which sits under /etc/ftpsync
The configuration files are essentially identical for Debian and Ubuntu - the only difference is in the source from which you mirror.
Configuring the ftpsync.conf script
The changes needed from the example configuration script in /usr/share/doc/ftpsync are minimal and straightforward: edit The TO directory and the source from which you mirror.
My ftpsync-debian.conf has the following lines changed from the supplied config file, for example.
TO="/srv/mirrors/debian/"
RSYNC_HOST="debian.hands.com"
RSYNC_PATH="debian"
Calling and usage:
Calling is simple: ftpsync sync:archive:debian which references the ftpsync-debian.conf above. Each ftpsync call must have a corresponding configuration file or the ftpsync script complains.
Usage is from mirror's crontab - excerpt below
13 03 * * * /usr/bin/ftpsync sync:archive:debian
37 03 * * * /usr/bin/ftpsync sync:archive:centos
Hope this helps someone: will add a couple of bits on configuring Apache to serve this in another post.
[EDIT - changed rsync host listed above at request of jcristau]
I've a mirror machine sitting next door: it's a private mirror of CentOS, Debian, Debian CDs, EPEL [extra packages for Enterprise Linux from Fedora], Ubuntu releases, and Ubuntu CDs. I've run this intermittently for a few years for myself: it's grown recently because I added CentOS and EPEL.
A 6TB LVM volume is now at about 62% full, so there's plenty of space.
Layout
My machine has these mirrors placed under /srv. The mirrors directory is root owned, the individual directories are owned by a non-root user which also runs the scripts. So you end up with /srv/mirrors/debian or /srv/mirrors/ubuntureleases, for example, where the final directory is owned by a mirror user and the mirror user owns the scripts and the crontab that runs them.
Package used for mirroring
The ftpsync Debian package is pretty much all you need: each individual download source has its own ftpsync.conf script which sits under /etc/ftpsync
The configuration files are essentially identical for Debian and Ubuntu - the only difference is in the source from which you mirror.
Configuring the ftpsync.conf script
The changes needed from the example configuration script in /usr/share/doc/ftpsync are minimal and straightforward: edit The TO directory and the source from which you mirror.
My ftpsync-debian.conf has the following lines changed from the supplied config file, for example.
TO="/srv/mirrors/debian/"
RSYNC_HOST="debian.hands.com"
RSYNC_PATH="debian"
Calling and usage:
Calling is simple: ftpsync sync:archive:debian which references the ftpsync-debian.conf above. Each ftpsync call must have a corresponding configuration file or the ftpsync script complains.
Usage is from mirror's crontab - excerpt below
13 03 * * * /usr/bin/ftpsync sync:archive:debian
37 03 * * * /usr/bin/ftpsync sync:archive:centos
Hope this helps someone: will add a couple of bits on configuring Apache to serve this in another post.
[EDIT - changed rsync host listed above at request of jcristau]
Subscribe to:
Posts (Atom)