And the last few architectures are getting done - Debian 10.7 edging ever closer

 And we're through the live images as far as we can reasonably go. There remain a couple of MIPS architectures to build and IBM s390x - we have no hardware to test these.

It's now 23:30 UTC - so that's 12 1/2 hours since we started and Sledge has another couple of hours to wait before the last images can be signed and the image release can be pushed.

Thanks so much to Sledge, RattusRattus, Isy, Schweer and jlsantos - who came back to us after a significant internet outage local to them in California - to those who popped by - hi Kibi and Manty - and to all who labour behind the scenes to make each release "just work"

Here's to the next one - in another couple of months, I'm guessing.

Lots further forwards - but moving slowly through live CD testing

All of the testing of the normal Debian images has gone through the steps we lay out. We're now working on the Debian live CDs which take significantly longer per image.

Looks like a long night until Steve is ready to sign the release and begin the final transfer - however much we try to minimise it, it's a 12-15 hour process and my thanks go to RattusRattus and Steve for the final stages of a very long day.

Part way through Debian CD release process - always fun to do - Debian 10.7 in process

 As is normal for these days: I post a quick update on how we're doing.

Working through the test suite: RattusRattus, Sledge, Isy, Schweer and I. We've been joined by somebody new - jlsantos - who has done his first test for us :)

A couple of changes: there's now a different automatic partitioning layout. /boot has been resized to ~500MB - to allow for more than one kernel but also the other initramfs files. The filesystem size allocated for a swap partition is now 1GB by default.

We're doing fairly well - one hardware failure on an old laptop replaced by a newer model from Sledge's stock - everybody cheerful and all working well.

Separately, there was a quiet release of Bullseye Alpha3 last night - that has had minimal testing - as ever, we're not yet near the release of Bullseye yet.

Preparing for release of Debian 10.7 over the weekend and CentOS / Scientific Linux 6.x and EPEL for 6 now EOL

For those keeping score:

This weekend - 5th December 2020 - should see us release Debian 10.7 - an update to Debian stable (Buster) so I should be spending a day or so in the company of my friends and colleagues.

Red Hat 6.10 is now out of support unless you pay Extended Update subscriptions for individual Red Hat machines. This means that CentOS 6.* has now been removed from CentOS mirrors since these were dependent on Red Hat 6 sources.. Similarly, Scientific Linux have also removed their fork of 6.*. They are continuing to support a Scientific Linux 7 but suggest a move to CentOS 8 thereafter.

Separately, Fedora Linux have removed EPEL 6.* - definitely time to update to Red Hat/CentOS/Scientific Linux 7.* or greater for those affected.

 All the very best to all:

Final post from media team for the day - most of the ordinary images and live images have been tested

 Winding down slightly - we've worked our way through most of the images and testing. Schweer tested all of the Debian Edu/Skolelinux images for which many thanks

Sledge, RattusRattus, Isy and I have been working pretty much solidly for 10 3/4 hours. There's still some images to build - mips, mipsel and s390x but these are all images that we don't have hardware to test on particularly.

Another good and useful day - bits and pieces done throughout. NOTE: There appear to have been some security updates since the main release this morning so, as ever, it's worth updating machines on a regular basis.

Waiting for the final images to finish building so that we can check the archive for completeness and then publish to the media mirrors. All the best until next time: thanks as ever to Sledge for his invaluable help. See you again in a couple of months in all likelihood. 

A much smaller release: some time in the next month we hope to be able to build and test an Alpha release for Bullseye. Bullseye is likely to be released somewhere round the middle of next year so we'll have additional Buster stable point releases in the meantime.

Chunking through the tests for various media images ...

We're working our way through some of the CD/DVD/Blu-Ray media images, doing test installs, noting failures and so on. It's repetitive work but vital if we're going to provide some assurance that folk can install from the images we make. 

There's always the few things that catch us out and there's always something to note for next time. Schweer has joined us and is busy chasing down debian-edu/Skolelinux installs from Germany. We're getting there, one way and another, and significantly ahead of where we were last time around when the gremlins got in and delayed us. All good :)

There are things that money can't buy - and sensible Debian colleagues are worth gold and diamonds :)

 Participating in the Debian media testing on debian-cd. One of my colleagues has just spent time to sort out an email issue having spent a couple of hours with me the other night. I now have good, working email for the first time in years - I can't value that highly enough.

Sledge, RattusRattus, Isy and myself are all engaged in testing various CD images. At the same time, they're debugging a new application to save us from wiki problems when we do this - and we're also able to use a video link which is really handy to chat backwards and forwards and means I can sit virtually in Cambridge :)

Lots of backchat and messages flying backwards and forwards - couldn't wish for a better way to spend an afternoon with friends.

There's a Debian point release for Debian stable happening this weekend - 10.6

 Nothing particularly new or unexpected: there's a point release happening at some point this weekend for Debian stable. Usual rules apply: if you've already got a system current and up to date, there's not much to do but the base files version will change at some point to reflect 10.6 when you next update. 

If you have media from 10.5, you may not _have_ to go and get media this weekend but it's always useful to get new media in due course. There's an updated kernel and an ABI bump. You _will_ need to reboot at some time to use the new kernel image.

This point release will contain security fixes, consequent changes etc. as usual - it is always good and useful to keep machines up to date.

Working with the CD team to eventually test, build and release CD / DVD images and media as and when files gradually become available. As ever, this may take 12-16 hours. As ever, I'll post some blog entries as we go.

Currently "sitting in Cambridge" via video link with Sledge, RattusRattus and Isy who are all involved in the testing and we'll have a great day, as ever.

Just coming to the end of Debconf 20 2020 - and a preview.

 One more talk from Jon "maddog" Hall and then the closing wrap up. This has been a blast: I've enjoyed it a lot and it's made me more enthusiastic than I have been for a long time.

 So once more with thanks to the video team
It's almost the end of this year's DebConf dream
As we all break up, there's a favour to ask
Stay safe - wear a smile - and where needed, a mask

We'll not take you to task ; it's a favour we ask
Stay safe - wear a smile - and where needed, a mask

Haifa, Pristina or Kochi - we'll see how that lands
There's not much left to tidy up - wash your own hands
So now if you'll join us in virtual beer
We'll bring this to a close - and we'll meet up next year

So now if you'll join us - let us all raise a cheer
To Debconf21 - and we'll see you next year

The Debconf20 song

The DebConf 20 song - a sea shanty - to the tune of "Fathom the bowl"

Here's to DebConf 20, the brightest and best
Now it's this year's orga team getting no rest
We're not met in Haifa - it's all doom and gloom
And I'm sat like a lifer here trapped in my room

I'm sat in my room, it's all doom and gloom
And I'm sat at my keyboard here trapped in my room

Now there's IRC rooms and there's jitsi and all
But no fun conversations as we meet in the hall
No hugs for old friends, no shared wine and cheese
Just shared indigestion as we take our ease

I'm sat in my room, it's all doom and gloom
And I'm sat with three screens around me in my room

But there's people to chat to, and faces we know
And new things to learn and we're all on the go
Algo en espanol - there's no cause for alarm
An Indic track showcasing Malayalam

I'm sat in my room, it's all doom and gloom
And I'm sat with my Thinkpads and cats in my room

With webcams and buffering, with lag and delay
It's as well that there's Debconf time all through the day
The effects of tiredness are hard to foresee
For the Debian clocks all are timezone UTC

I'm sat in my room, it's all doom and gloom
And I'll sing out of tune as I'm sat in my room

There's no social drinking, there's no games of Mao
Keeping social distance, we can't think quite how
This year is still friendly though minus some fun
We'll catch up next year when we'll all get some sun

I'm sat in my room, it's all doom and gloom
I'm sat with my friends around here in my room

There's loopy@debconf and snippets and such
To cheer us all up, sure, it doesn't take much
For we're all one big family, though we each code alone
And we sometimes switch off or just complain and moan

I'm sat in my room, it's all doom and gloom
And there's space for us all in the debconf chat room

This is my first DebConf - hope it won't be my last
And we'll meet up somewhere when this COVID is past
To all who have done this - we deserve the credit
Now if you'll excuse me - I've web pages to edit

I'm sat in my room, it's not all doom and gloom
And we're met as one Debian here in my room


 

Debian 10.5 media testing - 202001082250 - last few debian-live images being tested for amd64 - Calamares issue - Post 5 of several.

Last few debian-live images being tested for amd64. We have found a bug with the debian-live Gnome flavour. This specifically affects installs after booting from the live media and then installing to the machine using  the Calamares installer found on the desktop. The bug was introduced as a fix for one issue that has produced further buggy behaviour as a result.

Fixes are known - we've had highvoltage come and debug them with us - but will not be put out with this release but will wait for the 10.6 release which will allow for a longer time for debugging overall.

You can still run from the live-media, you can still install with the standard Debian installers found in the menu of the live-media disk - this is _only_ a limited time issue with the Calamares installer. At this point in the release cycle, it's been judged better to release the images as they are - with known and documented issues - than to try and debug them in a hurry and risk damaging or delaying a stable point release.

Debian 10.5 media testing - 202008012055 - post 4 of several

We've more or less finished testing on the Debian install images. Now moving on to the debian-live images. Bugs found and being triaged live as I type. Lots of typing and noises in the background of the video conference. Now at about 12-14 hours in on this for some of the participants. Lots of good work still going on, as ever.

Debian 10.5 media testing - pause for supper - 202001081715 - post 3 of several

Various of the folk doing this have taken a food break until 1900 local. A few glitches, a few that needed to be tried over again - but it's all going fairly well.

It is likely that at least one of the CD images will be dropped. The XFCE desktop install CD for i386 is now too large to fit on CD media. The netinst .iso files / the DVD 1 file / any of the larger files available via Jigdo will all help you achieve the same result.

There are relatively few machines that are i386 architecture only - it might be appropriate for people to use 64 bit amd64 from this point onwards as pure i386 machines are now approaching ten years old as a minimum. If you do need a graphical user environment for a pure i386 machine, it can be installed by using an expert install or using tasksel in the installation process.

Debian 10.5 media testing - continuing quite happily - 202001081320 - post 2 of several

We've now settled into a reasonable rhythm: RattusRattus and Isy and Sledge all working away hard in Cambridge: Schweer in Germany and me here in Cheltenham.

Lots of chat backwards and forwards and a good deal of work being done, as ever.

It's really good to be back in the swing of it and we owe thanks to folk for setting up infrastructure for us to use for video chat, which makes a huge difference: even though I know what they're like, it's still good to see my colleagues.

Debian 10.5 media testing process started 202008011145 - post 1 of several.

The media testing process has started slightly late. There will be a _long_ testing process over much of the day: the final media image releases are likely to be at about 0200-0300UTC tomorrow.

Just settling in for a long day of testing: as ever, it's good to be chatting with my Debian colleagues in Cambridge and with Schweer in Germany. It's going to be a hot one - 30 Celsius (at least) and high humidity for all of us.

EDIT: Corrected for UTC :)

Debian 10.5 Buster point release 20200801 - all of the fixes :)

The point release is happening today for Debian Buster 10.5. This is an important release because it incorporates all the recent security fixes from the latest GRUB / Secure Boot "Boothole" security problems.

Behind the scenes, there has been a lot of work to get this right: a release subject to an embargo to allow all the Linux releases to co-ordinate this as far as possible, lots of consistent effort, lots of cooperation - the very best of Free/Libre/Open Source working together.

Secure Boot shims are signed with a different key to go to upstream this time around: in due course, when revocation of old, insecure code happens to plug the security hole, older media may be deny-listed. All the updates for all the affected packages (listed in https://www.debian.org/security/2020-GRUB-UEFI-SecureBoot/ ) are included in this release.

This has been a major wake-up call: the work behind the scenes has meant that each affected Linux distribution will be in a much better position going forward and working together is always good.

How to use the signed checksum files to verify Debian media images

Following on from the blog post the other day in some sense: someone has asked on the debian-user list: "I do not understand from the given page (https://www.debian.org/CD/verify)  how to use .sign files and gpg in order to check verify the authenticity of debian cds. I understand the part with using sha256sum or sha512sum or md5sum to check whether the files were downloaded correctly."

Distributed with the CD and other media images on Debian CD mirrors, there are files like MD5SUM, MD5SUM.sign, SHA256SUM, SHA256SUM.sign and so on.

SHA512SUM is a plain text list of the SHA512SUMs for each of the files in the directory. SHA512SUM.sign is the GPG-signed version of that file. This allws for non-repudiation - if the signature is valid, then the plain text file has been signed by the owner of that key. Nothing has tampered with the checksums file since it was signed.

After downloading the SHA1SUM, SHA256SUM and SHA512SUM files and the corresponding .sign files from, say, the prime Debian CD mirror at

http://cdimage.debian.org

Assuming that you already have GPG installed: sha256sum and sha512sum are installed by the coreutils package, which Debian installs by default.

gpg --verify SHA512SUMS.sign SHA512SUMS will verify the .sign signature file against the signed file.

gpg --verify SHA512SUMS.sign SHA512SUMS
gpg: Signature made Sun 10 May 2020 00:16:52 UTC
gpg:                using RSA key DF9B9C49EAA9298432589D76DA87E80D6294BE9B

The signature is as given on the Debian CD verification page given above.

You can import that key from the Debian key servers if you wish.

gpg --keyserver keyring.debian.org --recv-keys DF9B9C49EAA9298432589D76DA87E80D6294BE9B

You can import the signature for checking from the SKS keyservers which are often more available:

gpg --keyserver pool.sks-keyservers.net --recv-keys DF9B9C49EAA9298432589D76DA87E80D6294BE9B 

and you then get:

gpg --verify SHA512SUMS.sign SHA512SUMS
gpg: Signature made Sun 10 May 2020 00:16:52 UTC
gpg:                using RSA key DF9B9C49EAA9298432589D76DA87E80D6294BE9B
gpg: Good signature from "Debian CD signing key " [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: DF9B 9C49 EAA9 2984 3258  9D76 DA87 E80D 6294 BE9B

My own key isn't in the Debian CD signing key ring - but this does now show me that this is a good signature from the primary key fingerprint as given.

Repeating the exercise from the other day and producing a Debian amd64 netinst file using jigdo, I can now check the checksum on the local .iso file against the checksum file distributed by Debian. If they match, it's a good sign that the CD I've generated is bit for bit identical. For my locally generated file:

sha512sum debian-10.4.0-amd64-netinst.iso
ec69e4bfceca56222e6e81766bf235596171afe19d47c20120783c1644f72dc605d341714751341051518b0b322d6c84e9de997815e0c74f525c66f9d9eb4295  debian-10.4.0-amd64-netinst.iso

and for the file checksum as distributed by Debian:

less SHA512SUMS | grep *iso
ec69e4bfceca56222e6e81766bf235596171afe19d47c20120783c1644f72dc605d341714751341051518b0b322d6c84e9de997815e0c74f525c66f9d9eb4295  debian-10.4.0-amd64-netinst.iso

and they match! 

As ever, I hope this blog post will help somebody.

[Edit: Someone has kindly pointed out that grep *iso SHA512SUMS | sha512sum -c will check this more efficiently.]

 


 

How to use jigdo to download media images

I worked with the CD release team over the weekend for the final release of Debian Stretch. One problem: we have media images which we cannot test because the team does not have the hardware. I asked questions on the debian-cd mailing list about the future of these and various other .iso images.

Could we replace some DVDs and larger files with smaller jigdo files so that people can download files to build the DVD locally?

People asked me:

• How do you actually use jigdo to produce a usable media image? 
• What advantages does jigdo bring over just downloading a large .iso image?

Why jigdo?

• Downloading large files on a slow or lossy link is difficult.
• Downloading large (several GB) files via http can be unreliable.
• Jigdo can be quicker than trying to download one large file and failing.
• There are few CD mirrors worldwide: jigdo can use a local Debian mirror.
• The transport mechanism is http - no need for a particular port to be opened.

Using jigdo

Jigdo uses information from a template file to reconstruct an .iso file by downloading Debian packages from a mirror. The image is checksummed and verified at the end of the complete download. if the download is interrupted, you can import the previously downloaded part of the file.

It's a command line application - the GUI never really happened - but it is fairly easy to use.  apt install jigdo-file then find the .jigdo file and .template files that you need for the image from a CD mirror: https://cdimage.debian.org/debian-cd/current/amd64/jigdo-cd/

To build the netinst CD for AMD64, for example: you need the .jigdo file as a minimum: debian-10.4.0-amd64-netinst.jigdo

If you only have this file, jigdo-lite will download the template later but you can save the template in the same directory and save time. The jigdo file is only 25k or so and the template is 4.6M rather than 336M. I copied them into my home directory to build there. The process does not need root permissions.

Run the command jigdo-lite This prompts you for a .jigdo file to use. By default, this uses http to fetch the file from a distant webserver.

(If the files are local, you can use the file:/// syntax.For example: file:///home/amacater/debian-10.4.0-amd64-netinst.jigdo)

jigdo-lite then reads the .jigdo file and outputs some information about the .iso

It offers the chance to reload any failed download, then prompts for a mirror name. The download pulls in small numbers of files at a time, saves to a temporary directory and will checksum the eventual .iso file.

This will work for any larger file including the 16GB .iso distributed only as a .jigdo

For i386 and AMD, the images are bootable when copied to a USB stick. Use dd to write them and verify the copy.

• Plug in a USB that can be overwritten.
  
• Use dmesg as root to work out which device this is.
  
• Change to the directory in which you have your .iso image.
  
• Write the image to the stick in 4M blocks and display progress with the syntax of the command below (all one line if wrapped).

dd if=debian-10.4.0-amd64-netinst.iso of=/dev/sdX obs=4M oflag=sync status=progress

Debian Stretch release 9.13 - testing of images now complete 202007181950 or so

And we're getting a lot closer. Last edits to the wiki for the testing page. Tests marked as passed. There are a bunch of architectures where the media images are still to be built and the source media will still need to be written as well.

It's been a really good (and fairly long day). I'm very glad indeed that I'm not staying up until the bitter end. It's always fun - I'm also very glad that, as it panned out, we didn't end up also doing the next Buster point release this weekend - it would have been a massive workload.

So Stretch passes to LTS, Jessie has passed to ELTS - and we go onwards and upwards to the next point release - and then, in due course, to Bullseye when it's ready (probably the first quarter of next year?).

As ever, thanks due to all involved - the folks behind the scenes who maintain cdimage.debian.org, to Sledge, RattusRattus, Isy and schweer. Done for about a fortnight or so until the next Buster release.

I will try to blog on other things, honest :)

Debian Stretch 9.13 release - blog post 2 - testing of basic .iso images ongoing as at 202007181655

The last of the tests for the basic .iso images are finishing. Live image testing is starting. Lots from RattusRattus, Isy, Sledge and schweer besides myself. New this time round is an experiment in videoconference linking which has made this whole experience much more human in lockdown - and means I'm missing Cambridge.

Two questions that have come up as we've been going through. There are images made for Intel-based mac that were first made for Jessie or so: there are also images for S390X. These can't be tested because none of the testers have the hardware. Nobody has yet recorded issues with them but it could be that nobody has ever used them recently or has reported problems

If nobody is bothered with these: they are prime candidates for removal prior to Bullseye.

Debian "Stretch" 9.13 release preparations ongoing

Just checking in. Debian "Jessie" == oldoldstable == Debian 8 was the previous Debian Long Term Support release. Debian LTS seeks to provide support for Debian releases for five years. LTS support for Jessie ended on 30th June 2020.

A limited subset of Jessie will now move to ELTS - Extended Long Term Support and another two years projected support.

Neither LTS nor ELTS are supported  any longer by the main Debian folks: instead, they are supported on a commercial basis by a group of Debian volunteers and companies, coordinated by a company led by Raphael Hertzog.

Debian releases are fully supported by the Debian project for two years after the release of the next version.  Today is the final release of Stretch by Debian to incorporate security fixes and so on up to the handover to LTS.

if you are currently running Stretch: you do not need the new CD images. Apt / apt-get update will supply you the updates up until today. Hereafter, Stretch will be supported only as LTS - see LTS

And a nice quick plug - Apple WWDC keynote

Demonstrating virtualisation on the new Apple to be powered by Apple silicon and ARM. Parallels virtualisation: Linux gets mentioned - two clear shots, one of Debian 9 and one of Debian 10. That's good publicity, any which way you cut it, with a worldwide tech audience hanging on every word from the presenters.

Now - developer hardware is due out this week - Apple A12 chip inside Mac mini format. Can we run Debian natively there?

CD / DVD testing for Buster release 4 - 202005092130 - Slowing down a bit - but still going.

Last few architectures are being built in the background. Schweer has just confirmed successful testing of all the Debian Edu images - thanks to him, as ever, and to all involved. We're slowing up a bit - it's been a long, hot day and it's not quite over yet. The images release looks to be well on course. As ever, the point release incorporates security fixes and some packages have been removed. The release announcement at https://www.debian.org/News/2020/20200509 gives the details.

CD image testing for Buster release 4 - 202005091950 - Most install images checking out well

Lots of hard work going on. schweer has just validated all of the Debian Edu images.  Most of the normal install images have gone through tests with only a few minor hitches. Now moving on to the Live images. These take longer to download and test but we're working through them gradually.

As ever: a point release doesn't mean that the Debian you have is now obsolete - an apt-get / aptitude update will bring you up to the latest release very quickly. If you are updating regularly, you will have most of these files anyway. One small thing: the tools may report that the release version has changed. This is quite normal - base files have changed to reflect the new point release and this causes the notification. The notification is a small warning so that you are not taken by complete surprise but it is quite normal in the circumstances of a Debian point release.

Thanks to the other folk doing the hard work: 10+ hours and continuing.

CD image testing for Buster release 4 - 202005091538 - Coming along nicely

We're most of the way through testing the various installs, with good success. Waiting on a few things to be built. It's a good way to pass an afternoon / evening.

CD / DVD image testing happening now - 202005091330

The usual suspects very much involved - Isy, myself, RattusRattus, schweer and Sledge. Image testing going on to good effect. Update pulses are starting to hit mirrors: just done a dist-upgrade on one of the machines here. All good

And again - CD release for Buster release 4 is going to happen over the weekend ...

And for our regular readers - it's happening again. I'm sitting here with two laptops, a desktop, a connection to IRC - and friends in Cambridge and elsewhere involved in this too.

All good fun, as ever :)
 

CD release time post 3 20202029 1653 - Release for Oldstable / Squeeze / 9.12 has happened

Just tidying up on the last of my tests. The release to the mirrors for 9.12 has happened.

Much sterling work by  amacater, Isy, RattusRattus, Sledge and Schweer (in alphabetical order). This one seemed to have turned up fewer bugs.

Lots of fun, the usual bouts of problems between chair and keyboard and a feeling for me of something worthwhile to share in.

My private CD mirror is now up to date with 10.3 and a few machines here have already been upgraded :) On a day when the weather is cold, wet, windy and rainy, it's been a good excuse to sit in the warm with a laptop.

Here's to the next one, then.

CD release time post 2 20200208 2152 - lots of install CD/DVD/BD image testing going on

It's been a busy day. Lots of installs, a couple of bugs found, mostly going very well. The main 10.3 Debian update has been pushed to the mirror network and I've updated two of my machines so far. All good.

Hofstadter's Law states that "It always takes longer than you think, even taking account of Hofstadter's Law" ( the Douglas Hofstadter who wrote Goedel Escher Bach) and these long days always disappear into the memory until the next time.

There's also an update planned for the images for oldstable - so we'll be busy for a fair while yet.

UPDATE: The release looks good: some CDs for the smaller arches remain to be built but it's done for now.

Oldstable testing will happen tomorrow

As has become traditional - another blog post round about CD release time

Just waiting to start: Isy, Sledge, RattusRattus and Schweer are all standing by on IRC

Debian 10.3 update is out - the CDs will begin being built and tested this weekend. Would much rather be in Cambridge, as ever :)

Also dotting between various machines doing general updates and stuff: this is being written from my newer laptop and is the first update from this one.
As ever, it should be fun.

Rebuilding a mirror - Fedora EPEL mirroring and Apache config

This follows on from the previous post: Fedora does things differently. EPEL is a sub-project: software packaged for CentOS/Red Hat Enterprise Linux by the Fedora Project maintainers.

Mirroring Fedora appears to be unusual if you're not a major (Tier 1) mirror.

The Fedora Wiki page at https://fedoraproject.org/wiki/Infrastructure/Mirroring
is very useful.

Public mirrors are required to have minimum bandwidth and, ideally, to include everything. Fedora is large: fedora-buffet is the whole thing under /pub to download for those who have many TB to spare. fedora-enchilada is everything under /pub/fedora (including the Fedora distribution itself), fedora-epel is EPEL alone.

Public mirrors are required to register with the Fedora Account System [FAS]: those undertaking private mirroring are requested to mirror from Tier1 mirrors or below. The script below relies on mirrors which include fedora-buffet. These are used to achieve rsync speed-ups (by accessing cached bundles of timestamps to speed up differential transfers based on the time files changed).

Since I'm not running a public mirror and not reporting back to the public mirror network stats using mirror-manager, I needed to find a Tier1 mirror that would permit me to mirror privately and my nearest is Hoch Schule Esslingen at ftp-stud.hs-esslingen.de

The suggested mirroring script is quick-fedora-mirror available from Fedora's git repositories at https://pagure.io/quick-fedora-mirror This is licensed under Creative Commons CC0.

The script is written in zsh (with features not included in bash or other shells) and Python3 so I had to do a quick apt-get install zsh. There is a quick-fedora-mirror.conf.dist file included as a template for editing. The suggestion for the completed quick-fedora-mirror.conf file is to move it to /etc

The script requires space somewhere to store a couple of files: timefile and timefile.prev which record the times that the script was last run. I found it easier to just put these into /home/mirror and I copied the quick-fedora-mirror script itself into /usr/local/bin/

One easy mistake to make: this script should be configured to write into the top level directory - so into /srv/mirrors rather than /srv/mirrors/epel or you end up with /srv/mirrors/epel/epel

Apache2 configuration

This was relatively easy - set the document root to /srv/mirrors by uncommenting the relevant lines for /srv in /etc/apache2/apache.conf and amend the /etc/apache2/000-default to show /srv/mirrors as the new DocumentRoot

This results in the six separate directories all being appropriately served under http://localhost as /centos, /debian, /debian-cd, /epel, /ubuntu and /ubuntu-releases.

Rebuilding a mirror - software mirroring of Linux distributions for fun

... and also because sometimes it's just easier to have a local cache

I've a mirror machine sitting next door: it's a private mirror of CentOS, Debian, Debian CDs, EPEL [extra packages for Enterprise Linux from Fedora], Ubuntu releases, and Ubuntu CDs. I've run this intermittently for a few years for myself: it's grown recently because I added CentOS and EPEL.

A 6TB LVM volume is now at about 62% full, so there's plenty of space.

Layout

My machine has these mirrors placed under /srv. The mirrors directory is root owned, the individual directories are owned by a non-root user which also runs the scripts. So you end up with /srv/mirrors/debian or /srv/mirrors/ubuntureleases, for example, where the final directory is owned by a mirror user and the mirror user owns the scripts and the crontab that runs them.

Package used for mirroring

The ftpsync Debian package is pretty much all you need: each individual download source has its own ftpsync.conf script which sits under /etc/ftpsync

The configuration files are essentially identical for Debian and Ubuntu - the only difference is in the source from which you mirror.

Configuring the ftpsync.conf script

The changes needed from the example configuration script in /usr/share/doc/ftpsync are minimal and straightforward: edit The TO directory and the source from which you mirror.

My ftpsync-debian.conf has the following lines changed from the supplied config file, for example.

TO="/srv/mirrors/debian/"

RSYNC_HOST="debian.hands.com"
RSYNC_PATH="debian"

Calling and usage:

Calling is simple: ftpsync sync:archive:debian which references the ftpsync-debian.conf above. Each ftpsync call must have a corresponding configuration file or the ftpsync script complains.

Usage is from mirror's crontab - excerpt below

13 03 * * * /usr/bin/ftpsync sync:archive:debian
37 03 * * * /usr/bin/ftpsync sync:archive:centos

Hope this helps someone: will add a couple of bits on configuring Apache to serve this in another post.

[EDIT - changed rsync host listed above at request of jcristau]