Mini-DebCamp ARM Cambridge day 2

 Another really good day at ARM. Still lots of coffee and good food - supplemented by a cooked breakfast if you were early enough :)

Lots of small groups of people working earnestly in the main lecture theatre and a couple of meeting rooms and the soft seating area: various folk arriving  ready for tomorrow. Video team setting up in the afternoon and running up servers and cabling - all ready for a full schedule tomorrow and Sunday.

Many thanks to our sponsors - and especially the helpful staff at ARM who were helping us in and out, sorting out meeting rooms and generally coping with a Debian invasion. More people tomorrow for the weekend.

20231123 - UEFI install on a Raspberry Pi 4 - step by step instructions to a modified d-i

Motivation

 Andy (RattusRattus) and I have been formalising instructions for using Pete Batard's version of Tianocore (and therefore UEFI booting) for the Raspberry Pi 4 together with a Debian arm64 netinst to make a modified Debian installer on a USB stick which "just works" for a Raspberry Pi 4.

Thanks also to Steve McIntyre for initial notes that got this working for us and also to Emmanuele Rocca for putting up some useful instructions for copying.

Recipe

Plug in a USB stick - use dmesg or your favourite method to see how it is identified.

Make a couple of mount points under /mnt - /mnt/data and /mnt/cdrom

1. Grab a USB stick, Partition using MBR. Make a single VFAT
   partition, type 0xEF (i.e. EFI System Partition)

For a USB stick (identified as sdX) below:

$ sudo parted --script /dev/sdX mklabel msdos 

$ sudo parted --script /dev/sdX mkpart primary fat32 0% 100% 

$ sudo mkfs.vfat /dev/sdX1 $ sudo mount /dev/sdX1 /mnt/data/

Download an arm64 netinst.iso

https://cdimage.debian.org/debian-cd/current/arm64/iso-cd/debian-12.2.0-arm64-netinst.iso

2. Copy the complete contents of partition *1* from a Debian arm64
   installer image into the filesystem (partition 1 is the installer
   stuff itself) on the USB stick, in /

$ sudo kpartx -v -a debian-12.2.0-arm64-netinst.iso 

# Mount the first partition on the ISO and copy its contents to the stick 

$ sudo mount /dev/mapper/loop0p1 /mnt/cdrom/ 

$ sudo rsync -av /mnt/cdrom/ /mnt/data/ 

$ sudo umount /mnt/cdrom

3. Copy the complete contents of partition *2* from that Debian arm64
  installer image into that filesystem (partition 2 is the ESP) on
   the USB stick, in /

# Same story with the second partition on the ISO

$ sudo mount /dev/mapper/loop0p2 /mnt/cdrom/

$ sudo rsync -av /mnt/cdrom/ /mnt/data/ $ sudo umount /mnt/cdrom

$ sudo kpartx -d debian-testing-amd64-netinst.iso $ sudo umount /mnt/data

4. Grab the rpi edk2 build from https://github.com/pftf/RPi4/releases
   (I used 1.35) and extract it. I copied the files there into *2*
   places for now on the USB stick:

   /      (so the Pi will boot using it)
   /rpi4  (so we can find the files again later)

5. Add the preseed.cfg file (attached) into *both* of the two initrd
   files on the USB stick

   - /install.a64/initrd.gz and
   - /install.a64/gtk/initrd.gz

   cpio is an awful tool to use :-(. In each case:

$ cp /path/to/initrd.gz .
$ gunzip initrd.gz
$ echo preseed.cfg | cpio -H newc -o -A -F initrd 

$ gzip -9v initrd

$ cp initrd.gz /path/to/initrd.gz

If you look at the preseed file, it will do a few things:

   - Use an early_command to unmount /media (to work around Debian bug
     #1051964)

   - Register a late_command call for /cdrom/finish-rpi (the next
     file - see below) to run at the end of the installation.

   - Force grub installation also to the EFI removable media path,
     needed as the rpi doesn't store EFI boot variables.

   - Stop the installer asking for firmware from removable media (as
     the rpi4 will ask for broadcom bluetooth fw that we can't
     ship. Can be ignored safely.)

6. Copy the finish-rpi script (attached) into / on the USB stick. It
   will be run at the end of the installation, triggered via the
   preseed. It does a couple of things:

   - Copy the edk2 firmware files into the ESP on the system that's
     just been installer

   - Remove shim-signed from the installed systems, as there's a bug
     that causes it to fail on rpi4. I need to dig into this to see
     what the issue is.

That's it! Run the installer as normal, all should Just Work (TM).

BlueTooth didn't quite work : raspberrypi-firmware didn't install until adding a symlink for boot/efi to /boot/firmware

20231127 - This may not be necessary because raspberrypi-firmware path has been fixed

Preseed.cfg

# The preseed file itself causes a problem - the installer medium is
# left mounted on /medis so things break in cdrom-detect. Let's see if
# we can fix that!
d-i preseed/early_command string umount /media || true

# Run our command to do rpi setup before reboot
d-i preseed/late_command string /cdrom/finish-rpi

# Force grub installation to the RM path
grub-efi-arm64  grub2/force_efi_extra_removable boolean true

# Don't prompt for missing firmware from removable media,
# e.g. broadcom bluetooth on the rpi.
d-i hw-detect/load_firmware boolean false

Finish.rpi

 !/bin/sh

set -x

grep -q -a RPI4 /sys/firmware/acpi/tables/CSRT
if [ $? -ne 0 ]; then
    echo "Not running on a Pi 4, exit!"
    exit 0
fi

# Copy the rpi4 firmware binaries onto the installed system.
# Assumes the installer media is mounted on /cdrom.
cp -vr /cdrom/rpi4/. /target/boot/efi/.

# shim-signed doesn't seem happy on rpi4, so remove it
mount --bind /sys /target/sys
mount --bind /proc /target/proc
mount --bind /dev /target/dev

in-target apt-get remove --purge --autoremove -y shim-signed

Arm Cambridge - mini-Debcamp 23 November 2023

 At Arm for two days before the mini-Debconf this weekend.

First time at Arm for a few years: huge new buildings, shiny lecture theatre.

Arm have made us very welcome. A superb buffet lunch and unlimited coffee plus soft drinks - I think they know what Debian folk are like.
Not enough power blocks laid out at the beginning - only one per table - but we soon fixed that 😀

The room is full of Debian folk: some I know, some new faces. Reminiscing about meeting some of them from 25 years ago - and the chance to thank people for help over a long time.

Andy (RattusRattus) and I have been working out the bugs on an install script using UEFI for a Raspberry Pi 4. More on that in the next post, maybe.

As ever, it's the sort of place where "I can't get into the wiki" is sorted by walking three metres across the room or where an "I can't find where to get X for Raspberry Pi" can be solved by asking the person who builds Raspbian. "Did you try and sign up to the Debian wiki last week - you didn't follow the instructions to mail wiki@ - I _know_ you didn't because I didn't see the mail ... "

My kind of place and my kind of people, as ever.

Thanks again to Arm who are one of our primary sponsors for this mini-Debconf.

Point release weekend for Debian: two releases this weekend: 202311071653

Over in Cambridge with RattusRattus, Sledge, egw and Isy. Andy is very kindly putting us up.

We're almost all of the way through testing 12.2 and some of the way through testing 11.8.

It's a LONG day - heads down into laptops and relatively quiet - I think we're all tired and we've a way to go yet.

Building a mirror of various Red Hat oriented "stuff"

Building a mirror for rpm-based distributions.

I've already described in brief how I built a mirror that currently mirrors Debian and Ubuntu on a daily basis. That was relatively straightforward given that I know how to install Debian and configure a basic system without a GUI and the ftpsync scripts are well maintained, I can pull some archives and get one pushed to me such that I've always got up to date copies of Debian and Ubuntu.

I wanted to do something similar using Rocky Linux to pull in archives for Almalinux, Rocky Linux, CentOS, CentOS Stream and (optionally) Fedora.

(This was originally set up using Red Hat Enterprise Linux on a developer's subscription and rebuilt using Rocky Linux so that the machine could be passed on to someone else if necessary. Red Hat 9.1 has moved to x86_64v2 - on the machine I have (HP Microserver gen 8) 9.1 it fails immediately. It has been rebuilt to use Rocky 8.8).

This is a minimal install of Rocky as console only - the machine it's on only has 4G of memory so won't run a GUI reliably. It will run Cockpit so can  be remotely administered. One user to run everything - mirror.

Minimal install of Rocky 8.7 from DVD .iso. SELinux is enabled, SSH works for remote access. SELinux had to be tweaked to allow /srv/ the appropriate permissions to be served by nginx. /srv is a large LVM volume rather than a RAID 6 - I didn't have enough disks

Adding nginx, enabling Cockpit and editing the Rocky Linux mirroring scripts resulted in something straightforward to reproduce.

nginx

I cheated and stole large parts of my Debian config. The crucial part to remember is that there is no autoindexing by default and I had to dig to find the correct configuration snippet.

  # Load configuration files for the default server block.

        include /etc/nginx/default.d/*.conf;

        location / {
                autoindex on;
                autoindex_exact_size off;
                autoindex_format html;
                autoindex_localtime off;
                # First attempt to serve request as file, then
                # as directory, then fall back to displaying a 404.
                try_files $uri $uri/ =404;
        }

 Rocky Linux mirroring scripts

Systemd unit file for service

[Unit]
Description=Rocky Linux Mirroring script

[Service]
Type=simple
User=mirror
Group=mirror
ExecStart=/usr/local/bin/rockylinux

[Install]
WantedBy=multi-user.target

 Rocky linux  timer file

[Unit]
Description=Run Rocky Linux mirroring script daily

[Timer]
OnCalendar=*-*-* 08:13:00
OnCalendar=*-*-* 22:13:00
Persistent=true

[Install]
WantedBy=timers.target

Mirror script

#!/bin/env bash
#
# mirrorsync - Synchronize a Rocky Linux mirror
# By: Dennis Koerner <koerner@netzwerge.de>
#
# The latest version of this script can be found at:
# https://github.com/rocky-linux/rocky-tools
#
# Please read https://docs.rockylinux.org/en/rocky/8/guides/add_mirror_manager
# for further information on setting up a Rocky mirror.
#
# Copyright (c) 2021 Rocky Enterprise Software Foundation

This is a very long script in total.

Crucial parts I changed only listed the mirror to pull from and the place to put it.

# A complete list of mirrors can be found at
# https://mirrors.rockylinux.org/mirrormanager/mirrors/Rocky
src="mirrors.vinters.com::rocky"

# Your local path. Change to whatever fits your system.
# $mirrormodule is also used in syslog output.
mirrormodule="rocky-linux"
dst="/srv/${mirrormodule}"

filelistfile="fullfiletimelist-rocky"
lockfile="/home/mirror/rocky.lockfile"
logfile="/home/mirror/rocky.log"

 Logfile looks something like this: the single time spec file is used to check whether another rsync needs to be run

deleting 9.1/plus/x86_64/os/repodata/3585b8b5-90e0-4856-9df2-95f646bc62c7-PRIMARY.xml.gz

sent 606,565 bytes  received 38,808,194,155 bytes  44,839,746.64 bytes/sec
total size is 1,072,593,052,385  speedup is 27.64
End: Fri 27 Jan 2023 08:27:49 GMT
fullfiletimelist-rocky unchanged. Not updating at Fri 27 Jan 2023 22:13:16 GMT
fullfiletimelist-rocky unchanged. Not updating at Sat 28 Jan 2023 08:13:16 GMT

It was essentially easier to store fullfiletimelist-rocky in /home/mirror than anywhere else.

Very similar small modifications to the Rocky mirroring scripts  were used to mirror the other distributions I'm mirroring. (Almalinux, CentOS, CentOS Stream, EPEL and Rocky Linux).

 

 

 

20230828 - OMGWTFBBQ - Breakfast is happening more or less

 And nothing changes: rediscovered from past Andrew at his first Cambridge BBQ and almost the first blog post here:

"House full of people I knew only from email, some very old friends. Wires and leads filling the front room floor - laptops _everywhere_ .

...

Thirty second rule on sofa space - if you left for more than about 30 seconds you had to sit on the floor when you got back (I jammed myself onto a corner of the sofa once I realised I'd barely get through the crush :) )
[Forget students in a mini / UK telephone box - how many DDs can you fit into a very narrow kitchen :) ]

It's a huge, dysfunctional family with its own rules, geeky humour and in-jokes but it's MINE - it's the people I want to hang out with and, as perverse as it sounds, just being there gave me a whole new reaffirmed sense of identity and a large amount of determination to carry on "wasting my time with Linux" and Debian"

The *frightening* thing - this is from August 31st 2009 ... where have the years gone in between.

20230826 - OMGWTFBBQ - BBQ still in full swing

 There's been a very successful barbeque running in the garden: burgers, sausages, beer, vegetarian dishes and then ice cream.

The chance to catch up with people you only meet in IRC. Talking and laughter - and probably a couple of games of Mao.

Thanks also to our sponsors - Collabora, Codethink and RattusRattus for contributions to food and drink.