So... For some time now at our office we've been using Pi3's, 4's, and 02W's for various appliance type roles, including a whole ton of digital signage. Rather than install a base Raspbian image and go through a long and involved process of altering the resulting installation for each host, I took some steps to save us loads of time. Specifically, I built a system that will blow away the contents of a microSD card, create three partitions (the third of which will be used if we have to just completely replace the operating system remotely), mounts them appropriately, and then simply untars the contents of a baseline /boot/firmware filesystem and the contents of the / filesystem. Not exactly rocket science.
Some years went by and now we have a few hundred of these things rolling around on our LAN and it's going to be easier to just refactor everything and swap out microSD cards than try a dist-upgrade on each and every one of the things out there. So, I started attempting to rebuild everything (considering everything custom that I'm dropping into these things is either a shell script or a perl script along with a few unit files, it ain't like a lot of refactoring has to be done there) beginning with a baseline image. This is where I ran into the almost inscrutable problem... I downloaded the most recent Bookworm build, flashed it to a bog-standard SanDisk 32Gb microSD card, then booted it, and ran through the basic configuration steps: setting the locale, country code, timezone (ahem!), the maintenance role account with ssh key, did a quick round of apt update && apt --assume-yes upgrade, you know... fairly basic stuff, then rebooted a few times to make sure everything was behaving as expected. Powered down, and then siphoned p1 and p2 into tarballs and dropped them into a new profile directory with all the other baseline stuff (scripts that blow away /etc/machine-id, the ssh host keys, and a unit file that generates new ones) and... when I stick it back into a Pi it fails to boot. Notably it's showing a bit of scroll, then blanking the screen and... nothing. The numlock LED even stops working. Kind of predictably a kernel panic of some sort.
Mind you, the odds that there is something wrong with the somewhat heavyweight shells script that generates these new images is just hilariously low (I've been doing Unix/Linux engineering for a couple of decades now). I expected low-knowledge people to be using it, so it checks a bunch of stuff to make sure they can't do something catastrophic like tell the thing to install the image over the top of the running operating system on the host box, calls sync, tells them when it's safe to remove the card, etc etc.
So, I pare a new baseline image down even further, this time explicitly disabling plymouth and other things that might keep me from seeing what's happening, and sure enough, it's doing a kernel panic because it claims it can't find any form of init. We can safely rule out the PARTUUID hooks being wrong because that's practically the only thing that needs to be adjusted--or has needed adjusting--for a system to reboot after being transplanted and that bit of script was one of the first things I wrote. I checked the result and those are definitely set correctly (hit /boot/firmware/cmdline.txt as well as /etc/fstab and /etc/.fstab if you're interested) but still the thing comes a bit and kernel panics. I've even gone back and run a diff on before and after images with zilch of relevance showing up.
About twenty iterations later--because typically when I run into a mystery problem I just start doing it over and over to eliminate the chance of flukes and maybe because I'll see something that will give me a clue--I get irritated enough with the whiny little messages from tar about timestamps in the future to just slap /boot/firmware in the face with find/touch and make that go away. Up until now, I've gotten the result of all this to boot once but this one boots. WTF. I do it again, including forcing those timestamps. Again, it boots. Double WTF. There is now the distinct possibility that the one previous success was caused by me missing (or just despairing too much to care) the step of changing the image's timezone from whatever British one to America/Chicago.
At this point it looks really like somehow the timestamps on the files in /boot/firmware being from the future was making the bootstrap's somewhat "magical" kernel/initrd-hunting process reject the images it should otherwise be happy with. I have since run through all these steps again six times and had six successes.
My question is, what on earth is going on with the hardware's bootstrap that makes it sensitive to timestamps in the boot partition? If these files were cryptographically signed, then perhaps it would be reasonable to care about the timestamps but they're not so what's the thinking there because I'm just not seeing it.
Some years went by and now we have a few hundred of these things rolling around on our LAN and it's going to be easier to just refactor everything and swap out microSD cards than try a dist-upgrade on each and every one of the things out there. So, I started attempting to rebuild everything (considering everything custom that I'm dropping into these things is either a shell script or a perl script along with a few unit files, it ain't like a lot of refactoring has to be done there) beginning with a baseline image. This is where I ran into the almost inscrutable problem... I downloaded the most recent Bookworm build, flashed it to a bog-standard SanDisk 32Gb microSD card, then booted it, and ran through the basic configuration steps: setting the locale, country code, timezone (ahem!), the maintenance role account with ssh key, did a quick round of apt update && apt --assume-yes upgrade, you know... fairly basic stuff, then rebooted a few times to make sure everything was behaving as expected. Powered down, and then siphoned p1 and p2 into tarballs and dropped them into a new profile directory with all the other baseline stuff (scripts that blow away /etc/machine-id, the ssh host keys, and a unit file that generates new ones) and... when I stick it back into a Pi it fails to boot. Notably it's showing a bit of scroll, then blanking the screen and... nothing. The numlock LED even stops working. Kind of predictably a kernel panic of some sort.
Mind you, the odds that there is something wrong with the somewhat heavyweight shells script that generates these new images is just hilariously low (I've been doing Unix/Linux engineering for a couple of decades now). I expected low-knowledge people to be using it, so it checks a bunch of stuff to make sure they can't do something catastrophic like tell the thing to install the image over the top of the running operating system on the host box, calls sync, tells them when it's safe to remove the card, etc etc.
So, I pare a new baseline image down even further, this time explicitly disabling plymouth and other things that might keep me from seeing what's happening, and sure enough, it's doing a kernel panic because it claims it can't find any form of init. We can safely rule out the PARTUUID hooks being wrong because that's practically the only thing that needs to be adjusted--or has needed adjusting--for a system to reboot after being transplanted and that bit of script was one of the first things I wrote. I checked the result and those are definitely set correctly (hit /boot/firmware/cmdline.txt as well as /etc/fstab and /etc/.fstab if you're interested) but still the thing comes a bit and kernel panics. I've even gone back and run a diff on before and after images with zilch of relevance showing up.
About twenty iterations later--because typically when I run into a mystery problem I just start doing it over and over to eliminate the chance of flukes and maybe because I'll see something that will give me a clue--I get irritated enough with the whiny little messages from tar about timestamps in the future to just slap /boot/firmware in the face with find/touch and make that go away. Up until now, I've gotten the result of all this to boot once but this one boots. WTF. I do it again, including forcing those timestamps. Again, it boots. Double WTF. There is now the distinct possibility that the one previous success was caused by me missing (or just despairing too much to care) the step of changing the image's timezone from whatever British one to America/Chicago.
At this point it looks really like somehow the timestamps on the files in /boot/firmware being from the future was making the bootstrap's somewhat "magical" kernel/initrd-hunting process reject the images it should otherwise be happy with. I have since run through all these steps again six times and had six successes.
My question is, what on earth is going on with the hardware's bootstrap that makes it sensitive to timestamps in the boot partition? If these files were cryptographically signed, then perhaps it would be reasonable to care about the timestamps but they're not so what's the thinking there because I'm just not seeing it.
Statistics: Posted by Dagmar — Thu Mar 06, 2025 3:34 pm — Replies 0 — Views 28