After a power outage at the hosting location (3ti), the server wouldnt boot.
Connected remotely to a computer located at the hosting provider, connected to IPMI/iDRAC then used virtual console to see what was being displayed on the server.
Error was:
Time out for waiting the udev queue being empty. Time out for waiting the udev queue being empty. Gave up waiting for root file system device. Common problems: - Boot args (cat /proc/cmdline) - Check rootdelay= (did the system wait long enough?) - Missing modules (cat /proc/modules; ls /dev) ALERT! /dev/mapper/pve-root does not exist. Dropping to a shell! BusyBox v1.35.0 (Debian 1:1.13.0-4+b7) built-in shell (ash) Enter help for a list of built-in commands. (initramfs)_
This built-in shell lacks required tools to perform basic diagnostics let alone any rescue operations, but ls /dev wouldnt list the hard drives.
I went on to check the DRAC/Perc/BIOS configuration to see if the drives we working, they did. The RAID and virtual disks are all healthy so there is probably no hardware failure.
When rebooting GRUB does load (until it fails with the aforementionned message), so the drives HAVE to be working... right...?
A search for /dev/mapper/pve-root does not exist proxmox shows this is a common issue and there are a few ways people have fixed them:
According to Wikipedia chroot is "a shell command and a system call on Unix and Unix-like operating systems that changes the apparent root directory for the current running process and its children."
The whole article is very small and worth a read IMO. The history is fascinating, I remember reading an article in a French magazine at the time (I wonder if it's the same article mentionned in the Wikipedia apge) and couldn't understand how to make that work on a Windows server (silly me, I was young and naive 😂)
"Changes the apparent root directory" this is some powerful system repair stuff, how did I not know about this before?!
Someone in the Proxmox forum posted this chroot tutorial from the SUSE linux KB: https://support.scc.suse.com/s/kb/How-to-Chroot-in-Rescue-Mode?language=en_US
You basically need to mount the drive to a mount point and then "chroot" to it, simple enough.
I couldn't make the for-loop work but it's only 4-5 commands so I mounted the four folders manually by running:
I can't remember what was the disc name in Arch live, I believe it was sda1 through sda4 or sda5 so the above commands were actually something like: mount --rbind /dev/sda2/proc /mnt/proc.
Then chroot to the mounted broken install: chroot /mnt then mount -a to mount other partitions that might be needed.
Once in chroot, I checked which kernel I could try, the official documentation says 6.14 is available as an option.
mount --rbind /proc /mnt/proc
mount --rbind /sys mnt/sys
mount --rbind /dev /mnt/dev
mount --rbind /run /mnt/run
Using this procedure I installed the 6.14 kernel by adding an entry to the repository list.
The server boots fine after installing the 6.14 kernel and selecting it manually at boot. The comments said you could force a default kernel for Proxmox to boot to with the Proxmox-boot-tool kernel pin command.
sudo nano /etc/apt/sources.list.d/pve-install-repo.list
deb [arch=amd64] http://download.proxmox.com/debian/pve bookworm pve-no-subscription
sudo apt update
sudo apt install proxmox-kernel-6.14
Running proxmox-boot-tool kernel list command gives the following results:
Running proxmox-boot-tool kernel pin 6.14.11-8-bpo12-pve outputs:
Manually selected kernels:
None.
Automatically selected kernels:
6.14.11-8-bpo12-pve
6.8.12-23-pve
E: no kernel image found in /boot for '6.14.11-8-bpo12-pve', not setting default. Possible Proxmox kernel versions are: 6.14.11-8-bpo12-pve 6.8.12-23-pve
apt-mark hold proxmox-kernel-6.14.11-8-bpo12-pve-signed proxmox-kernel-6.14.11-8-bpo12-pve-signed set on hold. apt-mark hold proxmox-kernel-6.14 proxmox-kernel-6.14 set on hold. apt-mark hold proxmox-kernel-6.8.12-23-pve-signed proxmox-kernel-6.8.12-23-pve-signed set on hold. apt-mark hold proxmox-kernel-6.8.12-9-pve-signed proxmox-kernel-6.8.12-9-pve-signed set on hold. apt-mark hold proxmox-kernel-6.8 proxmox-kernel-6.8 set on hold.
WARNING: apt does not have a stable CLI interface. Use with caution in scripts. proxmox-default-kernel/stable,now 1.1.0 all [installed] proxmox-kernel-6.14.11-8-bpo12-pve-signed/stable,now 6.14.11-8~bpo12+1 amd64 [installed,automatic] proxmox-kernel-6.14/stable,now 6.14.11-8~bpo12+1 all [installed,upgradable to: 6.14.11-9~bpo12+1] proxmox-kernel-6.8.12-23-pve-signed/stable,now 6.8.12-23 amd64 [installed,automatic] proxmox-kernel-6.8.12-9-pve-signed/stable,now 6.8.12-9 amd64 [installed] proxmox-kernel-6.8/stable,now 6.8.12-23 all [installed,upgradable to: 6.8.12-25] proxmox-kernel-helper/stable,now 8.1.4 all [installed]
drivers/scsi/scsi.c | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/drivers/scsi/scsi.c b/drivers/scsi/scsi.c
index 3e0c0381277a..f0464db3f9de 100644
--- a/drivers/scsi/scsi.c
+++ b/drivers/scsi/scsi.c
@@ -350,6 +350,13 @@ static int scsi_get_vpd_size(struct scsi_device *sdev, u8 page)
if (result < SCSI_VPD_HEADER_SIZE)
return 0;
+ if (result > sizeof(vpd)) {
+ dev_warn_once(&sdev->sdev_gendev,
+ "%s: long VPD page 0 length: %d bytes\n",
+ __func__, result);
+ result = sizeof(vpd);
+ }
+
result -= SCSI_VPD_HEADER_SIZE;
if (!memchr(&vpd[SCSI_VPD_HEADER_SIZE], page, result))
return 0;
--
2.44.0
I believe that an update to the kernel probably caused the issue. Reading online seems to indicate that this is somewhat common with older cards like mine. I did not have good backups in place yet, they were automatically done but on the same RAID as the read of the data, which is of course bad practice, but I was planning on sending these to my other server automatically eventually. For now, I have an external USB drive connected where the VM backups are stored.
https://forum.proxmox.com/threads/timed-out-for-waiting-for-udev-queue-being-empty.129481/