+
Table of Contents
+ + +
After a power outage at the hosting
+location (3ti), the server wouldnt boot.
Connected remotely to a computer located at the
+hosting provider, connected to IPMI/iDRAC then used virtual console to see what
+was being displayed on the server.
The
+problem:
+
+Error was:
--------------------------------------------
Time out for waiting the udev queue
+being empty.
Time out for waiting the udev queue
+being empty.
Gave up waiting for root file system
+device. Common problems:
-
+Boot
+args (cat /proc/cmdline)
-
+Check
+rootdelay= (did the system wait long enough?)
-
+Missing
+modules (cat /proc/modules; ls /dev)
ALERT! /dev/mapper/pve-root does not
+exist. Dropping to a shell!
BusyBox v1.35.0 (Debian
+1:1.13.0-4+b7) built-in shell (ash)
Enter help for a list of built-in
+commands.
(initramfs)_
--------------------------------------------
This built-in shell lacks required
+tools to perform basic diagnostics let alone any rescue operations, but ls /dev
+wouldnt list the hard drives.
Checked in the DRAC/Perc/BIOS and
+the drives, RAID and virtual disks are all healthy so there is probably no
+hardware failure.
A reboot shows that GRUB does load
+so the drives ARE working.
A search for /dev/mapper/pve-root
+does not exist proxmox shows this is a common issue and there are a few ways
+people have fixed them.
1. Use a different kernel. When
+updating the kernel Proxmox keeps older kernel versions and they can be
+accessed by selecting the Advanced options for Proxmox Virtual Environment
+GNU/Linux boot option in GRUB.
a. https://www.reddit.com/r/homelab/comments/1cetljo/i_need_help_devmapperpveroot_does_not_exist/
2. Use Chroot to perform a rescue
+operation on the Linux installation. Including installing a new kernel if the
+Proxmox doesnt have other installed kernels.
a. https://forum.proxmox.com/threads/alert-dev-mapper-pve-root-does-not-exist.148029/
3. Add rootdelay=10
+to GRUB
a. https://forum.proxmox.com/threads/solved-i-need-help-dev-mapper-pve-root-does-not-exist.116967/
4. +Adding +GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on iommu=pt" or quiet +intel_iommu=off intremap=off to GRUB
+ +I tried option 3 by adding it directly in the GRUB
+selection menu to no avail, it didnt change anything. I assumed option 4 didnt have
+anything to do with my issue and wasnt even worth a try.
Option
+1 did not help
+either. I couldnt see an older kernel in the advanced option. Either the
+system didnt update or it didnt keep the previous kernel version for me to
+boot on.
I then went on to try and boot from
+an Ubuntu desktop live USB drive. But even from there I couldnt see the hard
+drive listed in /dev, or in lsblk or lshw, etc. At least I now had access to
+better tools that initramfs was giving me that I could use to perform a better
+diagnosis.
1. +dmesg -T | +tail -n 200 (look for PCIe, SAS, SATA, NVMe, link +resets)
+ +2. +lsblk -e7 -o +NAME,TYPE,SIZE,MODEL,SERIAL,TRAN,HCTL (see what the kernel created)
+ +3. lspci -nn | egrep -i 'sas|raid|sata|nvme|scsi'
+(confirm the controller exists)
As I was getting desperate, I was
+basically trying anything I could find only, I eventually saw megaraid_sas
+0000:01:00.0 in journalctl -k -b|egrep -i
+mpt3sas|megaraid|ahci|nvme|reset|timout|aer.
The RAID card I use is a Dell PERC
+h310 so where is that megaraid device coming from? It also has log entries
+saying reset/timeout, bingo, I thought, this is the issue, it. times out??
Now lspci does find something if I
+filter for megaraid (or lspci -k -s 01:00.0)
+instead of trying to find Dell or PERC.
There were also some messages in dmesg
+-T | egrep -I ahci|ata|SATA: SATA (1 to 5)
+link down (SStatus 0 SControl 300). I assumed this was related to the timeout
+with the megaraid card as it is a SAS/SATA compatible RAID device and my drives
+are all SATA.
I tried changing the drives to RAID
+in the server BIOS configuration even though it didnt make sense, and anyway
+no changes were made prior to the server stopping working. The fact that I
+could get to GRUB meant the system sees the disks, at least prior to GRUB or
+the OS being loaded. Not seeing the disks even in the Ubuntu live I took it
+that it meant that it was probably a type of driver issue, not a hardware
+problem. I began looking for a different PERC card that might be using a
+different driver like an H710 hoping that the new card would be seen properly
+by the OS and that the RAID configuration would transfer over to the new card.
The
+fix:
+
+From chroot I added the Proxmox to
+the deb list and installed kernel 6.14
To chroot:
Need an OS that can see the disks
+then mount the folders to the temporary OS
+mount --rbind +/proc /mnt/proc
+ ++mount --rbind +/sys /mnt/sys
+ ++mount --rbind +/dev /mnt/dev
+ ++mount --rbind +/run /mnt/run
+ +Then chroot to the mounted broken
+install: chroot /mnt then mount -a to mount other
+partitions that might be needed.
Once in chroot, I checked which kernel I could try, the official documentation says 6.14 is available as an option.
Using this procedure I installed the 6.14 kernel
+by adding an entry to the repository list.
+sudo nano /etc/apt/sources.list.d/pve-install-repo.list
deb [arch=amd64] +http://download.proxmox.com/debian/pve bookworm pve-no-subscription
+ +sudo apt update
+ +sudo apt install proxmox-kernel-6.14
The server boots fine after installing
+the 6.14 kernel. The comments said you could force a default kernel for Proxmox
+to boot to with the Proxmox-boot-tool
+kernel pin command.
Running proxmox-boot-tool kernel list gives the following
+results:
--------------------------------------------
Manually selected kernels:
None.
Automatically selected kernels:
6.14.11-8-bpo12-pve
6.8.12-23-pve
--------------------------------------------
Running proxmox-boot-tool kernel pin 6.14.11-8-bpo12-pve
+outputs:
--------------------------------------------
E: no kernel image found in /boot
+for '6.14.11-8-bpo12-pve', not setting default.
Possible Proxmox kernel versions
+are:
6.14.11-8-bpo12-pve
6.8.12-23-pve
--------------------------------------------
Assuming the Automatically selected
+kernels from kernel list means the new kernels are already pinned.
uname -r gives 6.8.12-23-pve which is good enough for
+me for now. I would like to make sure the kernel updates arent done without
+approval though, so I still need to figure out how to do that.
In the meantime, I ran these
+commands which hopefully works well to stop the kernel from updating.
+
+ apt-mark hold proxmox-kernel-6.14.11-8-bpo12-pve-signed
+ apt-mark hold proxmox-kernel-6.14
+ apt-mark hold proxmox-kernel-6.8.12-23-pve-signed
+ apt-mark hold proxmox-kernel-6.8.12-9-pve-signed
+ apt-mark hold proxmox-kernel-6.8
+
+
+
+
+
+--------------------------------------------
root@proxmox3:~#
+apt list --installed |grep kernel
WARNING: apt does not have a stable
+CLI interface. Use with caution in scripts.
proxmox-default-kernel/stable,now 1.1.0 all [installed]
proxmox-kernel-6.14.11-8-bpo12-pve-signed/stable,now 6.14.11-8~bpo12+1 amd64
+[installed,automatic]
proxmox-kernel-6.14/stable,now 6.14.11-8~bpo12+1 all [installed,upgradable to:
+6.14.11-9~bpo12+1]
proxmox-kernel-6.8.12-23-pve-signed/stable,now 6.8.12-23 amd64 [installed,automatic]
proxmox-kernel-6.8.12-9-pve-signed/stable,now 6.8.12-9 amd64
+[installed]
proxmox-kernel-6.8/stable,now 6.8.12-23 all [installed,upgradable to:
+6.8.12-25]
proxmox-kernel-helper/stable,now 8.1.4 all [installed]
root@proxmox3:~#
+apt-mark hold proxmox-kernel-6.14.11-8-bpo12-pve-signed
proxmox-kernel-6.14.11-8-bpo12-pve-signed
+set on hold.
root@proxmox3:~#
+apt-mark hold proxmox-kernel-6.14
proxmox-kernel-6.14 set on hold.
root@proxmox3:~#
+apt-mark hold proxmox-kernel-6.8.12-23-pve-signed
proxmox-kernel-6.8.12-23-pve-signed
+set on hold.
root@proxmox3:~#
+apt-mark hold proxmox-kernel-6.8.12-9-pve-signed
proxmox-kernel-6.8.12-9-pve-signed
+set on hold.
root@proxmox3:~#
+apt-mark hold proxmox-kernel-6.8
proxmox-kernel-6.8 set on hold.
--------------------------------------------
The bug seems to have been
+identified in this Proxmox forum thread: https://forum.proxmox.com/threads/pve-8-2-kernel-6-8-4-2-does-not-boot-cannot-find-root-device.145764/
By pschneider1968 (Peter Schneider)
+who submitted it to Proxmox Bugzilla (and to the Linux mailing list?) with the
+author of the offending patch (member of the kernel SCSI team) in CC. They then
+spent time troubleshooting the issue on Peters machine
The offending code is at the bottom
+of this commit I believe: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=6d20acbf3e3a32d331947dbc3802cf2d1a399e7d
--------------------------------------------
+
+ drivers/scsi/scsi.c | 7 +++++++
+ 1 file changed, 7 insertions(+)
+
+diff --git a/drivers/scsi/scsi.c b/drivers/scsi/scsi.c
+index 3e0c0381277a..f0464db3f9de 100644
+--- a/drivers/scsi/scsi.c
++++ b/drivers/scsi/scsi.c
+@@ -350,6 +350,13 @@ static int scsi_get_vpd_size(struct scsi_device *sdev, u8 page)
+ if (result < SCSI_VPD_HEADER_SIZE)
+ return 0;
+
++ if (result > sizeof(vpd)) {
++ dev_warn_once(&sdev->sdev_gendev,
++ "%s: long VPD page 0 length: %d bytes\n",
++ __func__, result);
++ result = sizeof(vpd);
++ }
++
+ result -= SCSI_VPD_HEADER_SIZE;
+ if (!memchr(&vpd[SCSI_VPD_HEADER_SIZE], page, result))
+ return 0;
+--
+2.44.0
+
+
+
+
+
+
+
+Conclusion:
+
+I believe that an update to the
+kernel probably caused the issue. Reading online seems to indicate that this is
+somewhat common with older cards like mine. I did not have good backups in
+place yet, they were automatically done but on the same RAID as the read of the
+data, which is of course bad practice, but I was planning on sending these to
+my other server automatically eventually.
For now, I have an external USB
+drive connected where the VM backups are stored.
Other
+potential fixes:
+
+https://forum.proxmox.com/threads/timed-out-for-waiting-for-udev-queue-being-empty.129481/
-
+Chroot
+then modify the thin_check_options in /etc/lvm/lvm.conf
+near line 1120
References:
+
+Proxmox boot tool how tos: https://dannyda.com/2023/11/07/proxmox-ve-pve-boot-tool-kernel-tools-how-to-change-kernel-manually-pin-kernel/
Other threads with similar issues:
General Linux help:
1) https://cr0x.net/en/proxmox-disks-not-detected-checklist/
3) How to chroot https://support.scc.suse.com/s/kb/How-to-Chroot-in-Rescue-Mode?language=en_US
4) How to chroot: https://www.turnkeylinux.org/docs/chroot-to-repair-system
5) Manually pin kernel https://dannyda.com/2023/11/07/proxmox-ve-pve-boot-tool-kernel-tools-how-to-change-kernel-manually-pin-kernel/
6)