Table of Contents
After a power outage at the hosting
location (3ti), the server wouldnt boot.
Connected remotely to a computer located at the
hosting provider, connected to IPMI/iDRAC then used virtual console to see what
was being displayed on the server.
The
problem:
Error was:
--------------------------------------------
Time out for waiting the udev queue
being empty.
Time out for waiting the udev queue
being empty.
Gave up waiting for root file system
device. Common problems:
-
Boot
args (cat /proc/cmdline)
-
Check
rootdelay= (did the system wait long enough?)
-
Missing
modules (cat /proc/modules; ls /dev)
ALERT! /dev/mapper/pve-root does not
exist. Dropping to a shell!
BusyBox v1.35.0 (Debian
1:1.13.0-4+b7) built-in shell (ash)
Enter help for a list of built-in
commands.
(initramfs)_
--------------------------------------------
This built-in shell lacks required
tools to perform basic diagnostics let alone any rescue operations, but ls /dev
wouldnt list the hard drives.
Checked in the DRAC/Perc/BIOS and
the drives, RAID and virtual disks are all healthy so there is probably no
hardware failure.
A reboot shows that GRUB does load
so the drives ARE working.
A search for /dev/mapper/pve-root
does not exist proxmox shows this is a common issue and there are a few ways
people have fixed them.
1. Use a different kernel. When
updating the kernel Proxmox keeps older kernel versions and they can be
accessed by selecting the Advanced options for Proxmox Virtual Environment
GNU/Linux boot option in GRUB.
a. https://www.reddit.com/r/homelab/comments/1cetljo/i_need_help_devmapperpveroot_does_not_exist/
2. Use Chroot to perform a rescue
operation on the Linux installation. Including installing a new kernel if the
Proxmox doesnt have other installed kernels.
a. https://forum.proxmox.com/threads/alert-dev-mapper-pve-root-does-not-exist.148029/
3. Add rootdelay=10
to GRUB
a. https://forum.proxmox.com/threads/solved-i-need-help-dev-mapper-pve-root-does-not-exist.116967/
4. Adding GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on iommu=pt" or quiet intel_iommu=off intremap=off to GRUB
I tried option 3 by adding it directly in the GRUB
selection menu to no avail, it didnt change anything. I assumed option 4 didnt have
anything to do with my issue and wasnt even worth a try.
Option
1 did not help
either. I couldnt see an older kernel in the advanced option. Either the
system didnt update or it didnt keep the previous kernel version for me to
boot on.
I then went on to try and boot from
an Ubuntu desktop live USB drive. But even from there I couldnt see the hard
drive listed in /dev, or in lsblk or lshw, etc. At least I now had access to
better tools that initramfs was giving me that I could use to perform a better
diagnosis.
1. dmesg -T | tail -n 200 (look for PCIe, SAS, SATA, NVMe, link resets)
2. lsblk -e7 -o NAME,TYPE,SIZE,MODEL,SERIAL,TRAN,HCTL (see what the kernel created)
3. lspci -nn | egrep -i 'sas|raid|sata|nvme|scsi'
(confirm the controller exists)
As I was getting desperate, I was
basically trying anything I could find only, I eventually saw megaraid_sas
0000:01:00.0 in journalctl -k -b|egrep -i
mpt3sas|megaraid|ahci|nvme|reset|timout|aer.
The RAID card I use is a Dell PERC
h310 so where is that megaraid device coming from? It also has log entries
saying reset/timeout, bingo, I thought, this is the issue, it. times out??
Now lspci does find something if I
filter for megaraid (or lspci -k -s 01:00.0)
instead of trying to find Dell or PERC.
There were also some messages in dmesg
-T | egrep -I ahci|ata|SATA: SATA (1 to 5)
link down (SStatus 0 SControl 300). I assumed this was related to the timeout
with the megaraid card as it is a SAS/SATA compatible RAID device and my drives
are all SATA.
I tried changing the drives to RAID
in the server BIOS configuration even though it didnt make sense, and anyway
no changes were made prior to the server stopping working. The fact that I
could get to GRUB meant the system sees the disks, at least prior to GRUB or
the OS being loaded. Not seeing the disks even in the Ubuntu live I took it
that it meant that it was probably a type of driver issue, not a hardware
problem. I began looking for a different PERC card that might be using a
different driver like an H710 hoping that the new card would be seen properly
by the OS and that the RAID configuration would transfer over to the new card.
The
fix:
From chroot I added the Proxmox to
the deb list and installed kernel 6.14
To chroot:
Need an OS that can see the disks
then mount the folders to the temporary OS
mount --rbind /proc /mnt/proc
mount --rbind /sys /mnt/sys
mount --rbind /dev /mnt/dev
mount --rbind /run /mnt/run
Then chroot to the mounted broken
install: chroot /mnt then mount -a to mount other
partitions that might be needed.
Once in chroot, I checked which kernel I could try, the official documentation says 6.14 is available as an option.
Using this procedure I installed the 6.14 kernel
by adding an entry to the repository list.
sudo nano /etc/apt/sources.list.d/pve-install-repo.list
deb [arch=amd64] http://download.proxmox.com/debian/pve bookworm pve-no-subscription
sudo apt update
sudo apt install proxmox-kernel-6.14
The server boots fine after installing
the 6.14 kernel. The comments said you could force a default kernel for Proxmox
to boot to with the Proxmox-boot-tool
kernel pin command.
Running proxmox-boot-tool kernel list gives the following
results:
--------------------------------------------
Manually selected kernels:
None.
Automatically selected kernels:
6.14.11-8-bpo12-pve
6.8.12-23-pve
--------------------------------------------
Running proxmox-boot-tool kernel pin 6.14.11-8-bpo12-pve
outputs:
--------------------------------------------
E: no kernel image found in /boot
for '6.14.11-8-bpo12-pve', not setting default.
Possible Proxmox kernel versions
are:
6.14.11-8-bpo12-pve
6.8.12-23-pve
--------------------------------------------
Assuming the Automatically selected
kernels from kernel list means the new kernels are already pinned.
uname -r gives 6.8.12-23-pve which is good enough for
me for now. I would like to make sure the kernel updates arent done without
approval though, so I still need to figure out how to do that.
In the meantime, I ran these
commands which hopefully works well to stop the kernel from updating.
apt-mark hold proxmox-kernel-6.14.11-8-bpo12-pve-signed
apt-mark hold proxmox-kernel-6.14
apt-mark hold proxmox-kernel-6.8.12-23-pve-signed
apt-mark hold proxmox-kernel-6.8.12-9-pve-signed
apt-mark hold proxmox-kernel-6.8
--------------------------------------------
root@proxmox3:~#
apt list --installed |grep kernel
WARNING: apt does not have a stable
CLI interface. Use with caution in scripts.
proxmox-default-kernel/stable,now 1.1.0 all [installed]
proxmox-kernel-6.14.11-8-bpo12-pve-signed/stable,now 6.14.11-8~bpo12+1 amd64
[installed,automatic]
proxmox-kernel-6.14/stable,now 6.14.11-8~bpo12+1 all [installed,upgradable to:
6.14.11-9~bpo12+1]
proxmox-kernel-6.8.12-23-pve-signed/stable,now 6.8.12-23 amd64 [installed,automatic]
proxmox-kernel-6.8.12-9-pve-signed/stable,now 6.8.12-9 amd64
[installed]
proxmox-kernel-6.8/stable,now 6.8.12-23 all [installed,upgradable to:
6.8.12-25]
proxmox-kernel-helper/stable,now 8.1.4 all [installed]
root@proxmox3:~#
apt-mark hold proxmox-kernel-6.14.11-8-bpo12-pve-signed
proxmox-kernel-6.14.11-8-bpo12-pve-signed
set on hold.
root@proxmox3:~#
apt-mark hold proxmox-kernel-6.14
proxmox-kernel-6.14 set on hold.
root@proxmox3:~#
apt-mark hold proxmox-kernel-6.8.12-23-pve-signed
proxmox-kernel-6.8.12-23-pve-signed
set on hold.
root@proxmox3:~#
apt-mark hold proxmox-kernel-6.8.12-9-pve-signed
proxmox-kernel-6.8.12-9-pve-signed
set on hold.
root@proxmox3:~#
apt-mark hold proxmox-kernel-6.8
proxmox-kernel-6.8 set on hold.
--------------------------------------------
The bug seems to have been
identified in this Proxmox forum thread: https://forum.proxmox.com/threads/pve-8-2-kernel-6-8-4-2-does-not-boot-cannot-find-root-device.145764/
By pschneider1968 (Peter Schneider)
who submitted it to Proxmox Bugzilla (and to the Linux mailing list?) with the
author of the offending patch (member of the kernel SCSI team) in CC. They then
spent time troubleshooting the issue on Peters machine
The offending code is at the bottom
of this commit I believe: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=6d20acbf3e3a32d331947dbc3802cf2d1a399e7d
--------------------------------------------
drivers/scsi/scsi.c | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/drivers/scsi/scsi.c b/drivers/scsi/scsi.c
index 3e0c0381277a..f0464db3f9de 100644
--- a/drivers/scsi/scsi.c
+++ b/drivers/scsi/scsi.c
@@ -350,6 +350,13 @@ static int scsi_get_vpd_size(struct scsi_device *sdev, u8 page)
if (result < SCSI_VPD_HEADER_SIZE)
return 0;
+ if (result > sizeof(vpd)) {
+ dev_warn_once(&sdev->sdev_gendev,
+ "%s: long VPD page 0 length: %d bytes\n",
+ __func__, result);
+ result = sizeof(vpd);
+ }
+
result -= SCSI_VPD_HEADER_SIZE;
if (!memchr(&vpd[SCSI_VPD_HEADER_SIZE], page, result))
return 0;
--
2.44.0
Conclusion:
I believe that an update to the
kernel probably caused the issue. Reading online seems to indicate that this is
somewhat common with older cards like mine. I did not have good backups in
place yet, they were automatically done but on the same RAID as the read of the
data, which is of course bad practice, but I was planning on sending these to
my other server automatically eventually.
For now, I have an external USB
drive connected where the VM backups are stored.
Other
potential fixes:
https://forum.proxmox.com/threads/timed-out-for-waiting-for-udev-queue-being-empty.129481/
-
Chroot
then modify the thin_check_options in /etc/lvm/lvm.conf
near line 1120
References:
Proxmox boot tool how tos: https://dannyda.com/2023/11/07/proxmox-ve-pve-boot-tool-kernel-tools-how-to-change-kernel-manually-pin-kernel/
Other threads with similar issues:
General Linux help:
1) https://cr0x.net/en/proxmox-disks-not-detected-checklist/
3) How to chroot https://support.scc.suse.com/s/kb/How-to-Chroot-in-Rescue-Mode?language=en_US
4) How to chroot: https://www.turnkeylinux.org/docs/chroot-to-repair-system
5) Manually pin kernel https://dannyda.com/2023/11/07/proxmox-ve-pve-boot-tool-kernel-tools-how-to-change-kernel-manually-pin-kernel/
6)