From c4290f29484d64037bad23e74356a7abfa3bb6ed Mon Sep 17 00:00:00 2001 From: Seb Trudel Date: Mon, 8 Jun 2026 17:33:17 -0400 Subject: [PATCH] Ongoing writing --- ServerIssueManual.html | 96 +++++++++++++++++++++++++++++++++++++----- 1 file changed, 85 insertions(+), 11 deletions(-) diff --git a/ServerIssueManual.html b/ServerIssueManual.html index 1b563f8..6291960 100644 --- a/ServerIssueManual.html +++ b/ServerIssueManual.html @@ -46,9 +46,9 @@ Enter help for a list of built-in commands.

This built-in shell lacks required tools to perform basic diagnostics let alone any rescue operations, but ls /dev wouldnt list the hard drives.

-Checked in the DRAC/Perc/BIOS and the drives, RAID and virtual disks are all healthy so there is probably no hardware failure.
-A reboot shows that GRUB does load so the drives ARE working.

-A search for /dev/mapper/pve-root does not exist proxmox shows this is a common issue and there are a few ways people have fixed them.
+I went on to check the DRAC/Perc/BIOS configuration to see if the drives we working, they did. The RAID and virtual disks are all healthy so there is probably no hardware failure. +When rebooting GRUB does load (until it fails with the aforementionned message), so the drives HAVE to be working... right...?

+A search for /dev/mapper/pve-root does not exist proxmox shows this is a common issue and there are a few ways people have fixed them:

  1. Use a different kernel. When updating the kernel Proxmox keeps older kernel versions and they can be accessed by selecting the Advanced options for Proxmox Virtual Environment GNU/Linux boot option in GRUB
      @@ -80,11 +80,88 @@ I then went on to try and boot from an Ubuntu desktop live USB drive. But even f
    1. lspci -nn | egrep -i 'sas|raid|sata|nvme|scsi' (confirm the controller exists)
    -As I was getting desperate, I was basically trying anything I could find only, I eventually saw megaraid_sas 0000:01:00.0 in journalctl -k -b|egrep -i mpt3sas|megaraid|ahci|nvme|reset|timout|aer. -The RAID card I use is a Dell PERC h310 so where is that megaraid device coming from? It also has log entries saying reset/timeout, bingo, I thought, this is the issue, it. times out?? -Now lspci does find something if I filter for megaraid (or lspci -k -s 01:00.0) instead of trying to find Dell or PERC. -There were also some messages in dmesg -T | egrep -I ahci|ata|SATA: SATA (1 to 5) link down (SStatus 0 SControl 300). I assumed this was related to the timeout with the megaraid card as it is a SAS/SATA compatible RAID device and my drives are all SATA. -I tried changing the drives to RAID in the server BIOS configuration even though it didnt make sense, and anyway no changes were made prior to the server stopping working. The fact that I could get to GRUB meant the system sees the disks, at least prior to GRUB or the OS being loaded. Not seeing the disks even in the Ubuntu live I took it that it meant that it was probably a type of driver issue, not a hardware problem. I began looking for a different PERC card that might be using a different driver like an H710 hoping that the new card would be seen properly by the OS and that the RAID configuration would transfer over to the new card. +As I was getting desperate, I was basically trying anything I could find only, I eventually saw "megaraid_sas 0000:01:00.0" in journalctl -k -b|egrep -i mpt3sas|megaraid|ahci|nvme|reset|timout|aer. +The RAID card I use is a Dell PERC h310 so where is that megaraid device coming from? It also has log entries saying reset/timeout, bingo, I thought, this has to be the issue it times out??? +Now lspci does find something if I filter for megaraid (or lspci -k -s 01:00.0) instead of trying to find Dell or PERC. +There were also some messages in dmesg -T | egrep -I ahci|ata|SATA: SATA (1 to 5) link down (SStatus 0 SControl 300). I assumed this was related to the timeout with the megaraid card as it is a SAS/SATA compatible RAID device and my drives are all SATA. +I tried changing the drives to RAID from AHCI in the server BIOS configuration even though it didnt make sense, no changes were made prior to the server stopping working anyway. Again, the fact that I could get to GRUB meant the system sees the disks to GRUB/OS being loaded. Not seeing the disks even in the Ubuntu live I took it that it meant that it was probably a type of driver issue, not a hardware problem. I began looking for a different PERC card that might be using a different driver like an H710 hoping that the new card would be seen properly by the OS and that the RAID configuration would transfer over to the new card.
    + +Both Ubuntu and Promox couldn't see the drives/card I figured maybe the issue is related to Debian based OS so instead of booting from a Ubuntu live usb thumb drive I made a new one with Arch instead and could then see the drives now! I felt such a relief, I really didn't want to have to rebuild the server/VMs. I would have also lost my Matrix/Synapse chat history, my girlfriend and I replaced SMS/RCS/instant messaging with this over the last few months.
    +But what do I do now? I had a few choices, I could extract the VMs, reinstall Proxmox and restore them, at least I could have my stuff back but how do I make sure I don't run in the same issue again (EDIT: the issue what with a commit in the linux kernel and should be fixed now)? I could get a new server or get a different RAID card and hope it works. + +Option 2 from the forum posts is intriguing but wtf is a "chroot"? +

    + + +

    The Fix

    + +

    chroot

    +

    + According to Wikipedia chroot is "a shell command and a system call on Unix and Unix-like operating systems that changes the apparent root directory for the current running process and its children."
    The whole article is very small and worth a read IMO. The history is fascinating, I remember reading an article in a French magazine at the time (I wonder if it's the same article mentionned in the Wikipedia apge) and couldn't understand how to make that work on a Windows server (silly me, I was young and naive 😂) +"Changes the apparent root directory" this is some powerful system repair stuff, how did I not know about this before?!

    + + +Someone in the Proxmox forum posted this chroot tutorial from the SUSE linux KB: https://support.scc.suse.com/s/kb/How-to-Chroot-in-Rescue-Mode?language=en_US
    + + +You basically need to mount the drive to a mount point and then "chroot" to it, simple enough.
    +I couldn't make the for-loop work but it's only 4-5 commands so I mounted the four folders manually by running: + + + +

    +
    
+    mount --rbind /proc /mnt/proc
+    mount --rbind /sys mnt/sys
+    mount --rbind /dev /mnt/dev
+    mount --rbind /run /mnt/run
+
+
    
+
    + + + + +I can't remember what was the disc name in Arch live, I believe it was sda1 through sda4 or sda5 so the above commands were actually something like: mount --rbind /dev/sda2/proc /mnt/proc. + +Then chroot to the mounted broken install: chroot /mnt then mount -a to mount other partitions that might be needed. +Once in chroot, I checked which kernel I could try, the official documentation says 6.14 is available as an option.
    +Using this procedure I installed the 6.14 kernel by adding an entry to the repository list. + + + +
    +
    
+    sudo nano /etc/apt/sources.list.d/pve-install-repo.list
+    deb [arch=amd64] http://download.proxmox.com/debian/pve bookworm pve-no-subscription
+    sudo apt update
+    sudo apt install proxmox-kernel-6.14
+
+
    
+
    +     +     + + +The server boots fine after installing the 6.14 kernel. The comments said you could force a default kernel for Proxmox to boot to with the Proxmox-boot-tool kernel pin command.

    + +Running proxmox-boot-tool kernel list command gives the following results: + + + +
    +
    
+    Manually selected kernels:
+    None.
+
+    Automatically selected kernels:
+    6.14.11-8-bpo12-pve
+    6.8.12-23-pve
+
+
    
+
    +     +     @@ -92,9 +169,6 @@ I tried changing the drives to RAID in the server BIOS configuration even though - -

    The Fix

    -

    Conclusion

    Other potential fixes