diff --git a/ServerIssueManual.html b/ServerIssueManual.html index e3ea3b6..dfde0c5 100644 --- a/ServerIssueManual.html +++ b/ServerIssueManual.html @@ -148,7 +148,6 @@ The server boots fine after installing the 6.14 kernel and selecting it manually Running proxmox-boot-tool kernel list command gives the following results: - 
 

     Manually selected kernels:
@@ -160,40 +159,105 @@ Running proxmox-boot-tool kernel list command gives the following results
- Running proxmox-boot-tool kernel pin 6.14.11-8-bpo12-pve outputs: +
 

-E: no kernel image found in /boot for '6.14.11-8-bpo12-pve', not setting default.
+    E: no kernel image found in /boot for '6.14.11-8-bpo12-pve', not setting default.
 
-Possible Proxmox kernel versions are:
-6.14.11-8-bpo12-pve
-6.8.12-23-pve
-

+    Possible Proxmox kernel versions are:
+    6.14.11-8-bpo12-pve
+    6.8.12-23-pve
+
+

- Assuming the Automatically selected kernels from kernel list means the new kernels are already pinned. +uname -r gives 6.8.12-23-pve which is good enough for me for now. I would like to make sure the kernel updates arent done without approval though, so I still need to figure out how to do that. -uname -r gives 6.8.12-23-pve which is good enough for me for now. I would like to make sure the kernel updates arent done without approval though, so I still need to figure out how to do that. +In the meantime, I ran these commands which hopefully works well to stop the kernel from updating.
+ + +
+

+    apt-mark hold proxmox-kernel-6.14.11-8-bpo12-pve-signed
+    proxmox-kernel-6.14.11-8-bpo12-pve-signed set on hold.
+
+    apt-mark hold proxmox-kernel-6.14
+    proxmox-kernel-6.14 set on hold.
+
+    apt-mark hold proxmox-kernel-6.8.12-23-pve-signed
+    proxmox-kernel-6.8.12-23-pve-signed set on hold.
+
+    apt-mark hold proxmox-kernel-6.8.12-9-pve-signed
+    proxmox-kernel-6.8.12-9-pve-signed set on hold.
+
+    apt-mark hold proxmox-kernel-6.8
+    proxmox-kernel-6.8 set on hold.
+
+
+ +

+ + +Running apt list --installed|grep kernel then results in: + +
+

+WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
+
+proxmox-default-kernel/stable,now 1.1.0 all [installed]
+proxmox-kernel-6.14.11-8-bpo12-pve-signed/stable,now 6.14.11-8~bpo12+1 amd64 [installed,automatic]
+proxmox-kernel-6.14/stable,now 6.14.11-8~bpo12+1 all [installed,upgradable to: 6.14.11-9~bpo12+1]
+proxmox-kernel-6.8.12-23-pve-signed/stable,now 6.8.12-23 amd64 [installed,automatic]
+proxmox-kernel-6.8.12-9-pve-signed/stable,now 6.8.12-9 amd64 [installed]
+proxmox-kernel-6.8/stable,now 6.8.12-23 all [installed,upgradable to: 6.8.12-25]
+proxmox-kernel-helper/stable,now 8.1.4 all [installed]
+
+

+ + +The bug seems to have been identified in this Proxmox forum thread:
+https://forum.proxmox.com/threads/pve-8-2-kernel-6-8-4-2-does-not-boot-cannot-find-root-device.145764/
+By pschneider1968 (Peter Schneider) who submitted it to Proxmox Bugzilla (and to the Linux mailing list?) with the author of the offending patch (member of the kernel SCSI team) in CC. They then spent time troubleshooting the issue on Peters machine.
+ +The offending code is at the bottom of this commit I believe: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=6d20acbf3e3a32d331947dbc3802cf2d1a399e7d +
+ + +
+
+drivers/scsi/scsi.c | 7 +++++++
+1 file changed, 7 insertions(+)
+
+diff --git a/drivers/scsi/scsi.c b/drivers/scsi/scsi.c
+index 3e0c0381277a..f0464db3f9de 100644
+--- a/drivers/scsi/scsi.c
++++ b/drivers/scsi/scsi.c
+@@ -350,6 +350,13 @@ static int scsi_get_vpd_size(struct scsi_device *sdev, u8 page)
+ 		if (result < SCSI_VPD_HEADER_SIZE)
+ 			return 0;
+ 
++		if (result > sizeof(vpd)) {
++			dev_warn_once(&sdev->sdev_gendev,
++				      "%s: long VPD page 0 length: %d bytes\n",
++				      __func__, result);
++			result = sizeof(vpd);
++		}
++
+ 		result -= SCSI_VPD_HEADER_SIZE;
+ 		if (!memchr(&vpd[SCSI_VPD_HEADER_SIZE], page, result))
+ 			return 0;
+-- 
+2.44.0
+
+
- - - - - - - - - - - -

@@ -201,11 +265,42 @@ uname -r gives 6.8.12-23-pve which is good enough for me for now. I would like t

Conclusion

+

+I believe that an update to the kernel probably caused the issue. Reading online seems to indicate that this is somewhat common with older cards like mine. I did not have good backups in place yet, they were automatically done but on the same RAID as the read of the data, which is of course bad practice, but I was planning on sending these to my other server automatically eventually. +For now, I have an external USB drive connected where the VM backups are stored. + + +

+

Other potential fixes

+

+https://forum.proxmox.com/threads/timed-out-for-waiting-for-udev-queue-being-empty.129481/ +

+ +

+ + +

References

+Other threads with similar issues: +
    +
  1. https://forum.proxmox.com/threads/q-weird-problem-after-upgrade-prox7-8-boot-newer-kernel-alert-dev-mapper-pve-root-does-not-exist-not-bootable.164002/
    2. +
+ + +General Proxmox/Linux help: +
    +
  1. https://cr0x.net/en/proxmox-disks-not-detected-checklist/
    2. +
  2. https://oneuptime.com/blog/post/2026-03-02-how-to-check-hardware-information-with-lshw-dmidecode-and-lspci-on-ubuntu/view
    3. +
  3. How to chroothttps://support.scc.suse.com/s/kb/How-to-Chroot-in-Rescue-Mode?language=en_US
    4. +
  4. How to chroot: https://www.turnkeylinux.org/docs/chroot-to-repair-system
    5. +
  5. Manually pin kernel https://dannyda.com/2023/11/07/proxmox-ve-pve-boot-tool-kernel-tools-how-to-change-kernel-manually-pin-kernel/
    6. +
\ No newline at end of file