The suffering continues

The invisible crash part 2

As far as I understand, the best way to find any normally recorded errors in linux (or ubuntu at least) is using the journalctl command in terminal. This is where I have in the past seen posts claiming errors will be without fail. As I stated previously, when looking at the times in the logs where a crash occured there is simply an end to the log with normal messages and then nothing until the next boot. I have now noticed that in booting after a crash, there is a message recorded about how there are indications of a previous crash. Unfortunately these indications found in boot don't seem to point to the cause of the crash.

kdump

kdump seems to be the common way for people working on kernel (or kernel module?) development to diagnose crashes and issues as far as I can tell. If I understand it right, kdump loads a special linux kernel at boot that reserves a good chunk of ram and saves data from a crash and writes it to disk so that it isn't lost like normal journal errors are if there is a serious crash. kdump has been a pain to configure and I still haven't got it right but I will document what I've learnt as proper documentation is disparate and sparse. Most good documentation is from redhat particularly this page. However certain things are different for redhat than other distros.

After installing kdump through the package manager it might set itself up automatically for you but as usual I wasn't so lucky. The following is used to see if kdump is active and ready to record a crash:

kdump-config show

If the output does not show that it is ready to kdump then there is a problem to fix. between the output of that command and

sudo dmesg

Looking for the output after

kdump-tools[964]: Starting kdump-tools:

In my case the problem was that not enough RAM was being reserved at boot. in the dmesg output I saw a message saying kdump failed to reserve a hexadecimal size of memory. After converting this to decimal it came out at something like 280MB. This means the memory reservation needs to be a value above this (512M is what I went for). The redhat guide gives a command to find this out which didn't work for me on ubuntu.

Setting reserve memory

This took me a long time to workout due to the differences from redhat. When linux is started by the boot manager (in most cases grub), the kernel is started with a command line. Mine currently starts like this:

linux   /boot/vmlinuz-5.15.0-67-generic

You can see this in grub at startup by pressing e and going down to the linux line.

I initially followed the redhat guide to increase the memory reservation by adding the following to /etc/default/grub.

GRUB_CMDLINE_LINUX="crashkernel=2G-:512M@16M

and then using the following command in the terminal:

sudo grub-mkconfig -o /boot/efi/EFI/ubuntu/grub.cfg

Unfortunately this didn't work for me as (maybe due to automatic config when installing kdump) there was a second crashkernel= line after the one I had added appearing in the linux boot command. This was overwriting my value and causing kdump to still not reserve enough memory. I eventually worked out that this was being set in /etc/default/grub.d/kdump-tools.cfg. I removed the previous edit from the /etc/default/grub file and edited the existing line to give 512M.

This did work for getting the kdump crash kernel loading.

The problem

Kubuntu boots but gets stuck before loading the graphical interface. I can get into a tty with ctrl + alt + F2 and logging in thankfully. I can start the graphical interface with

startx

Which gives me a strange version of my desktop with everything rendering a bit weird and stuck in a low resolution. Thankfully there is a recovery option of using an older kernel which is what I am using now. So now (even after uninstalling kdump) I am having to start my desktop through advanced options. I have tried to regenerate the boot options (and kernel I think?) by reverting to the old grub settings and using the grub-mkconfig command but this has not helped. I think my only option now may be a complete reinstall of the operating system to get it back to how it was.

It's quite annoying that my computer is now in a worse state than before and I am now well off the straightforward documentation. I hope that this at least answers how to set a higher memory reservation for kdump on ubuntu and that others have better luck with it than me but for now I'm stuck. I later found this so even kdump may not work but I won't drop it until I know.

The next day

I fixed my linux boot by removing and reinstalling my nvidia driver through the driver manager UI so I am now back to the beginning. I don't have much appetite for breaking it again so soon but I think I have a chance to get the crash log working now. I plan to reinstall kdump and set the memory value again and THEN reinstall nvidia drivers and hopefully it will finally work then. So I will update here if I get that working but for a little while longer I'll put up the crashes.