From 3f2e57a8a9714067f8cc97306a1508796254e715 Mon Sep 17 00:00:00 2001 From: "Guilherme G. Piccoli" Date: Thu, 27 Jan 2022 10:17:10 -0300 Subject: [PATCH] Improve documentation ...specially the HOW-TO section for new users. Signed-off-by: Guilherme G. Piccoli --- PKGBUILD | 2 +- README.md | 121 +++++++++++++++++++++++++++++++++++------------------- 2 files changed, 79 insertions(+), 44 deletions(-) diff --git a/PKGBUILD b/PKGBUILD index 299a91b..b3e37e0 100644 --- a/PKGBUILD +++ b/PKGBUILD @@ -30,7 +30,7 @@ sha256sums=('dbedff54addfb5dce51614c73df04c90fca9f27d0d3a690243259ccbbfcca07c' '06b38bd9f09da5fb22a765b6f1945fc349cc5f9d13cd32c9218b9b60b40a9010' '12a9124b907f208471ba7aaac0f3261cbbd34a168cce3260fa9e7793994beebd' '26bc2b64af0d468f050c0e0dd9e2053176d56886edad9146bc495797bf2c5810' - 'b87fb8e4c4602f8ddc3b0bf6d6175d0ee7b9e0942f4dca8f1b958ed3ad445470' + '0b602c27a5ff30c4fdb944135bbd9309225f31fb4ce354b7b07a03ce56b55531' '8f974b51f6fcf5ec4ae8bb9d585390d4354cf8d99ae98e0efe9484f08abb1949' 'cbb207ecc0f6bacefbeed41f0d4910daac6500ac2345366e1f95f09a7653c65a') diff --git a/README.md b/README.md index c92f9de..4bd0f61 100644 --- a/README.md +++ b/README.md @@ -3,47 +3,81 @@ # # Copyright (c) 2021 Valve. # -# Code by Guilherme G. Piccoli +# Maintained by Guilherme G. Piccoli # # # ########################################################################### # ############################ SteamOS Kdump ############################## # ########################################################################### # -# This is the first version of SteamOS Kdump infrastructure. The goal is to -# collect data whenever a kernel crash is detected. There is a lightweight +# +# This is the SteamOS Kdump/Pstore infrastructure; the goal is to collect +# data whenever a kernel crash is detected. There is a lightweight # collection, that only grabs dmesg, and a more complete setting to grab the -# whole (compressed) vmcore. The tunnings are available at /etc/default/kdump. +# whole (compressed) vmcore. See the DETAILS section below for more info. # -# Also, the infrastructure is able to configure and save pstore-RAM logs; -# this is the default option. # -# After installation and a reboot, things should be all set EXCEPT for GRUB -# config - please check the CAVEATS/INSTRUCTIONS section below. Notice the -# package is under active development, this version should still be considered -# a kind of "Proof Of Concept" - improvements are expected in the near future. -# Thanks for testing!!! +# ############################ HOW-TO USE IT ############################## # # +# 1. Install the package with pacman if not available in your image - there's +# a pre-built binary package in this gitlab; to check if it's already installed +# look the pacman installed package list. Also, be sure the systemd service was +# properly loaded by checking 'systemctl status kdump-steamos.service'. +# +# 2. Only the dmesg is collected, and by default this happens via the Pstore +# mechanism, i.e., no extra memory should be reserved and no GRUB change is +# required. If 'lsmod' shows "ramoops", then Pstore is in use. +# +# 3. The logs are stored in a ZIP file at "/home/.steamos/offload/var/kdump/"; +# besides the dmesg with some extra information, the image build version, +# running kernel version and dmidecode are stored in this ZIP file as well. +# This file is named as: "steamos-SERIAL-STEAM_USER.timestamp.zip", where +# SERIAL is the machine serial (from dmidecode), STEAM_USER is the Steam +# account name (based on the last logged Steam user) and timestamp tz is UTC. +# +# 4. (IMPORTANT) Please, test the infrastructure in order to see if a dummy +# crash log is collected before using it to try debugging complex issues. +# In order to do that, login to a shell and execute, as root user: +# 'echo 1 > /proc/sys/kernel/sysrq ; echo c > /proc/sysrq-trigger' +# +# This action will trigger a dummy crash and reboot the system; check if there +# is a ZIP file with the crash logs in the directory described in (3). +# +# 5. Some tunnings are available at "/etc/default/kdump"; for example users +# can choose Kdump instead of Pstore (USE_PSTORE_RAM), and if using Kdump, +# collect the full vmcore (FULL_COREDUMP). The vmcore is not stored in the +# ZIP file, but it's saved in "/home/.steamos/offload/var/kdump/crash/". +# NOTICE that, if Kdump is used instead of Pstore, the following flags are +# needed in GRUB cmdline: "crashkernel=192M crash_kexec_post_notifiers" and +# a regular reboot is necessary. +# +# +# ############################## DETAILS ################################## +# # CAVEATS / INSTRUCTIONS # ########################################################################### -# (a) For now, we don't automatically edit any GRUB config, so the minimum -# necessary action after installing this package is to add "crashkernel=192M" -# to your GRUB config in order subsequent boots pick this setting and do reserve -# the memory, or else kdump cannot work. The memory amount was empirically -# determined - 144M wasn't enough and 160M is unstable, so 192M seems good enough. -# If you prefer to rely on pstore-RAM, no GRUB setting should be required; this -# is currently the default (see /etc/default/kdump). +# (a) Currently, we don't automatically edit GRUB config; see TODO (1) below. +# This is not required if Pstore is used (which is the default). # -# (b) It requires (obviously) a RW rootfs - we've used tune2fs in order to make -# it read-write, since it's RO by default. Also, we assume the nvme partition -# scheme is default across all versions and didn't change with new updates -# for example - both kdump and pstore relies in mounting partitions, etc. +# In case Kdump is used, boot-time reserved memory is required, check step (5) +# in the HOW-TO above. The memory amount was empirically determined - setting +# 144M wasn't enough and 160M is unstable, so 192M seems good enough. This +# amount might change in future kernel versions, requiring tests using the +# approach suggested in the step (4) above. # -# (c) Due to a post-transaction hook executed by libalpm (90-dracut-install.hook), -# unfortunately after installing the kdump-steamos package *all* initramfs images -# are recreated - this is not necessary, we're thinking on how to prevent that, -# but for now be prepared: the installation take some (long) minutes due to that ={ +# (b) The kdump-steamos package requires a RW rootfs in case it's not currently +# embedded in your image. Users can make use of 'tune2fs' or 'steamos-readonly' +# in order to make the rootfs RW, since it's RO by default. Also, we assume the +# nvme partitioning scheme is default across all versions (A/B, nvme0n1p4 / p5 +# are the root ones, etc) and didn't change with new updates, for example. Both +# Kdump and Pstore facilities relies in mounting partitions. +# +# (c) Due to a post-transaction hook exec'ed by libalpm (90-dracut-install.hook) +# unfortunately after installing the kdump-steamos package *all* initramfs +# images are recreated - this is not necessary, we're thinking how to prevent +# that, but for now be prepared: the installation take some (long) minutes only +# due to that ={ # # (d) Unfortunately makedumpfile from Arch Linux is not available on official # repos, only in AUR. But it is available on Holo, so we make use of that. @@ -52,30 +86,31 @@ # https://aur.archlinux.org/packages/makedumpfile/#comment-843853 # # -# TODOs (for now - we expect to have more after some testing by the colleagues) -# +# TODOs +# ########################################################################### # (1) We'd like to be able to automatically edit GRUB and recreate its config -# file - after some future discussion on the proper parameters, this is expected -# to be added to the package. +# file - implementation tests are ongoing. # -# (2) Hopefully we can fix/prevent the unnecessary re-creation of all initramfs -# images - it happens due to our pkg installing files on /usr/lib/dracut/modules.d -# which is a trigger for this initramfs recreation. +# (2) The log submission mechanism is incomplete - we save the logs as a local +# ZIP file (as discussed in the HOW-TO), but they aren't submitted to a remote +# Valve server. There's an API in-place, so the implementation is starting. # -# (3) We have a "fragile" way of determining a mount point required for kdump; -# this is something to improve in order to make the kdump more reliable. +# (3) Hopefully we can fix/prevent the unnecessary re-creation of all initramfs +# images - it happens due to our package installing files on directory +# "/usr/lib/dracut/modules.d" which triggers the unfortunate initramfs rebuild. # -# (4) Add a more reliable reboot mechanism - we had seen issues with "reboot -f" -# in the past and relying in sysrq reboot as a quirk managed to be a safe option, -# so this is something to think about here. Should be easy to implement. +# (4) We have a "fragile" way of determining a mount point required for Kdump; +# this is something to improve maybe, in order to make the Kdump more reliable. # -# (5) The log submission mechanism is incomplete - we save the logs as tar.zst -# files, but they are not submitted to any remote server, etc. -# -# (6) Pstore ramoops backend has some limitations that we're discussing with +# (5) Pstore ramoops backend has some limitations that we're discussing with # the kernel community - right now we can only collect ONE dmesg and its # size is truncated on "record_size" bytes, not allowing a file split like -# efi-pstore; hopefully we can improve that. +# efi-pstore; thankfully we still can collect 2MiB dmesg, but hopefully we can +# improve that upstream. +# +# (6) Add a more reliable reboot mechanism - we had seen issues in the past +# with "reboot -f", and relying in sysrq reboot as a quirk managed to be a safe +# option, so this is something to think about. Should be easy to implement. # # (7) Maybe a good idea would be to allow creating the minimum image for any # specified kernel, not only for the running one (which is what we do now).