Troubleshooting • Raspberry Pi 5 : memory cgroup configuration issue for k3s

Code:

lsb_release -aNo LSB modules are available.Distributor ID:DebianDescription:Debian GNU/Linux 12 (bookworm)Release:12Codename:bookworm

Code:

uname -aLinux pi-01 6.1.74-v8-16k+ #1725 SMP PREEMPT Mon Jan 22 13:38:39 GMT 2024 aarch64 GNU/Linux

Code:

cat /boot/firmware/cmdline.txtconsole=serial0,115200 console=tty1 root=PARTUUID=147263d9-02 rootfstype=ext4 fsck.repair=yes rootwait cfg80211.ieee80211_regdom=US cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory

Code:

sudo rpi-update *** Raspberry Pi firmware updater by Hexxeh, enhanced by AndrewS and Dom *** Performing self-update *** Relaunching after update *** Raspberry Pi firmware updater by Hexxeh, enhanced by AndrewS and DomFW_REV:d86b5843d68b9972a5430a6d3da1b271cfc83521BOOTLOADER_REV:c987375f1d9633ea7d33d30f68a024055b76b9a7 *** We're running for the first time *** Backing up files (this will take a few minutes) *** Remove old firmware backup *** Backing up firmware *** Remove old modules backup *** Backing up modules 6.1.0-rpi8-rpi-2712WANT_32BIT:0 WANT_64BIT:1 WANT_PI4:1 WANT_PI5:1#############################################################WARNING: This update bumps to rpi-6.1.y linux treeSee: https://forums.raspberrypi.com/viewtopic.php?t=344246'rpi-update' should only be used if there is a specificreason to do so - for example, a request by a Raspberry Piengineer or if you want to help the testing effortand are comfortable with restoring if there are regressions.DO NOT use 'rpi-update' as part of a regular update process.##############################################################Would you like to proceed? (y/N)Downloading bootloader toolsDownloading bootloader images *** Downloading specific firmware revision (this will take a few minutes)  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current                                 Dload  Upload   Total   Spent    Left  Speed  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0100  144M  100  144M    0     0  21.2M      0  0:00:06  0:00:06 --:--:-- 25.9M*** PREPARING EEPROM UPDATES ***BOOTLOADER: update available   CURRENT: Fri  5 Jan 15:57:40 UTC 2024 (1704470260)    LATEST: Mon  5 Feb 14:38:34 UTC 2024 (1707143914)   RELEASE: latest (/lib/firmware/raspberrypi/bootloader-2712/latest)            Use raspi-config to change the release.   CURRENT: Fri  5 Jan 15:57:40 UTC 2024 (1704470260)    UPDATE: Mon  5 Feb 14:38:34 UTC 2024 (1707143914)    BOOTFS: /boot/firmware'/tmp/tmp.tKmXhxVki6' -> '/boot/firmware/pieeprom.upd'Copying recovery.bin to /boot/firmware for EEPROM updateEEPROM updates pending. Please reboot to apply the update.To cancel a pending update run "sudo rpi-eeprom-update -r". *** Updating firmware *** Updating kernel modules *** depmod 6.1.74-v8-16k+ *** depmod 6.1.74-v8+ *** Updating VideoCore libraries *** Using SoftFP libraries *** Updating SDK *** Running ldconfig *** Storing current firmware revision *** Deleting downloaded files *** Syncing changes to disk *** If no errors appeared, your firmware was successfully updated to d86b5843d68b9972a5430a6d3da1b271cfc83521 *** A reboot is needed to activate the new firmware

Code:

cat /proc/cgroups#subsys_namehierarchynum_cgroupsenabledcpuset0391cpu0391cpuacct0391blkio0391memory0390devices0391freezer0391net_cls0391perf_event0391net_prio0391pids0391

Code:

░░ The job identifier is 30165 and the job result is done.Feb 07 00:17:24 pi-01 systemd[1]: Starting k3s.service - Lightweight Kubernetes...░░ Subject: A start job for unit k3s.service has begun execution░░ Defined-By: systemd░░ Support: https://www.debian.org/support░░ ░░ A start job for unit k3s.service has begun execution.░░ ░░ The job identifier is 30165.Feb 07 00:17:24 pi-01 sh[8038]: + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.serviceFeb 07 00:17:24 pi-01 k3s[8042]: time="2024-02-07T00:17:24-08:00" level=info msg="Starting k3s v1.28.6+k3s2 (c9f49a3b)"Feb 07 00:17:24 pi-01 k3s[8042]: time="2024-02-07T00:17:24-08:00" level=info msg="Configuring sqlite3 database connection pooling: maxIdleConns=2, maxOpenConns=0, connMaxLifetime=0s"Feb 07 00:17:24 pi-01 k3s[8042]: time="2024-02-07T00:17:24-08:00" level=info msg="Configuring database table schema and indexes, this may take a moment..."Feb 07 00:17:24 pi-01 k3s[8042]: time="2024-02-07T00:17:24-08:00" level=info msg="Database tables and indexes are up to date"Feb 07 00:17:24 pi-01 k3s[8042]: time="2024-02-07T00:17:24-08:00" level=info msg="Kine available at unix://kine.sock"Feb 07 00:17:24 pi-01 k3s[8042]: time="2024-02-07T00:17:24-08:00" level=info msg="Reconciling bootstrap data between datastore and disk"Feb 07 00:17:25 pi-01 k3s[8042]: time="2024-02-07T00:17:25-08:00" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --authorizat>Feb 07 00:17:25 pi-01 k3s[8042]: time="2024-02-07T00:17:25-08:00" level=info msg="Running kube-scheduler --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/sche>Feb 07 00:17:25 pi-01 k3s[8042]: time="2024-02-07T00:17:25-08:00" level=info msg="Waiting for API server to become available"Feb 07 00:17:25 pi-01 k3s[8042]: time="2024-02-07T00:17:25-08:00" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=>Feb 07 00:17:25 pi-01 k3s[8042]: I0207 00:17:25.209491    8042 options.go:220] external host was not specified, using 192.168.86.40Feb 07 00:17:25 pi-01 k3s[8042]: time="2024-02-07T00:17:25-08:00" level=info msg="Running cloud-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/cloud-controller.kubeconfig --authorization-kube>Feb 07 00:17:25 pi-01 k3s[8042]: I0207 00:17:25.210496    8042 server.go:156] Version: v1.28.6+k3s2Feb 07 00:17:25 pi-01 k3s[8042]: I0207 00:17:25.210529    8042 server.go:158] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""Feb 07 00:17:25 pi-01 k3s[8042]: time="2024-02-07T00:17:25-08:00" level=info msg="Server node token is available at /var/lib/rancher/k3s/server/token"Feb 07 00:17:25 pi-01 k3s[8042]: time="2024-02-07T00:17:25-08:00" level=info msg="To join server node to cluster: k3s server -s https://192.168.86.40:6443 -t ${SERVER_NODE_TOKEN}"Feb 07 00:17:25 pi-01 k3s[8042]: time="2024-02-07T00:17:25-08:00" level=info msg="Agent node token is available at /var/lib/rancher/k3s/server/agent-token"Feb 07 00:17:25 pi-01 k3s[8042]: time="2024-02-07T00:17:25-08:00" level=info msg="To join agent node to cluster: k3s agent -s https://192.168.86.40:6443 -t ${AGENT_NODE_TOKEN}"Feb 07 00:17:25 pi-01 k3s[8042]: time="2024-02-07T00:17:25-08:00" level=info msg="Wrote kubeconfig /etc/rancher/k3s/k3s.yaml"Feb 07 00:17:25 pi-01 k3s[8042]: time="2024-02-07T00:17:25-08:00" level=info msg="Run: k3s kubectl"Feb 07 00:17:25 pi-01 k3s[8042]: time="2024-02-07T00:17:25-08:00" level=fatal msg="failed to find memory cgroup (v2)"Feb 07 00:17:25 pi-01 systemd[1]: k3s.service: Main process exited, code=exited, status=1/FAILURE░░ Subject: Unit process exited░░ Defined-By: systemd░░ Support: https://www.debian.org/support░░ ░░ An ExecStart= process belonging to unit k3s.service has exited.░░ ░░ The process' exit code is 'exited' and its exit status is 1.Feb 07 00:17:25 pi-01 systemd[1]: k3s.service: Failed with result 'exit-code'.░░ Subject: Unit failed░░ Defined-By: systemd░░ Support: https://www.debian.org/support░░ ░░ The unit k3s.service has entered the 'failed' state with result 'exit-code'.Feb 07 00:17:25 pi-01 systemd[1]: Failed to start k3s.service - Lightweight Kubernetes.░░ Subject: A start job for unit k3s.service has failed░░ Defined-By: systemd░░ Support: https://www.debian.org/support░░ ░░ A start job for unit k3s.service has finished with a failure.░░ ░░ The job identifier is 30165 and the job result is failed.

Code:

$ vcgencmd get_config strdevice_tree=-overlay_prefix=overlays/hdmi_cvt:0=hdmi_cvt:1=hdmi_edid_filename:0=hdmi_edid_filename:1=hdmi_timings:0=hdmi_timings:1=

Code:

$ vcgencmd get_config intarm_64bit=1arm_boost=1arm_freq=2400arm_freq_min=1500arm_peri_high=1audio_pwm_mode=2auto_initramfs=1avs_temp=63659camera_auto_detect=1core_freq=910core_freq_min=500disable_commandline_tags=2disable_fw_kms_setup=1disable_l2cache=1disable_overscan=1display_auto_detect=1display_default_lcd=-1display_hdmi_rotate=-1display_lcd_rotate=-1dvfs=4enable_gic=1enable_uart=-1force_eeprom_read=1force_pwm_open=1framebuffer_depth=16framebuffer_ignore_alpha=1framebuffer_swap=1gpu_freq_min=500hdmi_enable_4kp60=1hevc_freq=910hevc_freq_min=500ignore_lcd=-1init_uart_clock=0x2dc6c00isp_freq=910isp_freq_min=500mask_gpu_interrupt1=16418max_framebuffers=2over_voltage_avs=0x18380pause_burst_frames=1pciex4_reset=1pmic_turbo_threshold=600program_serial_random=1total_mem=8192usb_max_current_enable=1v3d_freq=960v3d_freq_min=500vpred=8592hdmi_force_cec_address:0=65535hdmi_force_cec_address:1=65535

Code:

$ sudo apt updateHit:1 http://deb.debian.org/debian bookworm InReleaseHit:2 http://deb.debian.org/debian-security bookworm-security InReleaseHit:3 http://deb.debian.org/debian bookworm-updates InRelease   Hit:4 http://archive.raspberrypi.com/debian bookworm InReleaseReading package lists... DoneBuilding dependency tree... DoneReading state information... DoneAll packages are up to date.