This page documents planned maintenance for the Hackeriet Proxmox hosts in klynge001. It is a runbook and maintenance log, not inventory. Use NetBox for canonical device, IP, cabling, and VM placement data, but verify live Proxmox state before deciding maintenance impact.

Hosts currently covered by this procedure:

• host006
• host007

Moved or stopped service VMs should not be treated as direct reboot impact unless live Proxmox state says they are running on one of these hosts.

Known moved/stopped service VMs observed during the 2026-05-31 maintenance:

• 105 / blade was stopped on host007.
• 510 / ingress was stopped on host006.
• 511 / app-01 was stopped on host006.

Scope: host006 and host007 in the klynge001 Proxmox cluster.

Actions performed:

• Removed obsolete local backup archives on host006 for moved/stopped service VMs.
• Updated the scheduled Proxmox backup job to exclude moved/stopped VMs:
  • 105 / blade
  • 510 / ingress
  • 511 / app-01
• Upgraded host006 and host007 to Proxmox VE 8.4.19.
• Rebooted host006 and host007 one at a time to activate kernel 6.8.12-28-pve.
• Temporarily adjusted expected votes during single-node reboot windows so the remaining node stayed quorate.
• Verified storage, package state, cluster quorum, guest state, and basic guest reachability after reboots.

Final state after maintenance:

• host006: Proxmox VE 8.4.19, kernel 6.8.12-28-pve.
• host007: Proxmox VE 8.4.19, kernel 6.8.12-28-pve.
• Cluster: 2 nodes, expected votes 2, quorate.
• No pending package upgrades were listed on either node.
• Expected running guests were back online after the host reboots.
• host006 root filesystem usage was about 72% after cleanup and updates.
• host007 root filesystem usage was about 19% after updates.

• The cluster is currently operating as a two-node cluster. During a single-node reboot, the remaining node can temporarily lose quorum unless expected votes is adjusted.
• Expected votes returned to 2 after both nodes were back and joined.
• Some guests did not immediately appear as running right after host reboot, but came up shortly afterward. Wait briefly before manual intervention.
• Lack of ICMP is not by itself a failure. During the 2026-05-31 checks, haus-forum did not answer ping but SSH was open.
• munin-node.service is failed on both host006 and host007. This remains unresolved.
• host007 emitted GRUB/LVM warnings about a missing physical volume name pv1 during update-grub, but rebooted successfully on the new kernel.
• host006 emitted a GRUB warning that the removable EFI fallback path is not updated automatically. The explicit Proxmox EFI boot entry worked and the host rebooted successfully.

Work one host at a time. Do not reboot both host006 and host007 at once.

Before making changes on either host:

hostname -f
pveversion -v
uname -r
pvecm status
systemctl --failed --no-pager
pvesm status
df -h -x tmpfs -x devtmpfs
qm list
cat /etc/pve/jobs.cfg
apt update
apt list --upgradable
test -f /var/run/reboot-required && cat /var/run/reboot-required || true

For host006, also check local storage pressure:

du -sh /var/lib/vz/dump /var/lib/vz/template/iso /var/log /var/cache/apt /root/proxmox-templates /var/lib/fail2ban

Update flow for each host:

1. Confirm cluster health with pvecm status.
2. Confirm storage health with pvesm status and df -h.
3. Review running guests with qm list.
4. Review failed units with systemctl –failed –no-pager.
5. Simulate package changes if the update set is large or risky.
6. Apply updates only after reviewing the package set.
7. Reboot only if required or clearly useful, such as after a kernel update.
8. After reboot, wait for the node to return and confirm cluster health before touching the next host.

Suggested commands after review:

apt-get -s full-upgrade
apt full-upgrade

When one node is rebooted, the remaining node may temporarily lose quorum. If that happens during planned maintenance, set expected votes to 1 on the remaining node:

pvecm expected 1
pvecm status

After the rebooted node rejoins, confirm the cluster has returned to two nodes and expected votes 2:

pvecm status

Do not use this as an incident workaround without understanding which node has the correct cluster state.

After each host update or reboot:

hostname -f
pveversion
uname -r
pvecm status
systemctl --failed --no-pager
pvesm status
df -h -x tmpfs -x devtmpfs
qm list
apt list --upgradable

Check guests affected by the touched host:

qm status <vmid>
qm agent <vmid> ping
qm guest cmd <vmid> network-get-interfaces
qm config <vmid> | sed -n '/^ipconfig/p;/^net/p'
ping -c 2 <ip>
nc -vz -w3 <ip> 22

Interpretation:

• qm status running means the hypervisor sees the VM running.
• qm agent <vmid> ping means the guest OS and QEMU guest agent are responsive.
• ping means basic network path works, if ICMP is allowed.
• nc -vz -w3 <ip> 22 means SSH is listening, if SSH is expected for that guest.
• Lack of ping or SSH does not always mean the guest is broken. Investigate only failures that contradict expectations for that guest.

• Fix or intentionally retire Munin on host006 and host007.
• Review VM autostart settings for guests expected to return after host reboot.
• Investigate the host007 GRUB/LVM pv1 warning.
• Investigate the host006 EFI fallback warning.
• Review whether the two-node quorum procedure should be part of the standard klynge001 maintenance runbook.
• Keep NetBox VM placement aligned with the current reality for moved service VMs.

• Avoid guest-level changes during host maintenance unless needed for recovery.
• Do not change DNS during the maintenance window unless DNS itself is the incident.
• Do not delete backups or ISOs without understanding what they are.
• Keep local notes while working; wiki, NetBox, IDP, and public services may be affected depending on which host is down.
• Host006 storage and backup context is documented at Proxmox backups.
• Certificate automation for internal Proxmox hostnames is documented at Proxmox ACME DNS automation.