This is a runbook for planned maintenance on the Hackeriet Proxmox hosts in klynge001. It is documentation and procedure, not inventory. Use NetBox for canonical device, IP, cabling, and VM placement data, but verify live Proxmox state before deciding maintenance impact.

Planned hosts:

• host006
• host007

Current goals:

• Bring host006 and host007 up to date.
• Review failed services and storage health.
• Keep the cluster healthy while working one host at a time.
• Avoid guest-level changes unless needed for recovery.

This page covers Proxmox host maintenance. Service hosts that have moved away from this cluster should be checked as external service reachability, not treated as direct reboot impact from host006 or host007.

Planned Proxmox maintenance for Hackeriet

I plan to do maintenance on the Proxmox hosts host006 and host007 in the klynge001 cluster one of the next days.

Scope:

• OS and Proxmox package updates
• storage and backup health checks
• failed service review
• possible host reboots if required

Expected impact:

• VMs still running on host006 or host007 may be briefly unavailable if their host is rebooted.
• VMs affected by a reboot or migration will be checked from Proxmox and, when possible, by guest-agent and network probes.
• I will avoid guest-level changes unless needed for recovery.
• I will work on one host at a time and check cluster health between steps.

Observed on 2026-05-27:

• ingress is stopped in Proxmox on host006, while ingress.hackeriet.no is reachable elsewhere.
• app-01 is stopped in Proxmox on host006, while app-01.hackeriet.no is reachable elsewhere.
• blade is stopped in Proxmox on host007, while blade.hackeriet.no and public services hosted there are reachable elsewhere.
• idp1 is stopped in Proxmox on host007, while idp.hackeriet.no is reachable.

Current implication:

• Do not list moved services as expected reboot impact from host006 or host007.
• Use live qm list output, not stale inventory records, to decide what a Proxmox reboot will affect.
• Keep public service checks as regression checks, because DNS, routing, or documentation can still be wrong.

NetBox still appears to contain stale VM placement for some moved systems. Treat NetBox as inventory to clean up later, not as the final maintenance-impact source for those systems.

Before maintenance, refresh this with live qm list on each node.

Observed on 2026-05-27, host006 running guests included:

• matrix
• aleksei
• login
• jakkn
• m
• sub-planar
• emma
• foxboron
• meshtastic-br

Observed on 2026-05-27, host007 running guests included:

• jakkn2
• fr30n
• bergh
• haus-forum
• haus
• hyperboria
• nux
• chat

Stopped legacy or moved VMs should remain untouched unless the maintenance task explicitly includes cleanup or migration review.

For host006/host007 maintenance, check guests that are actually expected to be affected by the Proxmox work. Do not use unrelated public service checks as the default success criteria.

Use this order:

1. Confirm the VM is expected to run with qm list.
2. Confirm Proxmox sees the VM running with qm status <vmid>.
3. If the QEMU guest agent is configured, check it with qm agent <vmid> ping.
4. If the guest agent responds, discover guest IPs with qm guest cmd <vmid> network-get-interfaces.
5. If the guest uses cloud-init IP config, check qm config <vmid> for ipconfig0.
6. If needed, fall back to NetBox or neighbor/ARP information for the guest IP.
7. Probe network reachability only when it is expected for that guest.

Useful commands:

qm list
qm status <vmid>
qm agent <vmid> ping
qm guest cmd <vmid> network-get-interfaces
qm config <vmid> | sed -n '/^ipconfig/p;/^net/p'
ping -c 2 <ip>
nc -vz -w3 <ip> 22

Interpretation:

• qm status running means the hypervisor sees the VM running.
• qm agent <vmid> ping means the guest OS and guest agent are responsive.
• ping means basic network path works, if ICMP is allowed.
• nc -vz -w3 <ip> 22 means SSH is listening, if SSH is expected for that guest.
• Lack of ping or SSH does not always mean the guest is broken; only investigate failures that contradict expectations for that guest.

Observed on 2026-05-31, guest agent responded for:

• matrix
• login
• jakkn
• m
• sub-planar
• emma
• foxboron
• meshtastic-br
• jakkn2
• bergh
• haus-forum
• hyperboria

Observed on 2026-05-31, guest agent did not respond for:

• aleksei
• fr30n
• haus
• nux
• chat

DNS checks are only relevant if DNS, routing, authoritative nameservers, or hosts serving those services are in scope. They are not default host006/host007 Proxmox update checks.

Run on both host006 and host007 before making changes:

hostname -f
pveversion -v
pvecm status
systemctl --failed --no-pager
pvesm status
df -h
qm list
cat /etc/pve/jobs.cfg
apt update
apt list --upgradable
test -f /var/run/reboot-required && cat /var/run/reboot-required || true

On host006, also check local storage pressure:

du -sh /var/lib/vz/dump /var/lib/vz/template/iso /var/log /var/cache/apt /root/proxmox-templates /var/lib/fail2ban

Host006 storage and backup context is documented at Proxmox backups.

Work one host at a time. Do not reboot both host006 and host007 at once.

Suggested order:

1. Start with host006 if the main concern is storage and backup health.
2. Start with host007 if its running guests need to be handled first.

For each host:

1. Confirm cluster state with pvecm status.
2. Confirm storage state with pvesm status and df -h.
3. Review running guests with qm list.
4. Confirm no unexpected critical service VM is running on the host before rebooting.
5. Review failed units with systemctl –failed –no-pager.
6. Run apt update.
7. Review apt list –upgradable.
8. Apply updates only after reviewing the package set.
9. Reboot only if required or clearly useful.
10. After reboot, wait for the node to return and confirm cluster health before touching the next host.

Suggested update command, after review:

apt full-upgrade

Do not change guest VM configuration as part of host maintenance unless needed for recovery.

After each host update or reboot:

hostname -f
pveversion -v
pvecm status
systemctl --failed --no-pager
pvesm status
df -h
qm list

Check guests affected by the touched host:

qm list
qm status <vmid>
qm agent <vmid> ping
qm guest cmd <vmid> network-get-interfaces
qm config <vmid> | sed -n '/^ipconfig/p;/^net/p'
ping -c 2 <ip>
nc -vz -w3 <ip> 22

For migrated VMs, check both source and target:

qm list    # source: VM should be stopped or absent from the source host
qm list    # target: VM should be running on the target host

Check guest services only when the relevant guest has been touched and the expected service is known.

• host006 has about 1 TB physical storage, but Proxmox local storage is on the root filesystem. The root filesystem was previously close to full, and local backups under /var/lib/vz/dump were the main pressure point.
• During host006 maintenance, avoid casual LVM reshaping. It can put VM disks at risk and should only be done with a maintenance window and recovery plan.
• Certificate automation for internal Proxmox hostnames is documented at Proxmox ACME DNS automation.
• NetBox placement may be stale for moved service VMs. Verify live Proxmox state with qm list.

• There is no plan, and we should avoid, to touch guest VMs unless required for recovery.
• Do not change DNS during the maintenance window unless DNS itself is the incident.
• There is no plan, and we should avoid, delete backups or ISOs without understanding what they are.
• Keep notes locally while working; wiki, NetBox, IDP, and public services may be affected depending on which host is down.

Scope: host006 and host007 in the klynge001 Proxmox cluster.

Actions performed:

• Removed obsolete local backup archives on host006 for moved/stopped service VMs.
• Updated the scheduled Proxmox backup job to exclude moved/stopped VMs:
  • 105 / blade
  • 510 / ingress
  • 511 / app-01
• Upgraded host006 and host007 to Proxmox VE 8.4.19.
• Rebooted host006 and host007 one at a time to activate kernel 6.8.12-28-pve.
• Temporarily adjusted expected votes during single-node reboot windows so the remaining node stayed quorate.
• Verified storage, package state, cluster quorum, guest state, and basic guest reachability after reboots.

Final state after the maintenance:

• host006: Proxmox VE 8.4.19, running kernel 6.8.12-28-pve.
• host007: Proxmox VE 8.4.19, running kernel 6.8.12-28-pve.
• Cluster: 2 nodes, expected votes 2, quorate.
• No pending package upgrades were listed on either node after maintenance.
• Expected running guests were back online after the host reboots.
• host006 root filesystem pressure was reduced from the previous critical state to about 72% used after cleanup and updates.
• host007 root filesystem was about 19% used after updates.

Guest checks used:

• qm list to confirm expected running/stopped VM state.
• qm agent <vmid> ping where guest agent was available.
• qm guest cmd <vmid> network-get-interfaces where useful for IP discovery.
• ping and nc -z -w2 <ip> 22 for lightweight reachability checks where those probes matched expectations.

Operational notes learned:

• The cluster is currently operating as a two-node cluster. During planned single-node reboot windows, the remaining node can temporarily lose quorum unless expected votes is adjusted.
• Expected votes returned to 2 automatically after both nodes were back and joined.
• Some guests did not immediately appear as running right after host reboot, but came up shortly afterward. Wait briefly before manual intervention.
• Lack of ICMP is not by itself a failure. For example, haus-forum did not answer ping after reboot but SSH was open.
• munin-node.service is failed on both host006 and host007. This was pre-existing or at least unrelated to the package upgrade and remains unresolved.
• host007 emitted GRUB/LVM warnings about a missing physical volume name pv1 during update-grub, but rebooted successfully on the new kernel.
• host006 emitted a GRUB warning that the removable EFI fallback path is not updated automatically. The explicit Proxmox EFI boot entry worked and the host rebooted successfully.

Follow-up items:

• Fix or intentionally retire Munin on host006 and host007.
• Review VM autostart settings for guests expected to return after host reboot.
• Investigate the host007 GRUB/LVM pv1 warning.
• Investigate the host006 EFI fallback warning.
• Document the expected-votes procedure explicitly if this remains a two-node maintenance pattern.