Differences

This shows you the differences between two versions of the page.

--- infra:operations:proxmox-maintenance [2026/05/23 15:12] – atluxity_idp.hackeriet.no
+++ infra:operations:proxmox-maintenance [2026/06/01 17:32] (current) – Note LibreNMS alerting follow-up for klynge001 atluxity_idp.hackeriet.no
@@ Line 1: / Line 1: @@
 ====== Proxmox maintenance ======
-This is a runbook for planned maintenance on the Hackeriet Proxmox hosts in [[infra:clusters:klynge001|klynge001]]. It is documentation and procedure, not inventory. Use NetBox for canonical device, IP, cabling, and VM placement data.
+This page documents maintenance for the Hackeriet Proxmox hosts in [[infra:clusters:klynge001|klynge001]]. It is a runbook and maintenance log.
 ===== Current scope =====
-Planned hosts:
+Hosts currently covered by this procedure:
   * [[infra:hosts:host006|host006]]
   * [[infra:hosts:host007|host007]]
-Current goals:
-  * Bring host006 and host007 up to date.
+===== Last maintenance: 2026-05-31 =====
-  * Review failed services and storage health.
-  * Keep the cluster healthy while working one host at a time.
-  * Avoid guest-level changes unless needed for recovery.
-===== Announcement draft =====
+Scope: host006 and host007 in the ''klynge001'' Proxmox cluster.
-Planned Proxmox maintenance for Hackeriet
+Actions performed:
-I plan to do maintenance on the Proxmox hosts host006 and host007 in the klynge001 cluster one of the next days.
+  * Removed obsolete local backup archives on host006 for moved/stopped service VMs.
+  * Updated the scheduled Proxmox backup job to exclude moved/stopped VMs:
+    * ''105'' / ''blade''
+    * ''510'' / ''ingress''
+    * ''511'' / ''app-01''
+  * Upgraded host006 and host007 to Proxmox VE 8.4.19.
+  * Rebooted host006 and host007 one at a time to activate kernel 6.8.12-28-pve.
+  * Temporarily adjusted expected votes during single-node reboot windows so the remaining node stayed quorate.
+  * Verified storage, package state, cluster quorum, guest state, and basic guest reachability after reboots.
-Scope:
+Final state after maintenance:
-  * OS and Proxmox package updates
+  * host006: Proxmox VE 8.4.19, kernel 6.8.12-28-pve.
-  * storage and backup health checks
+  * host007: Proxmox VE 8.4.19, kernel 6.8.12-28-pve.
-  * failed service review
+  * Cluster: 2 nodes, expected votes 2, quorate.
-  * possible host reboots if required
+  * No pending package upgrades were listed on either node.
+  * host006 root filesystem usage was about 72% after cleanup and updates.
+  * host007 root filesystem usage was about 19% after updates.
-Expected impact:
+Follow-up actions completed after the maintenance:
-  * Some VMs and services may be briefly unavailable.
+  * Retired Munin on host006 and host007 by disabling and stopping ''munin-node.service''.
-  * I will avoid guest-level changes unless needed for recovery.
+  * Cleared stale ''lock: migrate'' locks on moved/stopped VMs.
-  * I will work on one host at a time and check cluster health between steps.
+  * Disabled autostart for moved/stopped VMs:
+    * ''105'' / ''blade''
+    * ''510'' / ''ingress''
+    * ''511'' / ''app-01''
+    * ''601'' / ''idp1''
+  * Verified that ''systemctl --failed'' was clear on both nodes after Munin retirement.
-===== DNS and service risk =====
+===== What we learned =====
-Live DNS checks on 2026-05-23 showed that hackeriet.no has two authoritative nameservers:
+  * The cluster is currently operating as a two-node cluster. During a single-node reboot, the remaining node can temporarily lose quorum unless expected votes is adjusted when either other node is rebooted.
+  * Expected votes returned to 2 after both nodes were back and joined.
+  * ''munin-node.service'' had been failing for months on both host006 and host007. It is now intentionally retired on these hosts.
+  * Moved/stopped service VMs may still carry stale Proxmox migration locks and ''onboot: 1'' from before migration. Clear stale locks only after confirming there is no active migration task.
+  * host007 emitted GRUB/LVM warnings about a missing physical volume name ''pv1'' during update-grub, but rebooted successfully on the new kernel. The active LVM metadata still contains an internal ''pv1'' label.
+  * host006 emitted a GRUB warning that the removable EFI fallback path is not updated automatically. The explicit Proxmox EFI boot entry worked and the host rebooted successfully.
-  * ns0.hackeriet.no - Hackeriet hosted, resolves to blade at 185.35.202.202 and 2a02:ed06::202
+===== Maintenance procedure =====
-  * ns.hyp.net - external nameserver, resolves to 194.63.248.53 and 2a01:5b40:0:248::53
-Both authoritative nameservers served the same SOA serial when checked. DNS resolution should survive a short outage of ns0 because ns.hyp.net is external and synced. Do not treat this as service redundancy.
+Work one host at a time. Do not reboot both host006 and host007 at once.
-Important service dependencies observed:
+Before making changes on either host:
-  * hackeriet.no and www.hackeriet.no point to blade.
+<code>
-  * wiki.hackeriet.no points to blade.
+hostname -f
-  * hackeriet.no MX points to blade.
+pveversion -v
-  * ns0.hackeriet.no points to blade.
+uname -r
-  * ip.hackeriet.no and nms.hackeriet.no point through ingress.
+pvecm status
-  * blade is currently documented as a VM on host007.
+systemctl --failed --no-pager
-  * ingress is VM 510 on host006.
+pvesm status
-  * idp1 was observed on host007.
+df -h -x tmpfs -x devtmpfs
+qm list
+cat /etc/pve/jobs.cfg
+apt update
+apt list --upgradable
+test -f /var/run/reboot-required && cat /var/run/reboot-required || true
+</code>
-Maintenance implications:
+For host006, also check local storage pressure:
-  * Rebooting host007 can affect blade, public web, wiki, mail target, ns0, and likely IDP.
+<code>
-  * Rebooting host006 can affect ingress-routed services such as NetBox and LibreNMS.
+du -sh /var/lib/vz/dump /var/lib/vz/template/iso /var/log /var/cache/apt /root/proxmox-templates /var/lib/fail2ban
-  * Avoid DNS zone edits during the maintenance window.
+</code>
-  * Keep this runbook available locally before starting, because wiki and NetBox may be affected.
-===== Pre-maintenance checks =====
+Update flow for each host:
-Run on both host006 and host007 before making changes:
+  - Confirm cluster health with ''pvecm status''.
+  - Confirm storage health with ''pvesm status'' and ''df -h''.
+  - Review running guests with ''qm list''.
+  - Review failed units with ''systemctl --failed --no-pager''.
+  - Simulate package changes if the update set is large or risky.
+  - Apply updates only after reviewing the package set.
+  - Reboot only if required or clearly useful, such as after a kernel update.
+  - After reboot, wait for the node to return and confirm cluster health before touching the next host.
-  * hostname -f
+Suggested commands after review:
-  * pveversion -v
-  * pvecm status
-  * systemctl --failed --no-pager
-  * pvesm status
-  * df -h
-  * qm list
-  * cat /etc/pve/jobs.cfg
-  * apt update
-  * apt list --upgradable
-  * test -f /var/run/reboot-required && cat /var/run/reboot-required || true
-On host006, also check local storage pressure:
+<code>
+apt-get -s full-upgrade
+apt full-upgrade
+</code>
-  * du -sh /var/lib/vz/dump /var/lib/vz/template/iso /var/log /var/cache/apt /root/proxmox-templates /var/lib/fail2ban
+===== Quorum during reboot =====
-Before rebooting anything, check DNS redundancy:
+When one node is rebooted, the remaining node may temporarily lose quorum. If that happens during planned maintenance, set expected votes to 1 on the remaining node:
-  * dig @ns0.hackeriet.no SOA hackeriet.no
+<code>
-  * dig @ns.hyp.net SOA hackeriet.no
+pvecm expected 1
+pvecm status
+</code>
-The SOA serial should match.
+After the rebooted node rejoins, confirm the cluster has returned to two nodes and expected votes 2:
-===== Maintenance procedure =====
+<code>
+pvecm status
+</code>
-Work one host at a time. Do not reboot both host006 and host007 at once.
+Do not use this as an incident workaround without understanding which node has the correct cluster state.
-Suggested order:
+===== Post-host checks =====
-  - Start with host006 if the main concern is storage and backup health.
+After each host update or reboot:
-  - Start with host007 if host006-hosted ingress services must stay stable first.
-For each host:
+<code>
+hostname -f
+pveversion
+uname -r
+pvecm status
+systemctl --failed --no-pager
+pvesm status
+df -h -x tmpfs -x devtmpfs
+qm list
+apt list --upgradable
+</code>
-  - Confirm cluster state with pvecm status.
+Check guests affected by the touched host:
-  - Confirm storage state with pvesm status and df -h.
-  - Review failed units with systemctl --failed --no-pager.
-  - Run apt update.
-  - Review apt list --upgradable.
-  - Apply updates only after reviewing the package set.
-  - Reboot only if required or clearly useful.
-  - After reboot, wait for the node to return and confirm cluster health before touching the next host.
-Suggested update commands, after review:
+<code>
+qm status <vmid>
+qm agent <vmid> ping
+qm guest cmd <vmid> network-get-interfaces
+qm config <vmid> | sed -n '/^ipconfig/p;/^net/p'
+ping -c 2 <ip>
+nc -vz -w3 <ip> 22
+</code>
-  * apt full-upgrade
+Interpretation:
-Do not change guest VM configuration as part of host maintenance unless needed for recovery.
+  * ''qm status running'' means the hypervisor sees the VM running.
+  * ''qm agent <vmid> ping'' means the guest OS and QEMU guest agent are responsive.
+  * ''ping'' means basic network path works, if ICMP is allowed.
+  * ''nc -vz -w3 <ip> 22'' means SSH is listening, if SSH is expected for that guest.
+  * Lack of ping or SSH does not always mean the guest is broken. Investigate only failures that contradict expectations for that guest.
-===== Post-host checks =====
+===== Monitoring / LibreNMS =====
-After each host update or reboot:
+As of 2026-06-01, host006 and host007 are monitored in LibreNMS as Proxmox hypervisors. This replaces the old Munin host monitoring for these nodes.
+LibreNMS records:
+  * host006.hackeriet.no: LibreNMS device ID 25.
+  * host007.hackeriet.no: LibreNMS device ID 26.
+  * Both devices have OS detected as ''proxmox''.
+  * Both devices have the LibreNMS ''proxmox'' application enabled with app instance ''klynge001''.
+Host-side setup:
+  * ''snmpd'' is installed, enabled, and running on both hosts.
+  * SNMP listens only on the management IP of each host:
+    * host006: ''10.10.50.26:161/udp''
+    * host007: ''10.10.50.27:161/udp''
+  * SNMP uses SNMPv3 authPriv with the security name ''librenms_klynge001''.
+  * Credentials are stored in Hackerpass at ''infrastructure/librenms-klynge001-snmpv3''. Do not put the credential values in the wiki or NetBox.
+  * The Proxmox LibreNMS agent-local script is installed as ''/usr/local/libexec/librenms-proxmox''.
+  * SNMP exposes Proxmox VM traffic through:
+<code>
+extend proxmox /usr/bin/sudo /usr/local/libexec/librenms-proxmox
+</code>
+  * ''/etc/sudoers.d/librenms-proxmox'' allows the ''Debian-snmp'' user to run only that script via sudo.
+Firewall setup:
+  * The Proxmox cluster firewall allows SNMP only from app-01 / LibreNMS source address ''10.10.50.51'' to the ''klynge001'' IP set.
+  * The firewall rule is in ''/etc/pve/firewall/cluster.fw'':
-  * hostname -f
+<code>
-  * pveversion -v
+IN ACCEPT -source 10.10.50.51/32 -dest +klynge001 -p udp -dport 161 -log nolog # LibreNMS SNMP polling from app-01
-  * pvecm status
+</code>
-  * systemctl --failed --no-pager
-  * pvesm status
-  * df -h
-  * qm list
-Check DNS and key service names:
+LibreNMS setup:
-  * dig @ns0.hackeriet.no SOA hackeriet.no
+  * ''enable_proxmox'' is set to ''true''.
-  * dig @ns.hyp.net SOA hackeriet.no
+  * The ''unix-agent'' poller module is disabled per device for host006 and host007, because this setup uses SNMP extend instead of the LibreNMS unix-agent on port 6556.
-  * dig hackeriet.no A
+  * No LibreNMS alert rules existed when this monitoring was added. Host metrics and Proxmox application polling are live, but alert policy still needs to be defined.
-  * dig wiki.hackeriet.no A
-  * dig idp.hackeriet.no A
-  * dig ip.hackeriet.no A
-  * dig nms.hackeriet.no A
-Check actual services, not only DNS, when the relevant host has been touched.
+Useful verification commands:
-===== Host006 notes =====
+<code>
+# On each Proxmox host
+systemctl is-active snmpd
+ss -lunp | grep ':161'
+sudo -u Debian-snmp sudo /usr/local/libexec/librenms-proxmox
-host006 has about 1 TB physical storage, but Proxmox local storage is on the root filesystem. The root filesystem was previously close to full, and local backups under /var/lib/vz/dump were the main pressure point.
+# From the LibreNMS container on app-01
+snmpwalk -v3 -l authPriv -u librenms_klynge001 -a SHA -A '<auth password>' -x AES -X '<privacy password>' host006.hackeriet.no SNMPv2-MIB::sysName.0
+snmpget -v3 -l authPriv -u librenms_klynge001 -a SHA -A '<auth password>' -x AES -X '<privacy password>' -Oqv host006.hackeriet.no .1.3.6.1.4.1.8072.1.3.2.3.1.2.7.112.114.111.120.109.111.120
+lnms device:poll -m applications host006.hackeriet.no
+</code>
-Known cleanup/remediation context is documented at [[infra:operations:proxmox-backups|Proxmox backups]].
+===== Follow-up items =====
-During maintenance, avoid casual LVM reshaping. It can put VM disks at risk and should only be done with a maintenance window and recovery plan.
+  * Define LibreNMS alert rules and notification routing for klynge001 host monitoring.
+  * Investigate the host007 GRUB/LVM ''pv1'' warning.
+  * Investigate the host006 EFI fallback warning.
+  * Review whether the two-node quorum procedure should be part of the standard klynge001 maintenance runbook.
+  * Keep NetBox VM placement aligned with the current reality for moved service VMs.
+  * If Munin package cleanup is desired later, remove it in a separate low-risk cleanup task.
-===== Safety notes =====
+===== Other notes =====
-  * There is no plan, and we should avoid, to touch guest VMs unless required for recovery.
+  * Host006 storage and backup context is documented at [[infra:operations:proxmox-backups|Proxmox backups]].
-  * Do not change DNS during the maintenance window unless DNS itself is the incident.
+  * Certificate automation for internal Proxmox hostnames is documented at [[infra:operations:proxmox-acme-dns|Proxmox ACME DNS automation]].
-  * There is no plan, and we should avoid, delete backups or ISOs without understanding what they are.
-  * Keep notes locally while working; wiki, NetBox, IDP, and public services may be affected depending on which host is down.