Differences

This shows you the differences between two versions of the page.

--- infra:operations:proxmox-maintenance [2026/05/27 16:36] – Revise Proxmox maintenance scope for moved services atluxity_idp.hackeriet.no
+++ infra:operations:proxmox-maintenance [2026/06/01 17:32] (current) – Note LibreNMS alerting follow-up for klynge001 atluxity_idp.hackeriet.no
@@ Line 1: / Line 1: @@
 ====== Proxmox maintenance ======
-This is a runbook for planned maintenance on the Hackeriet Proxmox hosts in [[infra:clusters:klynge001|klynge001]]. It is documentation and procedure, not inventory. Use NetBox for canonical device, IP, cabling, and VM placement data, but verify live Proxmox state before deciding maintenance impact.
+This page documents maintenance for the Hackeriet Proxmox hosts in [[infra:clusters:klynge001|klynge001]]. It is a runbook and maintenance log.
 ===== Current scope =====
-Planned hosts:
+Hosts currently covered by this procedure:
   * [[infra:hosts:host006|host006]]
   * [[infra:hosts:host007|host007]]
-Current goals:
-  * Bring host006 and host007 up to date.
+===== Last maintenance: 2026-05-31 =====
-  * Review failed services and storage health.
-  * Keep the cluster healthy while working one host at a time.
-  * Avoid guest-level changes unless needed for recovery.
-This page covers Proxmox host maintenance. Service hosts that have moved away from this cluster should be checked as external service reachability, not treated as direct reboot impact from host006 or host007.
+Scope: host006 and host007 in the ''klynge001'' Proxmox cluster.
-===== Announcement draft =====
+Actions performed:
-Planned Proxmox maintenance for Hackeriet
+  * Removed obsolete local backup archives on host006 for moved/stopped service VMs.
+  * Updated the scheduled Proxmox backup job to exclude moved/stopped VMs:
+    * ''105'' / ''blade''
+    * ''510'' / ''ingress''
+    * ''511'' / ''app-01''
+  * Upgraded host006 and host007 to Proxmox VE 8.4.19.
+  * Rebooted host006 and host007 one at a time to activate kernel 6.8.12-28-pve.
+  * Temporarily adjusted expected votes during single-node reboot windows so the remaining node stayed quorate.
+  * Verified storage, package state, cluster quorum, guest state, and basic guest reachability after reboots.
-I plan to do maintenance on the Proxmox hosts host006 and host007 in the klynge001 cluster one of the next days.
+Final state after maintenance:
-Scope:
+  * host006: Proxmox VE 8.4.19, kernel 6.8.12-28-pve.
+  * host007: Proxmox VE 8.4.19, kernel 6.8.12-28-pve.
+  * Cluster: 2 nodes, expected votes 2, quorate.
+  * No pending package upgrades were listed on either node.
+  * host006 root filesystem usage was about 72% after cleanup and updates.
+  * host007 root filesystem usage was about 19% after updates.
-  * OS and Proxmox package updates
+Follow-up actions completed after the maintenance:
-  * storage and backup health checks
-  * failed service review
-  * possible host reboots if required
-Expected impact:
+  * Retired Munin on host006 and host007 by disabling and stopping ''munin-node.service''.
+  * Cleared stale ''lock: migrate'' locks on moved/stopped VMs.
+  * Disabled autostart for moved/stopped VMs:
+    * ''105'' / ''blade''
+    * ''510'' / ''ingress''
+    * ''511'' / ''app-01''
+    * ''601'' / ''idp1''
+  * Verified that ''systemctl --failed'' was clear on both nodes after Munin retirement.
-  * VMs still running on host006 or host007 may be briefly unavailable if their host is rebooted.
+===== What we learned =====
-  * Public services such as wiki, IDP, NetBox, LibreNMS, app-01, and ingress will be checked before and after maintenance, but are not expected to be affected by these Proxmox reboots based on current observed state.
-  * I will avoid guest-level changes unless needed for recovery.
-  * I will work on one host at a time and check cluster health between steps.
-===== Current service placement notes =====
+  * The cluster is currently operating as a two-node cluster. During a single-node reboot, the remaining node can temporarily lose quorum unless expected votes is adjusted when either other node is rebooted.
+  * Expected votes returned to 2 after both nodes were back and joined.
+  * ''munin-node.service'' had been failing for months on both host006 and host007. It is now intentionally retired on these hosts.
+  * Moved/stopped service VMs may still carry stale Proxmox migration locks and ''onboot: 1'' from before migration. Clear stale locks only after confirming there is no active migration task.
+  * host007 emitted GRUB/LVM warnings about a missing physical volume name ''pv1'' during update-grub, but rebooted successfully on the new kernel. The active LVM metadata still contains an internal ''pv1'' label.
+  * host006 emitted a GRUB warning that the removable EFI fallback path is not updated automatically. The explicit Proxmox EFI boot entry worked and the host rebooted successfully.
-Observed on 2026-05-27:
+===== Maintenance procedure =====
-  * ''ingress'' is stopped in Proxmox on host006, while [[infra:hosts:ingress|ingress.hackeriet.no]] is reachable elsewhere.
+Work one host at a time. Do not reboot both host006 and host007 at once.
-  * ''app-01'' is stopped in Proxmox on host006, while [[infra:hosts:app-01|app-01.hackeriet.no]] is reachable elsewhere.
-  * ''blade'' is stopped in Proxmox on host007, while blade.hackeriet.no and public services hosted there are reachable elsewhere.
-  * ''idp1'' is stopped in Proxmox on host007, while idp.hackeriet.no is reachable.
-Current implication:
+Before making changes on either host:
-  * Do not list moved services as expected reboot impact from host006 or host007.
+<code>
-  * Use live ''qm list'' output, not stale inventory records, to decide what a Proxmox reboot will affect.
+hostname -f
-  * Keep public service checks as regression checks, because DNS, routing, or documentation can still be wrong.
+pveversion -v
+uname -r
+pvecm status
+systemctl --failed --no-pager
+pvesm status
+df -h -x tmpfs -x devtmpfs
+qm list
+cat /etc/pve/jobs.cfg
+apt update
+apt list --upgradable
+test -f /var/run/reboot-required && cat /var/run/reboot-required || true
+</code>
-NetBox still appears to contain stale VM placement for some moved systems. Treat NetBox as inventory to clean up later, not as the final maintenance-impact source for those systems.
+For host006, also check local storage pressure:
-===== Current Proxmox impact =====
+<code>
+du -sh /var/lib/vz/dump /var/lib/vz/template/iso /var/log /var/cache/apt /root/proxmox-templates /var/lib/fail2ban
+</code>
-Before maintenance, refresh this with live ''qm list'' on each node.
+Update flow for each host:
-Observed on 2026-05-27, host006 running guests included:
+  - Confirm cluster health with ''pvecm status''.
+  - Confirm storage health with ''pvesm status'' and ''df -h''.
+  - Review running guests with ''qm list''.
+  - Review failed units with ''systemctl --failed --no-pager''.
+  - Simulate package changes if the update set is large or risky.
+  - Apply updates only after reviewing the package set.
+  - Reboot only if required or clearly useful, such as after a kernel update.
+  - After reboot, wait for the node to return and confirm cluster health before touching the next host.
-  * ''matrix''
+Suggested commands after review:
-  * ''aleksei''
-  * ''login''
-  * ''jakkn''
-  * ''m''
-  * ''sub-planar''
-  * ''emma''
-  * ''foxboron''
-  * ''meshtastic-br''
-Observed on 2026-05-27, host007 running guests included:
+<code>
+apt-get -s full-upgrade
+apt full-upgrade
+</code>
-  * ''jakkn2''
+===== Quorum during reboot =====
-  * ''fr30n''
-  * ''bergh''
-  * ''haus-forum''
-  * ''haus''
-  * ''hyperboria''
-  * ''nux''
-  * ''chat''
-Stopped legacy or moved VMs should remain untouched unless the maintenance task explicitly includes cleanup or migration review.
+When one node is rebooted, the remaining node may temporarily lose quorum. If that happens during planned maintenance, set expected votes to 1 on the remaining node:
-===== DNS and service guardrails =====
-Live DNS checks on 2026-05-23 showed that hackeriet.no has two authoritative nameservers:
-  * ns0.hackeriet.no - Hackeriet hosted, resolves to blade at 185.35.202.202 and 2a02:ed06::202
-  * ns.hyp.net - external nameserver, resolves to 194.63.248.53 and 2a01:5b40:0:248::53
-Both authoritative nameservers served the same SOA serial when checked. DNS resolution should survive a short outage of ns0 because ns.hyp.net is external and synced. Do not treat this as service redundancy.
-Before rebooting anything, check DNS redundancy:
 <code>
-dig @ns0.hackeriet.no SOA hackeriet.no
+pvecm expected 1
-dig @ns.hyp.net SOA hackeriet.no
+pvecm status
 </code>
-The SOA serial should match.
+After the rebooted node rejoins, confirm the cluster has returned to two nodes and expected votes 2:
-Check public services as guardrails, not as direct Proxmox dependency claims:
 <code>
-curl -I https://hackeriet.no/
+pvecm status
-curl -I https://wiki.hackeriet.no/
-curl -I https://idp.hackeriet.no/
-curl -I https://ip.hackeriet.no/
-curl -I https://nms.hackeriet.no/
 </code>
-===== Pre-maintenance checks =====
+Do not use this as an incident workaround without understanding which node has the correct cluster state.
-Run on both host006 and host007 before making changes:
+===== Post-host checks =====
+After each host update or reboot:
 <code>
 hostname -f
-pveversion -v
+pveversion
+uname -r
 pvecm status
 systemctl --failed --no-pager
 pvesm status
-df -h
+df -h -x tmpfs -x devtmpfs
 qm list
-cat /etc/pve/jobs.cfg
-apt update
 apt list --upgradable
-test -f /var/run/reboot-required && cat /var/run/reboot-required || true
 </code>
-On host006, also check local storage pressure:
+Check guests affected by the touched host:
 <code>
-du -sh /var/lib/vz/dump /var/lib/vz/template/iso /var/log /var/cache/apt /root/proxmox-templates /var/lib/fail2ban
+qm status <vmid>
+qm agent <vmid> ping
+qm guest cmd <vmid> network-get-interfaces
+qm config <vmid> | sed -n '/^ipconfig/p;/^net/p'
+ping -c 2 <ip>
+nc -vz -w3 <ip> 22
 </code>
-Host006 storage and backup context is documented at [[infra:operations:proxmox-backups|Proxmox backups]].
+Interpretation:
-===== Maintenance procedure =====
+  * ''qm status running'' means the hypervisor sees the VM running.
+  * ''qm agent <vmid> ping'' means the guest OS and QEMU guest agent are responsive.
+  * ''ping'' means basic network path works, if ICMP is allowed.
+  * ''nc -vz -w3 <ip> 22'' means SSH is listening, if SSH is expected for that guest.
+  * Lack of ping or SSH does not always mean the guest is broken. Investigate only failures that contradict expectations for that guest.
-Work one host at a time. Do not reboot both host006 and host007 at once.
+===== Monitoring / LibreNMS =====
-Suggested order:
+As of 2026-06-01, host006 and host007 are monitored in LibreNMS as Proxmox hypervisors. This replaces the old Munin host monitoring for these nodes.
-  - Start with host006 if the main concern is storage and backup health.
+LibreNMS records:
-  - Start with host007 if its running guests need to be handled first.
-For each host:
+  * host006.hackeriet.no: LibreNMS device ID 25.
+  * host007.hackeriet.no: LibreNMS device ID 26.
+  * Both devices have OS detected as ''proxmox''.
+  * Both devices have the LibreNMS ''proxmox'' application enabled with app instance ''klynge001''.
-  - Confirm cluster state with ''pvecm status''.
+Host-side setup:
-  - Confirm storage state with ''pvesm status'' and ''df -h''.
-  - Review running guests with ''qm list''.
-  - Confirm no unexpected critical service VM is running on the host before rebooting.
-  - Review failed units with ''systemctl --failed --no-pager''.
-  - Run ''apt update''.
-  - Review ''apt list --upgradable''.
-  - Apply updates only after reviewing the package set.
-  - Reboot only if required or clearly useful.
-  - After reboot, wait for the node to return and confirm cluster health before touching the next host.
-Suggested update command, after review:
+  * ''snmpd'' is installed, enabled, and running on both hosts.
+  * SNMP listens only on the management IP of each host:
+    * host006: ''10.10.50.26:161/udp''
+    * host007: ''10.10.50.27:161/udp''
+  * SNMP uses SNMPv3 authPriv with the security name ''librenms_klynge001''.
+  * Credentials are stored in Hackerpass at ''infrastructure/librenms-klynge001-snmpv3''. Do not put the credential values in the wiki or NetBox.
+  * The Proxmox LibreNMS agent-local script is installed as ''/usr/local/libexec/librenms-proxmox''.
+  * SNMP exposes Proxmox VM traffic through:
 <code>
-apt full-upgrade
+extend proxmox /usr/bin/sudo /usr/local/libexec/librenms-proxmox
 </code>
-Do not change guest VM configuration as part of host maintenance unless needed for recovery.
+  * ''/etc/sudoers.d/librenms-proxmox'' allows the ''Debian-snmp'' user to run only that script via sudo.
-===== Post-host checks =====
+Firewall setup:
-After each host update or reboot:
+  * The Proxmox cluster firewall allows SNMP only from app-01 / LibreNMS source address ''10.10.50.51'' to the ''klynge001'' IP set.
+  * The firewall rule is in ''/etc/pve/firewall/cluster.fw'':
 <code>
-hostname -f
+IN ACCEPT -source 10.10.50.51/32 -dest +klynge001 -p udp -dport 161 -log nolog # LibreNMS SNMP polling from app-01
-pveversion -v
-pvecm status
-systemctl --failed --no-pager
-pvesm status
-df -h
-qm list
 </code>
-Check public services as regression checks:
+LibreNMS setup:
+  * ''enable_proxmox'' is set to ''true''.
+  * The ''unix-agent'' poller module is disabled per device for host006 and host007, because this setup uses SNMP extend instead of the LibreNMS unix-agent on port 6556.
+  * No LibreNMS alert rules existed when this monitoring was added. Host metrics and Proxmox application polling are live, but alert policy still needs to be defined.
+Useful verification commands:
 <code>
-curl -I https://hackeriet.no/
+# On each Proxmox host
-curl -I https://wiki.hackeriet.no/
+systemctl is-active snmpd
-curl -I https://idp.hackeriet.no/
+ss -lunp | grep ':161'
-curl -I https://ip.hackeriet.no/
+sudo -u Debian-snmp sudo /usr/local/libexec/librenms-proxmox
-curl -I https://nms.hackeriet.no/
+# From the LibreNMS container on app-01
+snmpwalk -v3 -l authPriv -u librenms_klynge001 -a SHA -A '<auth password>' -x AES -X '<privacy password>' host006.hackeriet.no SNMPv2-MIB::sysName.0
+snmpget -v3 -l authPriv -u librenms_klynge001 -a SHA -A '<auth password>' -x AES -X '<privacy password>' -Oqv host006.hackeriet.no .1.3.6.1.4.1.8072.1.3.2.3.1.2.7.112.114.111.120.109.111.120
+lnms device:poll -m applications host006.hackeriet.no
 </code>
-Check actual services, not only DNS, when the relevant host has been touched.
+===== Follow-up items =====
-===== Known caveats =====
+  * Define LibreNMS alert rules and notification routing for klynge001 host monitoring.
+  * Investigate the host007 GRUB/LVM ''pv1'' warning.
+  * Investigate the host006 EFI fallback warning.
+  * Review whether the two-node quorum procedure should be part of the standard klynge001 maintenance runbook.
+  * Keep NetBox VM placement aligned with the current reality for moved service VMs.
+  * If Munin package cleanup is desired later, remove it in a separate low-risk cleanup task.
-  * host006 has about 1 TB physical storage, but Proxmox local storage is on the root filesystem. The root filesystem was previously close to full, and local backups under ''/var/lib/vz/dump'' were the main pressure point.
+===== Other notes =====
-  * During host006 maintenance, avoid casual LVM reshaping. It can put VM disks at risk and should only be done with a maintenance window and recovery plan.
-  * Certificate automation for internal Proxmox hostnames is documented at [[infra:operations:proxmox-acme-dns|Proxmox ACME DNS automation]].
-  * NetBox placement may be stale for moved service VMs. Verify live Proxmox state with ''qm list''.
-===== Safety notes =====
+  * Host006 storage and backup context is documented at [[infra:operations:proxmox-backups|Proxmox backups]].
+  * Certificate automation for internal Proxmox hostnames is documented at [[infra:operations:proxmox-acme-dns|Proxmox ACME DNS automation]].
-  * There is no plan, and we should avoid, to touch guest VMs unless required for recovery.
-  * Do not change DNS during the maintenance window unless DNS itself is the incident.
-  * There is no plan, and we should avoid, delete backups or ISOs without understanding what they are.
-  * Keep notes locally while working; wiki, NetBox, IDP, and public services may be affected depending on which host is down.