Differences

This shows you the differences between two versions of the page.

--- infra:operations:proxmox-maintenance [2026/05/31 17:16] – Rewrite Proxmox maintenance page after 2026-05-31 maintenance atluxity_idp.hackeriet.no
+++ infra:operations:proxmox-maintenance [2026/06/01 17:32] (current) – Note LibreNMS alerting follow-up for klynge001 atluxity_idp.hackeriet.no
@@ Line 1: / Line 1: @@
 ====== Proxmox maintenance ======
-This page documents planned maintenance for the Hackeriet Proxmox hosts in [[infra:clusters:klynge001|klynge001]]. It is a runbook and maintenance log, not inventory. Use NetBox for canonical device, IP, cabling, and VM placement data, but verify live Proxmox state before deciding maintenance impact.
+This page documents maintenance for the Hackeriet Proxmox hosts in [[infra:clusters:klynge001|klynge001]]. It is a runbook and maintenance log.
 ===== Current scope =====
@@ Line 10: / Line 10: @@
   * [[infra:hosts:host007|host007]]
-Moved or stopped service VMs should not be treated as direct reboot impact unless live Proxmox state says they are running on one of these hosts.
-Known moved/stopped service VMs observed during the 2026-05-31 maintenance:
-  * ''105'' / ''blade'' was stopped on host007.
-  * ''510'' / ''ingress'' was stopped on host006.
-  * ''511'' / ''app-01'' was stopped on host006.
 ===== Last maintenance: 2026-05-31 =====
@@ Line 40: / Line 33: @@
   * Cluster: 2 nodes, expected votes 2, quorate.
   * No pending package upgrades were listed on either node.
-  * Expected running guests were back online after the host reboots.
   * host006 root filesystem usage was about 72% after cleanup and updates.
   * host007 root filesystem usage was about 19% after updates.
+Follow-up actions completed after the maintenance:
+  * Retired Munin on host006 and host007 by disabling and stopping ''munin-node.service''.
+  * Cleared stale ''lock: migrate'' locks on moved/stopped VMs.
+  * Disabled autostart for moved/stopped VMs:
+    * ''105'' / ''blade''
+    * ''510'' / ''ingress''
+    * ''511'' / ''app-01''
+    * ''601'' / ''idp1''
+  * Verified that ''systemctl --failed'' was clear on both nodes after Munin retirement.
 ===== What we learned =====
-  * The cluster is currently operating as a two-node cluster. During a single-node reboot, the remaining node can temporarily lose quorum unless expected votes is adjusted.
+  * The cluster is currently operating as a two-node cluster. During a single-node reboot, the remaining node can temporarily lose quorum unless expected votes is adjusted when either other node is rebooted.
   * Expected votes returned to 2 after both nodes were back and joined.
-  * Some guests did not immediately appear as running right after host reboot, but came up shortly afterward. Wait briefly before manual intervention.
+  * ''munin-node.service'' had been failing for months on both host006 and host007. It is now intentionally retired on these hosts.
-  * Lack of ICMP is not by itself a failure. During the 2026-05-31 checks, ''haus-forum'' did not answer ping but SSH was open.
+  * Moved/stopped service VMs may still carry stale Proxmox migration locks and ''onboot: 1'' from before migration. Clear stale locks only after confirming there is no active migration task.
-  * ''munin-node.service'' is failed on both host006 and host007. This remains unresolved.
+  * host007 emitted GRUB/LVM warnings about a missing physical volume name ''pv1'' during update-grub, but rebooted successfully on the new kernel. The active LVM metadata still contains an internal ''pv1'' label.
-  * host007 emitted GRUB/LVM warnings about a missing physical volume name ''pv1'' during update-grub, but rebooted successfully on the new kernel.
   * host006 emitted a GRUB warning that the removable EFI fallback path is not updated automatically. The explicit Proxmox EFI boot entry worked and the host rebooted successfully.
@@ Line 150: / Line 152: @@
   * ''nc -vz -w3 <ip> 22'' means SSH is listening, if SSH is expected for that guest.
   * Lack of ping or SSH does not always mean the guest is broken. Investigate only failures that contradict expectations for that guest.
+===== Monitoring / LibreNMS =====
+As of 2026-06-01, host006 and host007 are monitored in LibreNMS as Proxmox hypervisors. This replaces the old Munin host monitoring for these nodes.
+LibreNMS records:
+  * host006.hackeriet.no: LibreNMS device ID 25.
+  * host007.hackeriet.no: LibreNMS device ID 26.
+  * Both devices have OS detected as ''proxmox''.
+  * Both devices have the LibreNMS ''proxmox'' application enabled with app instance ''klynge001''.
+Host-side setup:
+  * ''snmpd'' is installed, enabled, and running on both hosts.
+  * SNMP listens only on the management IP of each host:
+    * host006: ''10.10.50.26:161/udp''
+    * host007: ''10.10.50.27:161/udp''
+  * SNMP uses SNMPv3 authPriv with the security name ''librenms_klynge001''.
+  * Credentials are stored in Hackerpass at ''infrastructure/librenms-klynge001-snmpv3''. Do not put the credential values in the wiki or NetBox.
+  * The Proxmox LibreNMS agent-local script is installed as ''/usr/local/libexec/librenms-proxmox''.
+  * SNMP exposes Proxmox VM traffic through:
+<code>
+extend proxmox /usr/bin/sudo /usr/local/libexec/librenms-proxmox
+</code>
+  * ''/etc/sudoers.d/librenms-proxmox'' allows the ''Debian-snmp'' user to run only that script via sudo.
+Firewall setup:
+  * The Proxmox cluster firewall allows SNMP only from app-01 / LibreNMS source address ''10.10.50.51'' to the ''klynge001'' IP set.
+  * The firewall rule is in ''/etc/pve/firewall/cluster.fw'':
+<code>
+IN ACCEPT -source 10.10.50.51/32 -dest +klynge001 -p udp -dport 161 -log nolog # LibreNMS SNMP polling from app-01
+</code>
+LibreNMS setup:
+  * ''enable_proxmox'' is set to ''true''.
+  * The ''unix-agent'' poller module is disabled per device for host006 and host007, because this setup uses SNMP extend instead of the LibreNMS unix-agent on port 6556.
+  * No LibreNMS alert rules existed when this monitoring was added. Host metrics and Proxmox application polling are live, but alert policy still needs to be defined.
+Useful verification commands:
+<code>
+# On each Proxmox host
+systemctl is-active snmpd
+ss -lunp | grep ':161'
+sudo -u Debian-snmp sudo /usr/local/libexec/librenms-proxmox
+# From the LibreNMS container on app-01
+snmpwalk -v3 -l authPriv -u librenms_klynge001 -a SHA -A '<auth password>' -x AES -X '<privacy password>' host006.hackeriet.no SNMPv2-MIB::sysName.0
+snmpget -v3 -l authPriv -u librenms_klynge001 -a SHA -A '<auth password>' -x AES -X '<privacy password>' -Oqv host006.hackeriet.no .1.3.6.1.4.1.8072.1.3.2.3.1.2.7.112.114.111.120.109.111.120
+lnms device:poll -m applications host006.hackeriet.no
+</code>
 ===== Follow-up items =====
-  * Fix or intentionally retire Munin on host006 and host007.
+  * Define LibreNMS alert rules and notification routing for klynge001 host monitoring.
-  * Review VM autostart settings for guests expected to return after host reboot.
   * Investigate the host007 GRUB/LVM ''pv1'' warning.
   * Investigate the host006 EFI fallback warning.
   * Review whether the two-node quorum procedure should be part of the standard klynge001 maintenance runbook.
   * Keep NetBox VM placement aligned with the current reality for moved service VMs.
+  * If Munin package cleanup is desired later, remove it in a separate low-risk cleanup task.
-===== Safety notes =====
+===== Other notes =====
-  * Avoid guest-level changes during host maintenance unless needed for recovery.
-  * Do not change DNS during the maintenance window unless DNS itself is the incident.
-  * Do not delete backups or ISOs without understanding what they are.
-  * Keep local notes while working; wiki, NetBox, IDP, and public services may be affected depending on which host is down.
   * Host006 storage and backup context is documented at [[infra:operations:proxmox-backups|Proxmox backups]].
   * Certificate automation for internal Proxmox hostnames is documented at [[infra:operations:proxmox-acme-dns|Proxmox ACME DNS automation]].