====== Host006 ======

===== Purpose =====

''host006.hackeriet.no'' is a Proxmox VE hypervisor in the ''klynge001'' cluster.

For inventory details, use the NetBox/IPAM entry ''host006'' (device ID ''75''). NetBox is the source of truth for device model, rack/location, cabling, IP assignments, tenant, and current inventory state.

===== Emergency Access =====

Normal access:

  * ''ssh root@host006.hackeriet.no''
  * Proxmox web UI: ''https://host006.hackeriet.no:8006/'' or ''https://10.10.50.26:8006/''

Access notes:

  * Proxmox access requires Hackeriet VPN/internal connectivity.
  * The Proxmox HTTPS certificate may have issues; browser/curl certificate warnings can be expected until fixed.
  * If DNS or public routing is confusing, NetBox currently documents the management address as ''10.10.50.26/24''.

===== Credentials =====

Do not paste secrets into the wiki.

Relevant ''hackeriet/pass'' entries:

  * ''root@host006.hackeriet.no.gpg''
  * ''host006-backup-age-key.gpg''

Keep decrypted values out of the wiki.

===== Operating System =====

Observed over SSH on 2026-05-20:

  * OS: ''Debian GNU/Linux 12 (bookworm)''
  * Kernel: ''6.8.12-9-pve''
  * Proxmox VE: ''pve-manager/8.3.5''

Treat this as emergency orientation only. Live state may differ.

===== Cluster =====

Observed over SSH on 2026-05-20:

  * Cluster: ''klynge001''
  * Local node IP: ''10.10.50.26''
  * Quorum: yes
  * Observed members: ''10.10.50.26'' and ''10.10.50.27''
  * Expected votes: ''3''
  * Total votes observed: ''2''

If cluster state matters during an incident, verify it live with ''pvecm status''.

===== Network =====

NetBox and LibreNMS currently agree on the physical uplink:

  * NetBox interface: ''host006'' / ''LAN''
  * NetBox cable: ''#38''
  * Switch: ''pit-sw''
  * Switch port: ''GigabitEthernet0/30''

Observed host-level bridges include:

  * ''vmbr50'' for infra hosts / management, address ''10.10.50.26/24''
  * ''vmbr130'' for server network / public-facing VM traffic
  * ''vmbr3'' for Hackeriet wired network

Do not treat this page as the source of truth for network inventory. Use NetBox for that.

===== Host-Level Backups =====

The host has a Proxmox ''vzdump'' backup job named ''backup-5b87a3ab-9074''.

Observed configuration on 2026-05-20:

  * Schedule: ''sun 03:00''
  * Storage: ''local''
  * Mode: ''snapshot''
  * Compression: ''zstd''
  * Retention: ''keep-last=1''
  * Mail: failure notifications to ''backupmail@hackeriet.no''

Recent observed failure pattern:

  * Some backups fail with ''vma_queue_write: write error - Broken pipe''.
  * Root filesystem and local Proxmox storage were nearly full during observation.

Do not delete backups or ISOs without operator approval.

===== Known Issues To Check =====

  * Root filesystem pressure: ''/'' was observed at 94% used on 2026-05-20.
  * Local Proxmox storage: ''local'' was observed at 88.33% used on 2026-05-20.
  * Failed service observed: ''munin-node.service''.
  * ACME renewal failures for ''host006.hackeriet.no'' were seen in Proxmox logs/config during earlier investigation.

===== Useful Paths =====

Paths commonly useful during incident response:

  * ''/etc/pve/jobs.cfg''
  * ''/etc/vzdump.conf''
  * ''/var/log/vzdump''
  * ''/var/lib/vz/dump''
  * ''/var/lib/vz/template/iso''
  * ''/etc/network/interfaces''
  * ''/etc/pve/nodes/host006/config''

===== Checks During Incidents =====

Basic host checks:

<code>
hostname -f
pveversion
pvecm status
systemctl --failed
df -h -x tmpfs -x devtmpfs
pvesm status
ip -br addr show scope global
</code>

Backup checks:

<code>
sed -n '1,220p' /etc/pve/jobs.cfg
sed -n '1,220p' /etc/vzdump.conf
find /var/log/vzdump -maxdepth 1 -type f -printf '%TY-%Tm-%Td %TH:%TM %p\n' | sort | tail -20
find /var/lib/vz/dump -maxdepth 1 -type f -printf '%TY-%Tm-%Td %TH:%TM %10s %f\n' | sort
</code>

Service checks:

<code>
systemctl status pve-cluster pvedaemon pveproxy pvestatd pvescheduler
systemctl status pve-firewall proxmox-firewall corosync
systemctl status munin-node
journalctl -u munin-node --since "1 hour ago"
</code>