Proxmox and Adversaries

❗take the time to read the bottom of this page.

Introduction

Proxmox is a virtualisation technology stack that is quickly becoming the go to product solution for organisations looking to heal the wounds left by Broadcoms acquisition of VMWARE. This post aims to equip detection engineers with the knowledge to identify adversary behaviour and implement their own detection logic. This post explores Proxmox deployments on Linux hosts only

The content shared in this article is also a demo of Sonny's (https://detection.wiki/labs/) initiative to promote more comprehensive sharing of artefacts and audit logs in threat research posts. This initiative aims to provide every reader with the raw logs for each item addressed in the research and the tools to easily upload and test those logs in a kusto cluster without any friction or manually parsing.

Explore the systems that make sharing logs with each other important here: https://detection.wiki/blog/triaging-a-threat-report/

ℹ️ Blogger supports artefact sharing poorly so all the key details are kept here: https://connectedlucy.github.io/proxmox/2026/06/13/proxmox_death.html

Skip to the above link for the most interesting parts of the post.

Adversary techniques

Proxmox's non appliance nature means that most deployment instances are directly compatible with old and well tested adversary playbooks for linux compromises. Generic Linux detections are plentiful and highly relevant to proxmox installations. This post focuses on unique complexity introduced by the proxmox installation and will only mention genetic Linux detections were the matching behaviour is exceptionally common.

Initial Access

Authentication is managed through realms that are configured individually in the Proxmox GUI interface. By default Proxmox provides two realms PVE and PAM. The PAM realm utilises the host based user management controls exposing the root account for the host into the Proxmox GUI experience. In addition to the default realms LDAP is also configurable. 

All configured realms are easily enumerated by making an unauthenticated GET request to /api2/json/access/domains/. Once successfully authenticated a list of assigned permissions is sent along with the granted ticket. Later in the deep dive section I explore how to detect adversaries exploring the API endpoints and brute forcing valid accounts.

Living off the land

After gaining access to a PVE node adversaries are afforded a wealth of native binaries that allow further compromise without the need for additional capability staging. 'pvesh' is the powerful utility available affording its users the ability to enumerate, change and destroy most the resources in a Proxmox environment. As such it should be closely monitored for abuse. 

Using the pvesh utility an adversary can enumerate and destroy Guest VMs with commands as simple as:

Guest VM details :: pvesh get /nodes/pve/qemu/707/status/current/

Destroy Guest VM :: pvesh delete /nodes/pve/qemu/707

Extensive effort should undertaken to model the usage of the Proxmox CLI tools in your environment. Use of tools like pvesh are not logged in the pveproxy audit trail and can only be observed via process execution and journald event streams.

In my experience system administrators rarely call pve cli tools from a direct SSH session and instead choose to utilise a shell spawned from the GUI interface inside which they execute commands.

🌟 Model PVE CLI usage using the following parameters:

• SSH session relationship
• Diversity of pve cli usage
• Directory cli tool invoked from

ℹ️ List of other Proxmox tools can be found here https://pve.proxmox.com/pve-docs/#_proxmox_ve_manual_pages

In addition to the native pve binaries Proxmox utilises a number of local config files that adversaries can abuse to maintain persistence and evict responders trying to contain a threat. 

In particular adversaries can append IP addresses using IP sets in the host file /etc/pve/firewall/cluster.fw. IP sets allow for control of the firewall without the need for configuring all the typical syntax a rule might need like destination, port and interface. Using the IP set '[IPSET blacklist]' adversaries can define a list of administrative subnets and block all remote access attempts.

These config files are assigned READ - WRITE for the root account only but should be monitored using the AuditD configuration detailed below.

Logs Overview

To ensure appropriate coverage of audit activity collected from the proxmox product, supplementary host system activity must also be collected to provide detection coverage into techniques that jeopardize the integrity of the systems at the host level.

AuditD

Host system visibility can primarily be achieved through auditd. Auditd is a well supported third party package that can collect telemetry for a wide range operating system features. Telemetry collection is defined in a configuration file stored locally on each PVE node.

Through the collection of the host based telemetry we can aim to generate extended coverage such as:

• Security control evasion
  • log collector tampering
  • local account changes
  • firewall changes
  • log file tampering
• Remote access monitoring
  • SSH
• File system monitoring
  • executable files in sensitive directories
  • direct modification of the PVE installation folder
• Process execution
• pve cli usage
• system enumeration
• remote connections

🌟 If you choose to utilise a common public auditd configuration such as those provided by app armour or Florian. Ensure you append the following lines to the config file:

## Proxmox VE
# configuration files
-w /etc/pve/ -p wa -k pve_config_changes

# management tools
-w /usr/sbin/qm -p x -k pve_qm_exec
-w /usr/sbin/pct -p x -k pve_pct_exec
-w /usr/bin/pvecm -p x -k pve_cluster_exec
-w /usr/sbin/pvesm -p x -k pve_storage_exec
-w /usr/sbin/pveum -p x -k pve_user_exec
-w /usr/bin/pvesh -p x -k pve_shell_exec

# Proxmox daemon
-w /usr/bin/pvedaemon -p x -k pve_daemon_exec
-w /usr/bin/pveproxy -p x -k pve_proxy_exec

# Perl Libraries.
-w /usr/share/perl5/PVE/ -p wa -k pve_core_perl_changes

Proxmox Environment Logs

Interactions directly in the proxmox GUI experience are stored in the log file /var/log/pveproxy/access.log. This log file contains API requests handled by the ‘pveproxy' system service. Once received the requests are then sent to 'pvedaemon’ and executed. Interactions are generally categorised through the corresponding HTTP request type where deleting a resource or configuration will generate a log with the DELETE request type and modifying configuration items will generate a POST or PATCH request type.

JournalD

The proxmox environment registers its own records directly to JournalD by default. These logs are a simplified abstraction of the ‘pveproxy’ logs mentioned above and extend into node or cluster tasks (jobs) that were not routed through the proxy. Importantly not all actions are logged into journal in particular items like user creation and modification or cluster changes must be captured through the other log sources detailed in this post.

Example: Event generated from deletion of virtual machine

<root@pam> delete snapshot VM 3411: blueturtle

To ensure appropriate log coverage the following unit files tracked by journalD must be collected:

• pvedaemon.service
• pve-cluster.service
• pveproxy.service
• pvefw-logger.service
• proxmox-firewall.service
• pvesh.service
• pve-guests.service
• pvescheduler.service
• ssh.service
• cron.service

Deep dive into analytics 

🌟 Testing your own detections or want to practice hunting? Upload the full log to your SIEM: sim_proxmox_full_log.json

Explore the systems that make sharing logs with each other important here: Will Any of This Fire?

Proxmox does not afford its users native controls for monitoring process executions nor file writes unlike VMWare ESXI and as such auditd or other similar technologies must be deployed. Using auditd we can easily capture adversary behaviour and develop a series of detection ideas that compliment each other.

I have utilised a parser to better prepare the logs for validation and in this parser ‘PROCTILE’ events are decode from hexadecimal to full strings. You must also do this using your own tools if you wish to process the events in the same capacity as I have in this blog.

Initial Access and Local System Discovery

Proxmox appliances can utilise the same user management provider for both terminal access and the virtual environment graphical interface. In this example the adversary has gained access to a Proxmox appliance through a remote SSH session but has chosen to pivot to the graphical interface with the knowledge that interactions are harder to trace and therefore less likely to be logged.

To maximise our coverage of this horizontal movement we can utilise the pveproxy and journald. These logs store API requests the daemon processes and executes.

GUI Authentication Logs:

In our example the adversary has attempted to brute force valid accounts across a number of domains. This is easily discoverable through the journald logs for ‘pvedaemon.service’. Aggregating on the key fields creates a simple view:

Basic explorative query in CQL:

regex(field=@rawstring, regex="rhost=(?:::ffff:)?(?<ip>.*?)\s+user=(?<user>[^@\s]+)(?:@(?<domain>[a-zA-Z0-9.\-]+))?")
// no realm means local host auth was tried
| case{

  domain != *
  | domain := "pam"; *
  
  }
  | groupBy([user, domain, ip], function=count(as=Total))| sort(Total)

[
  {
    "ip": "192.168.1.157",
    "domain": "pam",
    "Total": "30",
    "user": "root"
  },
  {
    "ip": "192.168.1.157",
    "domain": "open",
    "Total": "13",
    "user": "root"
  },
  {
    "ip": "192.168.1.157",
    "domain": "pve",
    "Total": "9",
    "user": "admin"
  },
  {
    "ip": "192.168.1.157",
    "domain": "pve",
    "Total": "5",
    "user": "root"
  }
]

Detection Analytics

Identify the volume of each key property per source

CQL

// extract key fields
regex(field=@rawstring, regex="rhost=(?:::ffff:)?(?<ip>.*?)\s+user=(?<user>[^@\s]+)(?:@(?<domain>[a-zA-Z0-9.\-]+))?")
// no realm means local host auth was tried
| case{

  domain != *
  | domain := "pam"; *
  
  }
// aggregate by source
| groupBy([ip], function=[
                          // count key fields
                          count(as=Total), 
                          count(domain, distinct=true, as=TotalDomains), 
                          count(user, distinct=true, as=TotalUsers),
                          // find start and end of activity
                          max(@timestamp, as=end), 
                          min(@timestamp, as=start)])

// calculate new values
| delta := end - start
| round("delta")
| formatDuration("delta")

KQL

| extend key_fields = extract_all(@"rhost=(?:::ffff:)?(?<ip>.*?)\s+user=(?<user>[^@\s]+)(?:@(?<domain>[a-zA-Z0-9.\-]+))?", rawstring)
| extend 
    ip = tostring(key_fields[0][0]),
    user = tostring(key_fields[0][1]),
    domain = tostring(key_fields[0][2])
| summarize 
            total = count(), 
            dcount(user), 
            dcount(domain), 
            max(TimeGenerated),
            min(TimeGenerated) by ip
| extend delta = max_TimeGenerated - min_TimeGenerate

Once an adversary has GUI access to a Proxmox environment their actions are only traceable through these API audit logs. Additionally the Proxmox interface also offers several options for a new shell to be spawned. Creating a shell is logged in the aforementioned API audit logs however it is not afforded any terminal logging forcing us to utilise auditd to trace any activity.

GUI Shell Logs:

Journalctl provides an auditable trace of which shells were spawned by the GUI under the pvedaemon.service unit.

VNC shell

<root@pam> starting task UPID:pve:00001508:00026749:6A33B297:vncshell::root@pam:

starting vnc proxy UPID:pve:00001508:00026749:6A33B297:vncshell::root@pam:

launch command: /usr/bin/vncterm -rfbport 5900 -timeout 10 -authpath /nodes/pve -perm Sys.Console -notls -listen localhost -c /bin/login -f root

launch command: /usr/bin/vncterm -rfbport 5900 -timeout 10 -authpath /nodes/pve -perm Sys.Console -notls -listen localhost -c /bin/login -f root

Spice Shell

<root@pam> starting task UPID:pve:000017F7:0002C7B3:6A33B38E:spiceshell::root@pam:

starting spiceterm UPID:pve:000017F7:0002C7B3:6A33B38E:spiceshell::root@pam: - Shell on 'pve'

launch command: /usr/bin/spiceterm --port 61000 --addr localhost --timeout 40 --authpath /nodes/pve --permissions Sys.Console --keymap en-gb -- /bin/login -f root

xterm.js (default shell option)

<root@pam> starting task UPID:pve:00001A2C:0003190A:6A33B45E:vncshell::root@pam:

starting termproxy UPID:pve:00001A2C:0003190A:6A33B45E:vncshell::root@pam:

View the full log here: https://connectedlucy.github.io/assets/proxmox_death/sim_proxmox_guishell_rawlog.json

In our example once an adversary has spawned a new shell via the GUI they begin executing shell commands to explore the pve nodes file system with the aim to identify where Guest VM backups are stored.

Guest VM components exist logically in a few key areas of each PVE node. Primarily the Guest VM sits as a logical volume on the selected disk. This is represented under /dev/pts/ and /dev/pve/. Additionally backups created for each Guest VM and any miscellaneous backup logs are stored in the directory ‘/var/lib/vz/dump/’.

Adversaries can easily enumerate these storage locations using the following command

find /var/lib/vz/dump/ -type f -name "*zst*"

Find: A LOLBIN kept on all debian hosts.

[
  {
    "first_event": "2026-06-18 09:09:24.486",
    "Vendor.audit_type": "EXECVE",
    "Vendor.audit_content": "argc=6 a0=\"find\" a1=\"/var/lib/vz/dump/\" a2=\"-type\" a3=\"f\" a4=\"-name\" a5=\"*zst*\""
  },
  {
    "first_event": "2026-06-18 09:09:24.486",
    "Vendor.audit_type": "PATH",
    "Vendor.audit_content": "item=0 name=\"/usr/bin/find\" inode=260723 dev=fc:01 mode=0100755 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0"
  },
  {
    "first_event": "2026-06-18 09:09:24.486",
    "Vendor.audit_type": "PATH",
    "Vendor.audit_content": "item=1 name=\"/lib64/ld-linux-x86-64.so.2\" inode=264121 dev=fc:01 mode=0100755 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0"
  },
  {
    "first_event": "2026-06-18 09:09:24.486",
    "Vendor.audit_type": "PROCTITLE",
    "Vendor.audit_content": "find\u0000/var/lib/vz/dump/\u0000-type\u0000f\u0000-name\u0000*zst*"
  },
  {
    "first_event": "2026-06-18 09:09:24.486",
    "Vendor.audit_type": "SYSCALL",
    "Vendor.audit_content": "arch=c000003e syscall=59 success=yes exit=0 a0=5edc57afb490 a1=5edc57f51760 a2=5edc57f2bcc0 a3=8 items=2 ppid=7764 pid=7882 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=5 comm=\"find\" exe=\"/usr/bin/find\" subj=unconfined key=\"exec\""
  }
]

lvdisplay: LOLbin that displays volume information

[
  {
    "first_event": "2026-06-18 09:23:18.143",
    "Vendor.audit_type": "EXECVE",
    "Vendor.audit_content": "argc=1 a0=\"lvdisplay\""
  },
  {
    "first_event": "2026-06-18 09:23:18.143",
    "Vendor.audit_type": "PATH",
    "Vendor.audit_content": "item=0 name=\"/usr/sbin/lvdisplay\" inode=265377 dev=fc:01 mode=0100755 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0"
  },
  {
    "first_event": "2026-06-18 09:23:18.143",
    "Vendor.audit_type": "PATH",
    "Vendor.audit_content": "item=1 name=\"/lib64/ld-linux-x86-64.so.2\" inode=264121 dev=fc:01 mode=0100755 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0"
  },
  {
    "first_event": "2026-06-18 09:23:18.143",
    "Vendor.audit_type": "PROCTITLE",
    "Vendor.audit_content": "lvdisplay"
  },
  {
    "first_event": "2026-06-18 09:23:18.143",
    "Vendor.audit_type": "SYSCALL",
    "Vendor.audit_content": "arch=c000003e syscall=59 success=yes exit=0 a0=5edc57adb980 a1=5edc57f504c0 a2=5edc57f2bcc0 a3=8 items=2 ppid=7764 pid=10088 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=5 comm=\"lvdisplay\" exe=\"/usr/sbin/lvm\" subj=unconfined key=\"exec\""
  }
]

Detection Analytics

Identify storage and volume enumeration activity

CQL

| case {
  
  // directory used in cmdline
  audit_type = PROCTITLE
  | process.command_line = /\/var\/lib\/vz\/dump\//i
  | direct_dir_access := @timestamp;

  // volume enumeration
  dataset = auditd.syscall
  | Vendor.comm = lvdisplay
  | lvdisplay := @timestamp;

  // pve shell enumeration
  audit_type = EXECVE
  |  process.command_line = /\/usr\/bin\/pvesh get/i
  | pve_shell_enum := @timestamp;
  
  //QEMU enumeration
  audit_type = EXECVE
  |  process.command_line = /\/usr\/sbin\/qm list/i
  | qemu_enum := @timestamp;

  //pve storage manager storage content
  audit_type = EXECVE
  |  process.command_line = /\/usr\/sbin\/pvesm list/i
  | storage_content := @timestamp;

  //pve storage manager volume paths
  audit_type = EXECVE
  |  process.command_line = /\/usr\/sbin\/pvesm path/i
  | volume_paths:= @timestamp;

  
}
| groupBy([@collect.host], function=[
                                      min(direct_dir_access, as=min_direct_dir_access),
                                      min(lvdisplay, as=min_lvdisplay),
                                      min(pve_shell_enum, as=min_pve_shell_enum),
                                      min(qemu_enum, as=min_qemu_enum),
                                      min(storage_content, as=min_storage_content),
                                      min(volume_paths, as=min_volume_paths),
                                      collect(process.command_line, Vendor.comm)
  
  
                          ])

Don't use AI

Often people think that delegating tasks to AI tools will free up time to do other more worthwhile things. This is the first mistake, the labour involved in a task is important even if the reason why your doing it is not.

Often people think that the world's adoption of AI tools is much like the adoption of the calculator or mobile phones. This is the second mistake. AI tools have torn from our world it's most valuable possessions and so born from the dark pits of soulless executives with dollar signs gleaming in their eyes they emerged and they do work. They work too well. It's creators have developed a medium from which humans can pour out unconstrained thoughts. Thoughts that carry no beauty or deliberateness.

The modern computer took away things from us too but in doing so it pushed the boundaries of what can be done into new areas never before perceived. This exchange seems to of been worthwhile. AI tools do not offer a similar exchange, their ability to act as a surface without rough edges or muddy reflections means there's no room left for us. No way to abstract the information further pressing it's users against a ceiling of higher order thinking that is in no way nourishing.

Researching adversary behaviour and building systems necessitates you doing the hard parts. If your new to the work you need the depth lost in AI tools to fall in love and if your already committed to the work your knowledge needs to stretch across as much complexity as possible to keep your wisdom turning into ramblings. Using AI tools writing detections will make you worse at detecting adversary behaviour.

All energy is only borrowed and one day you have to give it back. Don't waste yours on AI.