Proxmox and Adversaries

Introduction

❗take the time to read the bottom of this page.

Proxmox is a virtualisation technology stack that is quickly becoming the go to product solution for organisations looking to heal the wounds left by Broadcoms acquisition of VMWARE. This post aims to equip detection engineers with the knowledge to identify adversary behaviour and implement their own detection logic. This post explores Proxmox deployments on Linux hosts only

The content shared in this article is also a demo of Sonny's (puppy) (https://detection.wiki/labs/) initiative to promote more comprehensive sharing of artefacts and audit logs in threat research posts. This initiative aims to provide every reader with the raw logs for each item addressed in the research and the tools to easily upload and test those logs in a kusto cluster without any friction or manually parsing.

Explore the systems that make sharing logs with each other important here: https://detection.wiki/blog/triaging-a-threat-report/

ℹ️ Blogger supports artefact sharing poorly so all the key details are kept here: https://connectedlucy.github.io/proxmox/2026/06/13/proxmox_death.html

Skip to the above link for the most interesting parts of the post.



Adversary techniques

Proxmox's non appliance nature means that most deployment instances are directly compatible with old and well tested adversary playbooks for linux compromises. Generic Linux detections are plentiful and highly relevant to proxmox installations. This post focuses on unique complexity introduced by the proxmox installation and will only mention genetic Linux detections were the matching behaviour is exceptionally common.

Initial Access

Authentication is managed through realms that are configured individually in the Proxmox GUI interface. By default Proxmox provides two realms PVE and PAM. The PAM realm utilises the host based user management controls exposing the root account for the host into the Proxmox GUI experience. In addition to the default realms LDAP is also configurable. 

All configured realms are easily enumerated by making an unauthenticated GET request to /api2/json/access/domains/. Once successfully authenticated a list of assigned permissions is sent along with the granted ticket. Later in the deep dive section I explore how to detect adversaries exploring the API endpoints and brute forcing valid accounts.

Living off the land

After gaining access to a PVE node adversaries are afforded a wealth of native binaries that allow further compromise without the need for additional capability staging. 'pvesh' is the powerful utility available affording its users the ability to enumerate, change and destroy most the resources in a Proxmox environment. As such it should be closely monitored for abuse. 

Using the pvesh utility an adversary can enumerate and destroy Guest VMs with commands as simple as:

Guest VM details :: pvesh get /nodes/pve/qemu/707/status/current/

Destroy Guest VM :: pvesh delete /nodes/pve/qemu/707


Extensive effort should undertaken to model the usage of the Proxmox CLI tools in your environment. Use of tools like pvesh are not logged in the pveproxy audit trail and can only be observed via process execution and journald event streams.

In my experience system administrators rarely call pve cli tools from a direct SSH session and instead choose to utilise a shell spawned from the GUI interface inside which they execute commands.

🌟 Model PVE CLI usage using the following parameters:

  • SSH session relationship
  • Diversity of pve cli usage
  • Directory cli tool invoked from

ℹ️ List of other Proxmox tools can be found here https://pve.proxmox.com/pve-docs/#_proxmox_ve_manual_pages


In addition to the native pve binaries Proxmox utilises a number of local config files that adversaries can abuse to maintain persistence and evict responders trying to contain a threat. 

In particular adversaries can append IP addresses using IP sets in the host file /etc/pve/firewall/cluster.fw. IP sets allow for control of the firewall without the need for configuring all the typical syntax a rule might need like destination, port and interface. Using the IP set '[IPSET blacklist]' adversaries can define a list of administrative subnets and block all remote access attempts.

These config files are assigned READ - WRITE for the root account only but should be monitored using the AuditD configuration detailed below.


Logs Overview

To ensure appropriate coverage of audit activity collected from the proxmox product, supplementary host system activity must also be collected to provide detection coverage into techniques that jeopardize the integrity of the systems at the host level.

AuditD

Host system visibility can primarily be achieved through auditd. Auditd is a well supported third party package that can collect telemetry for a wide range operating system features. Telemetry collection is defined in a configuration file stored locally on each PVE node.

Through the collection of the host based telemetry we can aim to generate extended coverage such as:

  • Security control evasion
    • log collector tampering
    • local account changes
    • firewall changes
    • log file tampering
  • Remote access monitoring
    • SSH
  • File system monitoring
    • executable files in sensitive directories
    • direct modification of the PVE installation folder
  • Process execution
    • pve cli usage
    • system enumeration
    • remote connections
🌟 If you choose to utilise a common public auditd configuration such as those provided by app armour or Florian. Ensure you append the following lines to the config file:


## Proxmox VE
# configuration files
-w /etc/pve/ -p wa -k pve_config_changes

# management tools
-w /usr/sbin/qm -p x -k pve_qm_exec
-w /usr/sbin/pct -p x -k pve_pct_exec
-w /usr/bin/pvecm -p x -k pve_cluster_exec
-w /usr/sbin/pvesm -p x -k pve_storage_exec
-w /usr/bin/pveum -p x -k pve_user_exec
-w /usr/bin/pvesh -p x -k pve_shell_exec

# Proxmox daemon
-w /usr/bin/pvedaemon -p x -k pve_daemon_exec
-w /usr/bin/pveproxy -p x -k pve_proxy_exec

# Perl Libraries.
-w /usr/share/perl5/PVE/ -p wa -k pve_core_perl_changes




Proxmox Environment Logs

Interactions directly in the proxmox GUI experience are stored in the log file /var/log/pveproxy/access.log. This log file contains API requests handled by the ‘pveproxy' system service. Once received the requests are then sent to 'pvedaemon’ and executed. Interactions are generally categorised through the corresponding HTTP request type where deleting a resource or configuration will generate a log with the DELETE request type and modifying configuration items will generate a POST or PATCH request type.

JournalD

The proxmox environment registers its own records directly to JournalD by default. These logs are a simplified abstraction of the ‘pveproxy’ logs mentioned above and extend into node or cluster tasks (jobs) that were not routed through the proxy. Importantly not all actions are logged into journal in particular items like user creation and modification or cluster changes must be captured through the other log sources detailed in this post.

Example: Event generated from deletion of virtual machine 

<root@pam> delete snapshot VM 3411: blueturtle

To ensure appropriate log coverage the following unit files tracked by journalD must be collected:

  • pvedaemon.service
  • pve-cluster.service
  • pveproxy.service
  • pvefw-logger.service
  • proxmox-firewall.service
  • pvesh.service
  • pve-guests.service
  • pvescheduler.service
  • ssh.service
  • cron.service

Deep dive into analytics 

In the sister post at the link below you can find full log samples for you to upload into your own SIEM

ℹ️ This post continues in the resource 

Don't use AI

Often people think that delegating tasks to AI tools will free up time to do other more worthwhile things. This is the first mistake, the labour involved in a task is important even if the reason why your doing it is not.

Often people think that the world's adoption of AI tools is much like the adoption of the calculator or mobile phones. This is the second mistake. AI tools have torn from our world it's most valuable possessions and so born from the dark pits of soulless executives with dollar signs gleaming in their eyes they emerged and they do work. They work too well. It's creators have developed a medium from which humans can pour out unconstrained thoughts. Thoughts that carry no beauty or deliberateness.

The modern computer took away things from us too but in doing so it pushed the boundaries of what can be done into new areas never before perceived. This exchange seems to of been worthwhile. AI tools do not offer a similar exchange, their ability to act as a surface without rough edges or muddy reflections means there's no room left for us. No way to abstract the information further pressing it's users against a ceiling of higher order thinking that is in no way nourishing.

Researching adversary behaviour and building systems necessitates you doing the hard parts. If your new to the work you need the depth lost in AI tools to fall in love and if your already committed to the work your knowledge needs to stretch across as much complexity as possible to keep your wisdom turning into ramblings. Using AI tools writing detections will make you worse at detecting adversary behaviour.

All energy is only borrowed and one day you have to give it back. Don't waste yours on AI.

Popular

Brilliance in the Basics

Image
Introduction Tired of watching you and your friends get compromised, do exactly what's in this blog and start beating adversaries. Avoiding the memes adversaries win because of simple mistakes and neglect and we all already know what they are so I'm going to list them for you. Its all for free too. Enumerate To put it simply, there shouldn't be anything you don't know about your environment, you should know who all your users are, where all your electronic devices are and what they do, what applications you have and what versions they are. Enumeration is the jet fuel for making good defensive decisions. I wrote about how to enumerate your environment here  Securing your estate: The First Step (goblinloot.net) . Follow these steps and become the arbitrator of your own environment  Your perimeter is a bridge, not a wall Monitor your perimeter as best as you can but always assume it has already been defeated. Monitor endpoint system and process telemetry and southwest traf...
Read more

Investigate

Image
  Introduction This document details how an analyst should conduct investigations and triage in the normal duties of their job. It will describe concepts and logic that enable analysts to look at data, ask investigative questions, evaluate those questions and arrive at assertions about what they have found. An Alert For the most part, analysts begin their investigative work when an alert is generated. Alerts are fantastic because they are a statement that something has happened and something needs to be done about it. Typically alerts are filled with information that an analyst can then use in their investigation. Alerts are generated by detections and these detections can come in many shapes and sizes. Some just look for a particular action like someone unlocking their car door, others might have access to more context like someone who isn’t the owner of the car unlocking the door. Going even further some detections might look at how the car door was opened or at w...
Read more

Endpoint on Adrenaline : One

Image
Introduction   This blog series will capture how to maximise the protection of an endpoint using the various technologies in the Defender suite. The controls outlined in the series will each need to have their own considerations taken into account for a given environment but hopefully at a minimum it will be clear just how powerful of an offering Microsoft has.  Unfortunately due to the highly modular and distinct diversity in controls Microsoft offers things are often overlooked and misconfigured when tests are conducted. With this in mind, the shortest possible route to a solution for this is utilizing the recommendations, CSPM and Secure Score technology contained within Defender 365 and Defender for Cloud as it will automatically conduct posture assessments and inform you of things you have missed and may want to turn on (IE dont take these recommendations lightly) Lastly, I would recommend reading the book “Defender for Endpoint in depth” from packt as it showcases the co...
Read more

Endpoint on Adrenaline 3

Image
Introduction Now that I have covered the advanced features obtained from Defender for Cloud and the complexities of Defender for Endpoint with the objective of delivering as much protection as possible to the endpoint, I will now explore how to expand on this greatly and open up even more use cases. Microsoft Sentinel This technology will allow us to unify and correlate data generated by other controls. I have decided to deploy Microsoft Sentinel into 4 primary categories : System Administrator - Asset Uptime, Application Inventory, Device configuration and Troubleshooting Network Engineers - Netflow and session traffic, violations and alerts for potential issues Threat Detection - Analytics and playbooks Threat hunting - Workbooks & Machine learning Using Microsoft Sentinel we can achieve a plethora of incredibly powerful use cases with Defender 365 endpoint data. Below is an illustration of how my SIEM works to enable analysts to prevent and detect threats Data Sources In order t...
Read more

Endpoint on Adrenaline Two

Image
Introduction Continuing on from my last post that captured using Defender for Cloud to gain powerful additional features on top of defender for endpoint to protect your endpoints we are going to take a closer look at Defender for Endpoint itself and extend into some endpoint-specific use cases for Defender for Identity that tie directly back into Defender for Endpoint. Firstly a really important consideration when configuring Defender for Endpoint is that alot of the heavy lifting from a prevention standpoint actually happens in Defender Anti virus. This means that while im going to briefly cover some settings to make sure Defender Anti Virus isnt hindering Defender for Endpoint im going to reserve further Defender Anti Virus controls to the next part in this series. This also of course means that when your doing your own testing just installing the MSSense agent (What you get from the scripts, intune etc) is actually not Defender for Endpoint in its totality so you simply can not m...
Read more

Investigate Three

Image
 Analysis  In this post, I explain analysis and the associated techniques to mean at the lowest possible level a human’s ability to consume external stimulus of its near-infinite complexity and produce thoughtful and data-backed decisions. Within our industry, we are plagued with information twisted and corrupted by entities wishing to profit, As such this is where your analysis starts as a human. At every moment at every crossroad, you must take a data point and allow it to flower into thousands of other thoughts, then you must evaluate each subsequent thought and draw a line back to “what is already” known, The further away the thought the thinner the line back to known but the greater room the flower to bloom. That is to say don’t discount radical ideas. When presented with an alert security vendors expend significant energy into putting “options” available to the individual reviewing the alert. In general, there will never be an option you should not explore what is key ...
Read more

Investigate Two

Image
Introductions How you as an analyst handle true positives is life and death in the eyes of potential victims. Traditionally the industry elected to prioritize overzealousness and sending more than not to cover their failings, This however is no different to guesses and it is possible to arrive at strong data-backed decisions on events that could be the compromise of an estate True Positive True positives exist at the heart of all our jobs, they are what we wake up for in the morning and why we endlessly pursue understanding how to protect our clients. The strictest possible definition of true positive is where an outcome of a prediction or model is returned true. This broad definition is important because the assumption carried with a predication or model can change. As an example many security technologies aim to identify when an application often unwanted by an organization is present within a system, these are commonly dubbed PUPs. (Potentially Unwanted Programs). The general cons...
Read more

Writing detections when stuck with EDR

Image
Introduction Making the leap to purchasing and maintaining an EDR solution can be huge for organisations so huge in fact that they never really progress from there when it comes to visibility on assets. This is an important consideration because organisations should aim to achieve some level of detection engineering but will likely never get to explore the associated complexities. So how can organisations effectively and easily write detections that actually help to protect them when they are only able to leverage an EDR tool. Understanding your tools limitations To most people EDR tools are incredibly verbose in the telemetry they capture from any given asset but unfortunately relative to overall telemetry available they actually only capture what the vendors think is the most pertinent. This is for obvious reasons because ingesting telemetry at the scale vendors do comes at a significant cost so they implement "cost-saving measures" like limiting the amount of any one given...
Read more

Standardized Note Taking Format For Analysts

Image
  Introduction This post outlines a format for note-taking designed to aid analysts and ensure the knowledge they acquire over time is kept so that it can easily be consumed later. This format is designed to be technology agnostic so it can be applied to a note-taking tool or platform of the analyst’s choice. I designed this format because analysts often received what you could call unplanned information whereby at any given moment during their day they could learn something highly impactful to their role via any medium. This really exasperates the need for good note-taking practices for analysts. Illustrated above is the wireframe that analysts can implement into their note-taking platforms. Each section contains sub-sections that allow for different knowledge to remain segregated and more searchable. Meeting Notes Within any given week analysts can have countless meetings and so it is important to note down items from meetings for future review. Not every meeting needs to be ...
Read more