morning and why we endlessly pursue understanding how to protect our clients. The
strictest possible definition of true positive is where an outcome of a prediction or model
is returned true. This broad definition is important because the assumption carried with
a predication or model can change. As an example many security technologies aim to
identify when an application often unwanted by an organization is present within a
system, these are commonly dubbed PUPs. (Potentially Unwanted Programs).
organization however when you traditionally compare the qualities of a PUP vs another
file, qualities of that PUP will be more suggestive of an opportunity to be abused. As
such when security technologies identify PUPs they are considered True Positives.
establishing the criticality of your investigation and where on the spectrum of criticality
you land. Another common example is the deletion of shadow volumes. Often alerts
derived from detections looking for Shadow Volume copies being enumerated and
removed will label themselves “Ransomware Activity”.
removed but it was a necessary step in an organization’s backup process. So from the
perspective of the model or prediction it successfully raised a true positive however from
analysts’ perspective, it was a false positive because an analyst’s own model or
prediction will always be “Was this an adversary”
course even more false positives, but it is expected of an analyst to know out of the
noise which true positive needs escalating into greater human eyes or simply forward to
a client as a polite reminder.
Ugly True Positives
“Ugly True positive’ to denote those that must initiate major incident response
procedures when found. Ugly true positives are measured by how much impact the next
possible action could cause. As an example (colours will darken as the more ugly the investigation leads)
Immediately with the information we have in the table we will want to establish whether
authentication attempts from Canada normally occur.
While asking these investigative questions we can already start thinking about if this
was an event generated by an adversary they would now have access to the data the account does
on its own it meant nothing to us and now we are seeing activity occur that we wouldn’t
expect a human to do.’ At this point, you want to increase the amount of energy we are
expending because we have moved to post-compromise activity which is the ugliest of
An analysts nightmare
Below are some tell tall signs you have found an ugly true positive:
- Identification of initial access followed by escalating activity
- Recurrence of the same alert across multiple assets
- Unrelated alerts converging on a common objective
- Document its discovery and your classification
- Detail your investigative steps
- Inform the local team
- What actions you would like to take further
- Escalate to team leadership
- Articulate the severity and the data you used.
- Prepare for containment and post-compromise assessment
Engineering the Ugly True Positives
need to be measuring how severe a situation it is if the detection fires. Using context a
detection should behave in different ways.
Some detections should skip 200 and go straight to jail alerting and ultimately executing
major incident response procedures immediately for example when 200 of the largest
EC2 instance plans are span up in under an hour. But some detections need to serve as
little pieces to a puzzle like “connections made on high-ranged ports for new binaries”
being joined to “new binary spawning script interpreter” and as these little puzzle pieces
build up to a threshold that is established and if lapsed you have your self an Ugly true
How the detections manifest to analysts can go a long way too, the words High
Severity and the Colour Red doesn’t really carry any conscious weight in the mind