Goblin Detection Diary #1 - Data is queen

Introduction

Detection engineering underpins half of the entire cybersecurity industry but remains only ever softly spoken about or kept in some corner of the conference. So I've started this diary to capture the work I do in my roles and demonstrate different best practices implemented into the real world.

This diary has five goals set:

  • share with people the complexity of working in detection engineering
  • highlight best practice and how it fits
  • share candid details on inter-team work and break/fix tasks
  • ensure every entry has something that can be used immediately by other people
  • keep the format like diary with no formal structure and a personal tone


Note: The content I generate will be sanitised but will avoid high level overviews. I hope other detection engineers enjoy my pain with me and new aspirants become engorged with new ideas.


Starting Slowly..

This entry was made at the start of the week and so I spent time running my regular reporting. I run reporting and metrics at the start of the week because it allows me to setup my planner (Microsoft ToDo) with tasks and deadlines.

As always, key to creating a good reporting schedule is understanding where your data is so I maintain a map of all the alerting sources I manage and how to query them. It looks something like this:



I've already built the reporting to happen automatically for me but what's not mentioned in the diagram are analytics I use to discover new elements of infrastructure or data generated. These analytics essentially capture what was seen last week and compare it to what was captured this week to highlight new additions. This is important because I rely on third party platforms like Azure / Microsoft and Crowdstrike who don't keep their customers up to date on everything they change so whenever they add new logs or expand old ones I'm kept in the loop.

This week I shared a few observations with other teams as they lack the capacity to understand the data like I can but still benefit from the information I generate.

  • Quietest periods of the week were sent to SOC leaders for rota management
  • Playbook errors due to case sensitivity sent to other engineers

Besides sharing information about what I find in a reporting cycle I also set a new task in my planner to explore user account creation patterns at a particular client. They had generated in excess of 30 new accounts recently and with a quick cursory check I can see its not the first time so I want to explore whether that behaviour is relevant enough to adjust my own models and baselines.

Now the cogs are turning..

Another engineer had been working on changes to the deployment pipeline and was exploring detections I've written that were generating flags in his new linter. This was my favourite part of the day because I got to showcase detection methodologies I apply across a large number of rules and also demo the advantages and disadvantages of different algorithms and functions.

In particular I walked through how we evaluate events in sequence where two behaviours must occur in that the first is required to occur x times before the second. This is a really common logic found across many different detection ideas.

For new detection engineers I would recommend trying to learn query optimisation and good data modelling practices from outside the cybersecurity industry as the large vendors such as Splunk and Microsoft have in their generosity fostered a lot of lazy attitudes. In addition to this exploring different logic concepts in pure code will help build skills that you will find universally applicable as apposed to being focused on SPL or KQL.

There are a few ways to approach this and largely you want to make the decision based on the nature of the platform your using. SaaS platforms like Sentinel and Crowdstrike allow for inefficient approaches to function well. If you're using one of these you can usually approach the problem by generating a list of records in series and aggregating on a bin of those records for a timespan you give like say 45 minutes. This will then allow you to evaluate patterns inside the list/index and hopefully its obvious to most which option is better:

store it in a string

user.name @timestamp _duration event.type[1] source.ip AccessDenied AccessAllowed
fluffiest.cat@supercat.org Oct. 28, 2025 14:07:13.000 25000 denied denied denied denied denied allowed 172.16.8.11 36 139

store it in an array


New Data New Problems

Later in the day the SOC raised to me that a detection which evaluates Entra ID authentication events for properties that are either anomalous or known bad was firing too many false positives because a client had onboarded a new region. Looking at the data I immediately had two avenues to explore:


  • What activity are the new users performing in this new region
  • How much variance do the authentication properties have in a given week

Modelling this data to explore the two above points is really easy and can be done using basic aggregate functions. I added some additional logic to look at the volumes of activity for the users and calculate percentages for increases or decreases which allows me to easily understand how significant of a change has occurred in the data.

Across most tools you will simply need to compare two time windows of data and define your basic medium and upper bound along with calculating the standard deviation. You should get a result like this!

Present90 Historic50 Historic90 avg_daily_events daily_event_variation PercentDecrease PercentIncrease
0.4149804220344742 0.42 0.45448793562740775 231765.47 18291.06 -8.55 0.91
1.5183902122100648 1.63 2.802939744889062 173693.43 80118.5 -75 0.25
1.0924868727462969 0.84 1.200818894660522 215086.27 52950.67 15.96 1.16
1.6016681095583383 1.63 1.7209205113771453 218395.55 45422.7 -6.88 0.93

Often in analytics shared online you will see thresholds with hard coded values but gaining understanding of volume using basic statistics like this allows teams to move away from them because in the best of worlds hard coded thresholds require manual regular reviews to ensure coherency with changing understandings of the business and at worse are left unchanged for months. 

Iterate 

Another thing I was exploring today was how our analysts find documentation, while I was working across different detections I described above I had the thought that some of the fields I drop (remove from an output) before presenting the final result might be useful because inside the logic for all our detections is a name for the detection and the documentation assigned to each one uses this same value so if I kept the key:value  for analysts they can just copy and paste the whole string into our documentation store instead of searching on keywords. Luckily I keep logs on what searches analysts make so i can answer questions like what paths analysts are taking to find materials and key phrases they use that might help me improve search optimisation.

Constantly identifying points of improvement in the analyst experience through having a robust understanding of how the analysts interact with your systems and data can help alleviate fatigue and decrease both triage time and complexity. At the moment I'm in the process of migrating our old rule sets to a new framework that outlines an absolute minimum needed from any given alert. In some cases this means that our analytics grow in complexity by about a third (the number of functions and operators) however the value gained from the significant investment in time will be realised forever and hopefully appreciated by all analysts.

Example items in the alert framework:
  • All values with a relationship to an identity or asset such as a user name or IP must be surrounded by metadata such as privileges assigned or in which country the user typically authenticates.
  • All results are transformed into a table.
  • All results are compatible with graph.
  • Where external interfaces are required to triage the alert a link is surfaced to that interface in the alert.




Popular

Brilliance in the Basics

Image
Introduction Tired of watching you and your friends get compromised, do exactly what's in this blog and start beating adversaries. Avoiding the memes adversaries win because of simple mistakes and neglect and we all already know what they are so I'm going to list them for you. Its all for free too. Enumerate To put it simply, there shouldn't be anything you don't know about your environment, you should know who all your users are, where all your electronic devices are and what they do, what applications you have and what versions they are. Enumeration is the jet fuel for making good defensive decisions. I wrote about how to enumerate your environment here  Securing your estate: The First Step (goblinloot.net) . Follow these steps and become the arbitrator of your own environment  Your perimeter is a bridge, not a wall Monitor your perimeter as best as you can but always assume it has already been defeated. Monitor endpoint system and process telemetry and southwest traf...
Read more

Investigate

Image
  Introduction This document details how an analyst should conduct investigations and triage in the normal duties of their job. It will describe concepts and logic that enable analysts to look at data, ask investigative questions, evaluate those questions and arrive at assertions about what they have found. An Alert For the most part, analysts begin their investigative work when an alert is generated. Alerts are fantastic because they are a statement that something has happened and something needs to be done about it. Typically alerts are filled with information that an analyst can then use in their investigation. Alerts are generated by detections and these detections can come in many shapes and sizes. Some just look for a particular action like someone unlocking their car door, others might have access to more context like someone who isn’t the owner of the car unlocking the door. Going even further some detections might look at how the car door was opened or at w...
Read more

Endpoint on Adrenaline : One

Image
Introduction   This blog series will capture how to maximise the protection of an endpoint using the various technologies in the Defender suite. The controls outlined in the series will each need to have their own considerations taken into account for a given environment but hopefully at a minimum it will be clear just how powerful of an offering Microsoft has.  Unfortunately due to the highly modular and distinct diversity in controls Microsoft offers things are often overlooked and misconfigured when tests are conducted. With this in mind, the shortest possible route to a solution for this is utilizing the recommendations, CSPM and Secure Score technology contained within Defender 365 and Defender for Cloud as it will automatically conduct posture assessments and inform you of things you have missed and may want to turn on (IE dont take these recommendations lightly) Lastly, I would recommend reading the book “Defender for Endpoint in depth” from packt as it showcases the co...
Read more

Endpoint on Adrenaline 3

Image
Introduction Now that I have covered the advanced features obtained from Defender for Cloud and the complexities of Defender for Endpoint with the objective of delivering as much protection as possible to the endpoint, I will now explore how to expand on this greatly and open up even more use cases. Microsoft Sentinel This technology will allow us to unify and correlate data generated by other controls. I have decided to deploy Microsoft Sentinel into 4 primary categories : System Administrator - Asset Uptime, Application Inventory, Device configuration and Troubleshooting Network Engineers - Netflow and session traffic, violations and alerts for potential issues Threat Detection - Analytics and playbooks Threat hunting - Workbooks & Machine learning Using Microsoft Sentinel we can achieve a plethora of incredibly powerful use cases with Defender 365 endpoint data. Below is an illustration of how my SIEM works to enable analysts to prevent and detect threats Data Sources In order t...
Read more

Endpoint on Adrenaline Two

Image
Introduction Continuing on from my last post that captured using Defender for Cloud to gain powerful additional features on top of defender for endpoint to protect your endpoints we are going to take a closer look at Defender for Endpoint itself and extend into some endpoint-specific use cases for Defender for Identity that tie directly back into Defender for Endpoint. Firstly a really important consideration when configuring Defender for Endpoint is that alot of the heavy lifting from a prevention standpoint actually happens in Defender Anti virus. This means that while im going to briefly cover some settings to make sure Defender Anti Virus isnt hindering Defender for Endpoint im going to reserve further Defender Anti Virus controls to the next part in this series. This also of course means that when your doing your own testing just installing the MSSense agent (What you get from the scripts, intune etc) is actually not Defender for Endpoint in its totality so you simply can not m...
Read more

Investigate Two

Image
Introductions How you as an analyst handle true positives is life and death in the eyes of potential victims. Traditionally the industry elected to prioritize overzealousness and sending more than not to cover their failings, This however is no different to guesses and it is possible to arrive at strong data-backed decisions on events that could be the compromise of an estate True Positive True positives exist at the heart of all our jobs, they are what we wake up for in the morning and why we endlessly pursue understanding how to protect our clients. The strictest possible definition of true positive is where an outcome of a prediction or model is returned true. This broad definition is important because the assumption carried with a predication or model can change. As an example many security technologies aim to identify when an application often unwanted by an organization is present within a system, these are commonly dubbed PUPs. (Potentially Unwanted Programs). The general cons...
Read more

Investigate Three

Image
 Analysis  In this post, I explain analysis and the associated techniques to mean at the lowest possible level a human’s ability to consume external stimulus of its near-infinite complexity and produce thoughtful and data-backed decisions. Within our industry, we are plagued with information twisted and corrupted by entities wishing to profit, As such this is where your analysis starts as a human. At every moment at every crossroad, you must take a data point and allow it to flower into thousands of other thoughts, then you must evaluate each subsequent thought and draw a line back to “what is already” known, The further away the thought the thinner the line back to known but the greater room the flower to bloom. That is to say don’t discount radical ideas. When presented with an alert security vendors expend significant energy into putting “options” available to the individual reviewing the alert. In general, there will never be an option you should not explore what is key ...
Read more

Writing detections when stuck with EDR

Image
Introduction Making the leap to purchasing and maintaining an EDR solution can be huge for organisations so huge in fact that they never really progress from there when it comes to visibility on assets. This is an important consideration because organisations should aim to achieve some level of detection engineering but will likely never get to explore the associated complexities. So how can organisations effectively and easily write detections that actually help to protect them when they are only able to leverage an EDR tool. Understanding your tools limitations To most people EDR tools are incredibly verbose in the telemetry they capture from any given asset but unfortunately relative to overall telemetry available they actually only capture what the vendors think is the most pertinent. This is for obvious reasons because ingesting telemetry at the scale vendors do comes at a significant cost so they implement "cost-saving measures" like limiting the amount of any one given...
Read more

Standardized Note Taking Format For Analysts

Image
  Introduction This post outlines a format for note-taking designed to aid analysts and ensure the knowledge they acquire over time is kept so that it can easily be consumed later. This format is designed to be technology agnostic so it can be applied to a note-taking tool or platform of the analyst’s choice. I designed this format because analysts often received what you could call unplanned information whereby at any given moment during their day they could learn something highly impactful to their role via any medium. This really exasperates the need for good note-taking practices for analysts. Illustrated above is the wireframe that analysts can implement into their note-taking platforms. Each section contains sub-sections that allow for different knowledge to remain segregated and more searchable. Meeting Notes Within any given week analysts can have countless meetings and so it is important to note down items from meetings for future review. Not every meeting needs to be ...
Read more