Exploratory Threat Analytics using Jupyter Notebooks
Watch the detailed Presentation & Explaination of my talk on this topic here
Why Notebooks & not SIEM/EDR or any other Tool?
SIEM is for Log Collection & Detection, Case Management is for Notes & Annotation, Let's use Jupyter for Investigation
Vendor agnostic Query Language: Programming Language
Intersection of Code/Investigation/Annotation-notes
Programmatic control over Data/Logs
Enrichment and Context on the Fly!!
Orchestrated Approach and Flow
Let's Explore a simple Investigation of Finding Suspicious Powershell Executions.
Two Common Use-case that I have is :
Import bulk alerts that might have been triggered in the last 1 day due to a spike in data volume and perform statistical analysis on them to do a bulk investigation
Or Do bulk Analysis of alerts for a New Detection to analyze areas of Fine-tuning and improvement/Context.
Use-Case : Data from Wherever you want (Security Tools/S3/Online Datasets/Git repo)
Use-Case: Data Analysis Capabilities
Need Python Data Analysis capabilities?
Need More ??, SQL, Graphs, ML, Threat Intelligence, Alerts, Datasets, Visualization...?
Gather Analytical capabilities (pyspark, Seaborn,plotly, graphframes)
let me also Import Pyspark, you know for SQL capabilities
Let's do a function to query Elastic to Pull data! We can call this Function From whenever I Need, so subject data at our disposal is sorted!!.
Pull all elasticsearch Events from my SIEM - Elastic to investigate/Hunt for ['Command and Scripting Interpreter: PowerShell'] https://attack.mitre.org/techniques/T1059/001/
Use-Case : Statistical Capabilities at your disposal
Use-Case: SuperCharged API Utilization
use API powers of your tools - SQL API from Elasticsearch
0
2022-09-30T21:02:02.508Z
powershell.exe
192.168.1.104
58100
58.158.177.102
80
1
2022-11-18T05:58:53.998Z
powershell.exe
192.168.1.104
55563
58.158.177.102
80
2
2022-09-30T21:02:02.751Z
powershell.exe
192.168.1.104
58101
58.158.177.102
80
3
2022-11-18T05:58:53.998Z
powershell.exe
192.168.1.104
55563
58.158.177.102
80
4
2022-11-18T05:58:54.236Z
powershell.exe
192.168.1.104
55564
58.158.177.102
80
Popular Threat Hunting Techniques like stack counting/Grouping/Clustering are a breeze away!!
Stack Counting to Check Unique IPs and Connection Count
Interestingly All Connections are made to the Same Destination IP
Use-Case - Data Enrichment
Let's Enrich reputational Data from Virustotal
Gather Data & Intelligence: There are Product APIs, Webhooks (Siem/Case-management/Threat Intelligence Platform/EDRs/ Git/Slack), and Service-APIs(Virustotal, Curl Websites, scrape data), The possibility to gather data is endless.
Use-Case - Data Visualization powers (You are free to use your favourite library, Matplotlib, seaborn, plotly etc etc..)
Exploratory Analysis of Process events using plotly
Calling in Data from EDR Logs - this could be your EDR of choice, Defender/Crowdstrike/Carbon-black/sentinelOne/Elastic-EDR/OSQUERY etc etc.
I'm using Elastic-EDR along with Elastic SIEM for this Case study.
Use-Case - programmatic Control over Data, Wrangling, tuning, sanitization, enrichment, whatever you need !!!
Truly a Canvas limited by the Artist's Creativity.
0
2022-09-01
powershell.exe -ExecutionPolicy Restricted -Command Write-Host 'Final result: 1';
4
1
2022-09-02
powershell.exe -ExecutionPolicy Restricted -Command Write-Host 'Final result: 1';
2
2
2022-09-03
powershell.exe -ExecutionPolicy Restricted -Command Write-Host 'Final result: 1';
2
3
2022-09-04
"C:\Windows\SysWOW64\WindowsPowerShell\v1.0\powershell.exe"
2
4
2022-09-04
"C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe"
2

Use-Case - Case-Management- bleeding into the Lifecycle !!!
We'll use the Hive Case Management solution for the Demo
More Tips & use-cases
Tip: Wrap all your reusable functions into a separate python file -> Import and call them in all your notebooks wherever needed
Tip: Schedule your notebooks to perform periodic hunts/Data Analysis reports
Tip: Write a Web server to call Notebooks on Demand or Use CLoud services like AWS Sagemaker to Make it API Driven.
Usecase: Correlation - More Events from your security layers from Zeek/Suricata, Threat Intelligence Platform, MITRE, Firewall
Usecase: Containment Action, call your EDR/Tools API to contain a Host or Perform network isolation
What you've achieved by Using Jupyter Notebooks for conducting this Analysis :
What investigation was performed?
Notebook as tactical Investigation Report
Reusable Notebook - Variables not Constants
You Can Draft your :
Hunting Notebook
Data Analysis Notebook
Investigation Notebook
Response Notebook
Detection Notebooks
Threat Intelligence Tracking Notebooks
Also, this Notebook is available for use & download at my GitHub repository.
Last updated
Was this helpful?