Exploratory Threat Analytics using Jupyter Notebooks

Watch the detailed Presentation & Explaination of my talk on this topic here

Why Notebooks & not SIEM/EDR or any other Tool?

SIEM is for Log Collection & Detection, Case Management is for Notes & Annotation, Let's use Jupyter for Investigation

  • Vendor agnostic Query Language: Programming Language

  • Intersection of Code/Investigation/Annotation-notes

  • Programmatic control over Data/Logs

  • Enrichment and Context on the Fly!!

  • Orchestrated Approach and Flow

Let's Explore a simple Investigation of Finding Suspicious Powershell Executions.

  • Two Common Use-case that I have is :

    • Import bulk alerts that might have been triggered in the last 1 day due to a spike in data volume and perform statistical analysis on them to do a bulk investigation

    • Or Do bulk Analysis of alerts for a New Detection to analyze areas of Fine-tuning and improvement/Context.

Use-Case : Data from Wherever you want (Security Tools/S3/Online Datasets/Git repo)

Use-Case: Data Analysis Capabilities

  • Need Python Data Analysis capabilities?

Need More ??, SQL, Graphs, ML, Threat Intelligence, Alerts, Datasets, Visualization...?

  • Gather Analytical capabilities (pyspark, Seaborn,plotly, graphframes)

let me also Import Pyspark, you know for SQL capabilities

Let's do a function to query Elastic to Pull data! We can call this Function From whenever I Need, so subject data at our disposal is sorted!!.

Pull all elasticsearch Events from my SIEM - Elastic to investigate/Hunt for ['Command and Scripting Interpreter: PowerShell'] https://attack.mitre.org/techniques/T1059/001/

Use-Case : Statistical Capabilities at your disposal

Use-Case: SuperCharged API Utilization

  • use API powers of your tools - SQL API from Elasticsearch

Timestamp
Process
Source_ip
Source_port
Destination_ip
Destination_port

0

2022-09-30T21:02:02.508Z

powershell.exe

192.168.1.104

58100

58.158.177.102

80

1

2022-11-18T05:58:53.998Z

powershell.exe

192.168.1.104

55563

58.158.177.102

80

2

2022-09-30T21:02:02.751Z

powershell.exe

192.168.1.104

58101

58.158.177.102

80

3

2022-11-18T05:58:53.998Z

powershell.exe

192.168.1.104

55563

58.158.177.102

80

4

2022-11-18T05:58:54.236Z

powershell.exe

192.168.1.104

55564

58.158.177.102

80

Popular Threat Hunting Techniques like stack counting/Grouping/Clustering are a breeze away!!

  • Stack Counting to Check Unique IPs and Connection Count

Interestingly All Connections are made to the Same Destination IP

Use-Case - Data Enrichment

  • Let's Enrich reputational Data from Virustotal

  • Gather Data & Intelligence: There are Product APIs, Webhooks (Siem/Case-management/Threat Intelligence Platform/EDRs/ Git/Slack), and Service-APIs(Virustotal, Curl Websites, scrape data), The possibility to gather data is endless.

Use-Case - Data Visualization powers (You are free to use your favourite library, Matplotlib, seaborn, plotly etc etc..)

  • Exploratory Analysis of Process events using plotly

Calling in Data from EDR Logs - this could be your EDR of choice, Defender/Crowdstrike/Carbon-black/sentinelOne/Elastic-EDR/OSQUERY etc etc.

I'm using Elastic-EDR along with Elastic SIEM for this Case study.

Use-Case - programmatic Control over Data, Wrangling, tuning, sanitization, enrichment, whatever you need !!!

  • Truly a Canvas limited by the Artist's Creativity.

Timestamp_date
Commandline
0

0

2022-09-01

powershell.exe -ExecutionPolicy Restricted -Command Write-Host 'Final result: 1';

4

1

2022-09-02

powershell.exe -ExecutionPolicy Restricted -Command Write-Host 'Final result: 1';

2

2

2022-09-03

powershell.exe -ExecutionPolicy Restricted -Command Write-Host 'Final result: 1';

2

3

2022-09-04

"C:\Windows\SysWOW64\WindowsPowerShell\v1.0\powershell.exe"

2

4

2022-09-04

"C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe"

2

Powershell Commandlines could easily be based lines via visualization.

Use-Case - Case-Management- bleeding into the Lifecycle !!!

  • We'll use the Hive Case Management solution for the Demo

More Tips & use-cases

  • Tip: Wrap all your reusable functions into a separate python file -> Import and call them in all your notebooks wherever needed

  • Tip: Schedule your notebooks to perform periodic hunts/Data Analysis reports

  • Tip: Write a Web server to call Notebooks on Demand or Use CLoud services like AWS Sagemaker to Make it API Driven.

  • Usecase: Correlation - More Events from your security layers from Zeek/Suricata, Threat Intelligence Platform, MITRE, Firewall

  • Usecase: Containment Action, call your EDR/Tools API to contain a Host or Perform network isolation

What you've achieved by Using Jupyter Notebooks for conducting this Analysis :

  • What investigation was performed?

  • Notebook as tactical Investigation Report

  • Reusable Notebook - Variables not Constants

  • You Can Draft your :

    • Hunting Notebook

    • Data Analysis Notebook

    • Investigation Notebook

    • Response Notebook

    • Detection Notebooks

    • Threat Intelligence Tracking Notebooks

Also, this Notebook is available for use & download at my GitHub repository.

Last updated

Was this helpful?