CrowdStrike Falcon Event Stream to Humio
This project intend to provide a simple way of moving your CrowdStrike Falcon Event Stream data into Humio.
As is the only reliable way of getting Event Stream data is with CrowdStrike's SIEM connector, that dumps to multiple files.
We're trying to bypass the file stage and ship this directly to Humio and streamline ingest, providing CrowdStrike customers
with Humio a simple way to maintain, visualize and alert on Falcon Event Stream data.
Design
This project is build as a Python package (application) to be shipped within a Docker or other containerized environment. The application error handling could be better, and the primary way to respond to unexpected errors is to shut down, relying on docker to restart the process.
Prerequisite
For setting up the connection you need two types of credentials ready.
- Falcon API key-pair with read permissions to Event Streams
- Humio ingest token to the repository receiving the data
For running the code one of the following is needed.
- Docker with access to persistent volume
- Python3 (with virtual environment recommended!)
Obtaining Falcon API key-pair
- Login to your Falcon Console
- Go to Menu -> Support -> API Clients and Keys
- Click
Add new API client- Set the client a name
- Optional: fill the description
- Assign
Event streams/readaccess
Obtaining Humio Ingest Token
-
Login to your Humio cluster
-
Go to the repository you're going to use
-
Depending on if you're going to run enriched or not.
-
Enriched: Download the siem-connector-enriched.yaml parser
-
Go to
Parsersand clickNew Parserthen selectFrom template -
Give the parser a name, note, this is going to be assigned the
#typefield. E.g.siem-connector-enriched -
Upload the yaml specification. This is for now an empty parser, you can simply create an empty parser yourself as well.
-
-
Normal: Install the package
crowdstrike/siem-connectorby doing the following
Note that enriched event can use thhis content as well- Go to
Settings->Marketplace->crowdstrike/siem-connectorand clickInstall package->Install
- Go to
-
-
Go to
Settings->Ingest tokensand clickAdd token-
Give the ingest token a good name
-
Enriched: assign the parser you created in previous step
-
Normal: select the
crowdstrike/siem-connector->siem-connector
-
Installation
We recommend using the docker image unless you plan around creating this as a systemd service or similar.
# Clone the sample environment file
wget https://raw.githubusercontent.com/Trifork-Security/cses2humio/master/cses2humio.env.example -O cses2humio.env
Modify the attributes accordingly, for more information see Command line and arguments
Start the container with the newly configured environment file
docker run -v $HOST_DATA_DIR:/data \
--name=cses2humio \
--env-file=$PATH_TO_CONFIG_FILE \
--detach --restart=always \
ghcr.io/trifork-security/cses2humio:latest
See your data coming in!
docker logs -f cses2humio
Command line and arguments
You can specify run arguments as command lines or environment variables (same as command line, just all uppercase)
| Argument | Environment | Description |
|---|---|---|
| --offset-file | OFFSET_FILE | General: Where to save offsets for partitions. File will be created automatically Default: offset.dbNote that the cses2humio.env.example defaults to /data/offset.db |
| --enrich | ENRICH | General: Parses the events before shipping to Humio, and expands some fields due to such parsing in Humio can be tricky Default: False |
| --verbose | VERBOSE | General: Be verbose, use for debugging and troubleshooting Default: False |
| --falcon-url | FALCON_URL | Falcon: Url to the API, not the console Default: https://api.crowdstrike.com |
| --falcon-api-id | FALCON_API_ID | Falcon: API ID for the created key Default: N/A |
| --falcon-api-secret | FALCON_API_SECRET | Falcon: API Secret for the created key Default: N/A |
| --humio-url | HUMIO_URL | Humio: Url for the Humio Cluster for events to go Default: https://cloud.humio.com |
| --humio-token | HUMIO_TOKEN | Humio: Ingest token, remember to assign correct parser Default: N/A |
| --app-id | APP_ID | Advanced: Specific to Falcon Event Stream, don't change unless you know what you're doing! Default: cses2humio |
| --user-agent | USER_AGENT | Advanced: User agent used in HTTP requests Default: cses2humio/{version} |
| --bulk-max-size | BULK_MAX_SIZE | Advanced: Maximum number of events to send in bulk Default: 200` |
| --flush-wait-time | FLUSH_WAIT_TIME | Advanced: Maximum wait time before flushing queue Default: 10 |
You can also run the tool directly from commandline (using environment variables as well)
cses2humio -h
usage: cses2humio [-h] [--offset-file OFFSET_FILE] [--enrich] [-v] [--falcon-url FALCON_URL] [--falcon-api-id FALCON_API_ID] [--falcon-api-secret FALCON_API_SECRET] [--humio-url HUMIO_URL] [--humio-token HUMIO_TOKEN] [--app-id APP_ID] [--user-agent USER_AGENT] [--bulk-max-size BULK_MAX_SIZE]
[--flush-wait-time FLUSH_WAIT_TIME]
CrowdStrike Falcon Event Stream to Humio
optional arguments:
-h, --help show this help message and exit
General:
--offset-file OFFSET_FILE
Location including filename for where to store offsets, default is current directory as offset.db
--enrich Will parse some fields as they're hard to parse in Humio.Note this might be more resources intensive but spare Humio of parsing. Default is off
-v, --verbose Increase output verbosity
Falcon:
--falcon-url FALCON_URL
Falcon API URL, note this is for the API given when you create the API key. Defaults to US-1 API url
--falcon-api-id FALCON_API_ID
Falcon API ID to use for OAuth2
--falcon-api-secret FALCON_API_SECRET
Falcon API Secret to use for OAuth2
Humio:
--humio-url HUMIO_URL
Humio URL for the cluster going to ingest data. Default to https://cloud.humio.com
--humio-token HUMIO_TOKEN
Ingest token to use for ingesting data. Remember to assign the correct parser depending on parsing
Advanced:
--app-id APP_ID App ID to use for consuming events
--user-agent USER_AGENT
User agent used to connect to services
--bulk-max-size BULK_MAX_SIZE
Maximum number of events to send in bulk
--flush-wait-time FLUSH_WAIT_TIME
Maximum time to wait if bulk max size isn't reached
Building
# Clone the repo and switch to it
git clone https://github.com/Trifork-Security/cses2humio.git
cd cses2humio
# Create virtual environment and activate (optional, but recommended)
python3 -m venv venv
source venv/bin/activate
# Install build and build the package (used in Dockerfile)
pip3 install build
python3 -m build
# Build the docker image
docker build -t [TAG_FOR_IMAGE] .
Contributing
Please feel free to contribute at any time by doing a PR.
8 Dec 07, 2021
687 Nov 27, 2022
78 Dec 27, 2022
32 Dec 24, 2022
1 Oct 22, 2021
3 Jul 01, 2021
24.7k Jan 03, 2023
1 Jul 02, 2022
20.9k Jan 01, 2023
9k Jan 01, 2023
2 Jan 04, 2022
16 Dec 15, 2022
1 Dec 26, 2021
526 Jan 02, 2023
29 Nov 20, 2022
927 Jan 08, 2023
3.7k Dec 30, 2022
46 Nov 08, 2022
32 Mar 05, 2022
162 Dec 26, 2022