Cses2humio - CrowdStrike Falcon Event Stream to Humio

Overview

CI

CrowdStrike Falcon Event Stream to Humio

This project intend to provide a simple way of moving your CrowdStrike Falcon Event Stream data into Humio.
As is the only reliable way of getting Event Stream data is with CrowdStrike's SIEM connector, that dumps to multiple files.
We're trying to bypass the file stage and ship this directly to Humio and streamline ingest, providing CrowdStrike customers
with Humio a simple way to maintain, visualize and alert on Falcon Event Stream data.


Design

This project is build as a Python package (application) to be shipped within a Docker or other containerized environment. The application error handling could be better, and the primary way to respond to unexpected errors is to shut down, relying on docker to restart the process.


Prerequisite

For setting up the connection you need two types of credentials ready.

For running the code one of the following is needed.

  • Docker with access to persistent volume
  • Python3 (with virtual environment recommended!)

Obtaining Falcon API key-pair

  • Login to your Falcon Console
  • Go to Menu -> Support -> API Clients and Keys
  • Click Add new API client
    • Set the client a name
    • Optional: fill the description
    • Assign Event streams/read access

falcon_token


Obtaining Humio Ingest Token

  • Login to your Humio cluster

  • Go to the repository you're going to use

  • Depending on if you're going to run enriched or not.

    • Enriched: Download the siem-connector-enriched.yaml parser

      • Go to Parsers and click New Parser then select From template

      • Give the parser a name, note, this is going to be assigned the #type field. E.g. siem-connector-enriched

      • Upload the yaml specification. This is for now an empty parser, you can simply create an empty parser yourself as well.

    • Normal: Install the package crowdstrike/siem-connector by doing the following
      Note that enriched event can use thhis content as well

      • Go to Settings -> Marketplace -> crowdstrike/siem-connector and click Install package -> Install
  • Go to Settings -> Ingest tokens and click Add token

    • Give the ingest token a good name

    • Enriched: assign the parser you created in previous step

    • Normal: select the crowdstrike/siem-connector -> siem-connector

humio_token


Installation

We recommend using the docker image unless you plan around creating this as a systemd service or similar.

# Clone the sample environment file
wget https://raw.githubusercontent.com/Trifork-Security/cses2humio/master/cses2humio.env.example -O cses2humio.env

Modify the attributes accordingly, for more information see Command line and arguments

Start the container with the newly configured environment file

docker run -v $HOST_DATA_DIR:/data  \
    --name=cses2humio \
    --env-file=$PATH_TO_CONFIG_FILE \
    --detach --restart=always \
    ghcr.io/trifork-security/cses2humio:latest

See your data coming in!

docker logs -f cses2humio

Command line and arguments

You can specify run arguments as command lines or environment variables (same as command line, just all uppercase)

Argument             Environment Description
--offset-file OFFSET_FILE General: Where to save offsets for partitions. File will be created automatically
Default: offset.db
Note that the cses2humio.env.example defaults to /data/offset.db
--enrich ENRICH General: Parses the events before shipping to Humio, and expands some fields due to such parsing in Humio can be tricky
Default: False
--verbose VERBOSE General: Be verbose, use for debugging and troubleshooting
Default: False
--falcon-url FALCON_URL Falcon: Url to the API, not the console
Default: https://api.crowdstrike.com
--falcon-api-id FALCON_API_ID Falcon: API ID for the created key
Default: N/A
--falcon-api-secret FALCON_API_SECRET Falcon: API Secret for the created key
Default: N/A
--humio-url HUMIO_URL Humio: Url for the Humio Cluster for events to go
Default: https://cloud.humio.com
--humio-token HUMIO_TOKEN Humio: Ingest token, remember to assign correct parser
Default: N/A
--app-id APP_ID Advanced: Specific to Falcon Event Stream, don't change unless you know what you're doing!
Default: cses2humio
--user-agent USER_AGENT Advanced: User agent used in HTTP requests
Default: cses2humio/{version}
--bulk-max-size BULK_MAX_SIZE Advanced: Maximum number of events to send in bulk
Default: 200`
--flush-wait-time FLUSH_WAIT_TIME Advanced: Maximum wait time before flushing queue
Default: 10

You can also run the tool directly from commandline (using environment variables as well)

cses2humio -h
usage: cses2humio [-h] [--offset-file OFFSET_FILE] [--enrich] [-v] [--falcon-url FALCON_URL] [--falcon-api-id FALCON_API_ID] [--falcon-api-secret FALCON_API_SECRET] [--humio-url HUMIO_URL] [--humio-token HUMIO_TOKEN] [--app-id APP_ID] [--user-agent USER_AGENT] [--bulk-max-size BULK_MAX_SIZE]
                  [--flush-wait-time FLUSH_WAIT_TIME]

CrowdStrike Falcon Event Stream to Humio

optional arguments:
  -h, --help            show this help message and exit

General:
  --offset-file OFFSET_FILE
                        Location including filename for where to store offsets, default is current directory as offset.db
  --enrich              Will parse some fields as they're hard to parse in Humio.Note this might be more resources intensive but spare Humio of parsing. Default is off
  -v, --verbose         Increase output verbosity

Falcon:
  --falcon-url FALCON_URL
                        Falcon API URL, note this is for the API given when you create the API key. Defaults to US-1 API url
  --falcon-api-id FALCON_API_ID
                        Falcon API ID to use for OAuth2
  --falcon-api-secret FALCON_API_SECRET
                        Falcon API Secret to use for OAuth2

Humio:
  --humio-url HUMIO_URL
                        Humio URL for the cluster going to ingest data. Default to https://cloud.humio.com
  --humio-token HUMIO_TOKEN
                        Ingest token to use for ingesting data. Remember to assign the correct parser depending on parsing

Advanced:
  --app-id APP_ID       App ID to use for consuming events
  --user-agent USER_AGENT
                        User agent used to connect to services
  --bulk-max-size BULK_MAX_SIZE
                        Maximum number of events to send in bulk
  --flush-wait-time FLUSH_WAIT_TIME
                        Maximum time to wait if bulk max size isn't reached


Building

# Clone the repo and switch to it
git clone https://github.com/Trifork-Security/cses2humio.git
cd cses2humio
# Create virtual environment and activate (optional, but recommended)
python3 -m venv venv
source venv/bin/activate
# Install build and build the package (used in Dockerfile)
pip3 install build
python3 -m build 
# Build the docker image
docker build -t [TAG_FOR_IMAGE] .

Contributing

Please feel free to contribute at any time by doing a PR.


License

Apache License 2.0

Owner
Trifork.Security
Trifork.Security
A very simple asynchronous wrapper that allows you to get access to the Oracle database in asyncio programs.

cx_Oracle_async A very simple asynchronous wrapper that allows you to get access to the Oracle database in asyncio programs. Easy to use , buy may not

33 Jan 21, 2022
Daniel Vaz Gaspar 3.6k Jan 13, 2022
web.py is a web framework for python that is as simple as it is powerful.

web.py is a web framework for Python that is as simple as it is powerful. Visit http://webpy.org/ for more information. The latest stable release 0.62

5.7k Jan 13, 2022
Dockerized web application on Starlite, SQLAlchemy1.4, PostgreSQL

Production-ready dockerized async REST API on Starlite with SQLAlchemy and PostgreSQL

Artur Shiriev 3 Jan 18, 2022
PipeLayer is a lightweight Python pipeline framework

PipeLayer is a lightweight Python pipeline framework. Define a series of steps, and chain them together to create modular applications

greaterthan 63 Dec 25, 2021
Appier is an object-oriented Python web framework built for super fast app development.

Joyful Python Web App development Appier is an object-oriented Python web framework built for super fast app development. It's as lightweight as possi

Hive Solutions 118 Dec 17, 2021
The web framework for inventors

Emmett is a full-stack Python web framework designed with simplicity in mind. The aim of Emmett is to be clearly understandable, easy to be learned an

Emmett 743 Feb 13, 2022
JustPy is an object-oriented, component based, high-level Python Web Framework

JustPy Docs and Tutorials Introduction JustPy is an object-oriented, component based, high-level Python Web Framework that requires no front-en

799 Feb 04, 2022
NO LONGER MAINTAINED - A Flask extension for creating simple ReSTful JSON APIs from SQLAlchemy models.

NO LONGER MAINTAINED This repository is no longer maintained due to lack of time. You might check out the fork https://github.com/mrevutskyi/flask-res

1k Jan 26, 2022
The lightning-fast ASGI server. ?

The lightning-fast ASGI server. Documentation: https://www.uvicorn.org Community: https://discuss.encode.io/c/uvicorn Requirements: Python 3.6+ (For P

Encode 4.8k Jan 15, 2022
Web APIs for Django. 🎸

Django REST framework Awesome web-browsable Web APIs. Full documentation for the project is available at https://www.django-rest-framework.org/. Fundi

Encode 22.6k Jan 16, 2022
Web3.py plugin for using Flashbots' bundle APIs

This library works by injecting a new module in the Web3.py instance, which allows submitting "bundles" of transactions directly to miners. This is done by also creating a middleware which captures c

Georgios Konstantopoulos 147 Jan 21, 2022
Low code web framework for real world applications, in Python and Javascript

Full-stack web application framework that uses Python and MariaDB on the server side and a tightly integrated client side library.

Frappe 3.4k Jan 27, 2022
The source code to the Midnight project

MidnightSniper Started: 24/08/2021 Ended: 24/10/2021 What? This is the source code to a project developed to snipe minecraft names Why release? The ad

Kami 2 Dec 02, 2021
The comprehensive WSGI web application library.

Werkzeug werkzeug German noun: "tool". Etymology: werk ("work"), zeug ("stuff") Werkzeug is a comprehensive WSGI web application library. It began as

The Pallets Projects 5.9k Jan 13, 2022
Python implementation of the Javascript Object Signing and Encryption (JOSE) framework

Python implementation of the Javascript Object Signing and Encryption (JOSE) framework

Demonware 93 Oct 24, 2021
Cses2humio - CrowdStrike Falcon Event Stream to Humio

CrowdStrike Falcon Event Stream to Humio This project intend to provide a simple

Trifork.Security 3 Feb 16, 2022
Djask is a web framework for python which stands on the top of Flask and will be as powerful as Django.

Djask is a web framework for python which stands on the top of Flask and will be as powerful as Django.

Andy Zhou 18 Jan 20, 2022
Light, Flexible and Extensible ASGI API framework

Starlite Starlite is a light, opinionated and flexible ASGI API framework built on top of pydantic and Starlette. Check out the Starlite documentation

Na'aman Hirschfeld 148 Jan 17, 2022
easyopt is a super simple yet super powerful optuna-based Hyperparameters Optimization Framework that requires no coding.

easyopt is a super simple yet super powerful optuna-based Hyperparameters Optimization Framework that requires no coding.

Federico Galatolo 8 Dec 26, 2021