A Python application to transfer Zeek ASCII (not JSON) logs to Elastic/OpenSearch.

Related tags

JSONzeek2es
Overview

zeek2es.py

This Python application translates Zeek's ASCII TSV logs into ElasticSearch's bulk load JSON format. For JSON logs, see Elastic's File Beats application.

This application will recognize gzip or uncompressed logs.

This application assumes you have Elasticsearch set up on your localhost at the default port.

Run this program on a system with the same timezone that was logged by Zeek originally, as zeek-cut -d -u translates the timestamps into UTC for Elasticsearch.

Command Line:

python zeek2es.py your_zeek_log.gz -i your_es_index_name

This script can be run in parallel on all connection logs, 10 at a time, with the following command:

find /some/dir -name “conn*.log.gz” | parallel -j 10 python zeek2es.py {1} :::: -

Command Line Options:

$ python zeek2es.py -h
usage: zeek2es.py [-h] [-i ESINDEX] [-u ESURL] [-l LINES] [-n NAME] [-c] [-s] [-b] filename

Process Zeek ASCII logs into Elasticsearch.

positional arguments:
  filename              The Zeek log in *.log or *.gz format. Include the full path.

optional arguments:
  -h, --help            show this help message and exit
  -i ESINDEX, --esindex ESINDEX
                        The Elasticsearch index name.
  -u ESURL, --esurl ESURL
                        The Elasticsearch URL. (default: http://localhost:9200/)
  -l LINES, --lines LINES
                        Lines to buffer for RESTful operations. (default: 50,000)
  -n NAME, --name NAME  The name of the system to add to the index for uniqueness. (default: empty string)
  -c, --checkindex      Check for the ES index first, and if it exists exit this program.
  -s, --stdout          Print JSON to stdout instead of sending to Elasticsearch directly.
  -b, --nobulk          Remove the ES bulk JSON header. Requires --stdout.

Requirements:

  • A Unix-like environment (MacOs works!)
  • Python
  • zeek-cut in your path
You might also like...
Python script for converting .json to .md files using Mako templates.

Install Just install poetry and update script dependencies Usage Put your settings in settings.py and .json data (optionally, with attachments) in dat

Alexey Borontov 6 Dec 7, 2021
json|dict to python object

Pyonize convert json|dict to python object Setup pip install pyonize Examples from pyonize import pyonize

bilal alpaslan 45 Nov 25, 2022
Editor for json/standard python data
Editor for json/standard python data

Editor for json/standard python data

null 1 Dec 7, 2021
Convert your JSON data to a valid Python object to allow accessing keys with the member access operator(.)

JSONObjectMapper Allows you to transform JSON data into an object whose members can be queried using the member access operator. Unlike json.dumps in

Owen Trump 4 Jul 20, 2022
Python script to extract news from RSS feeds and save it as json.

Python script to extract news from RSS feeds and save it as json.

Alex Trbznk 14 Dec 22, 2022
Define your JSON schema as Python dataclasses

Define your JSON schema as Python dataclasses

null 62 Sep 20, 2022
A Python tool that parses JSON documents using JsonPath

A Python tool that parses JSON documents using JsonPath

null 8 Dec 18, 2022
Simple Python Library to convert JSON to XML
Simple Python Library to convert JSON to XML

json2xml Simple Python Library to convert JSON to XML

Vinit Kumar 79 Nov 11, 2022
Marshall python objects to and from JSON

Pymarshaler - Marshal and Unmarshal Python Objects Disclaimer This tool is in no way production ready About Pymarshaler allows you to marshal and unma

Hernan Romer 9 Dec 20, 2022
Comments
  • Datastreams for zeek should be inside the logs-* naming

    Datastreams for zeek should be inside the logs-* naming

    Hi

    I just discovered this great repository. I personally think that when using datastreams the naming convention should be logs-zeek-default. This would follow our datastream recommendations for type-dataset-namespace.

    By placing the index with starting logs it will show up automatically within the logs stream and the security pages and all other sorts of places within Kibana.

    documentation enhancement helper scripts 
    opened by philippkahr 7
Releases(v0.3.15)
Owner
Corelight, Inc.
Corelight is the most powerful network visibility solution for information security professionals, founded by the creators of open-source Zeek.
Corelight, Inc.
GitHub Repository
With the help of json txt you can use your txt file as a json file in a very simple way

json txt With the help of json txt you can use your txt file as a json file in a very simple way Dependencies re filemod pip install filemod Installat

Kshitij 1 Dec 14, 2022
A Python tool that parses JSON documents using JsonPath

A Python tool that parses JSON documents using JsonPath

8 Dec 18, 2022
MOSP is a platform for creating, editing and sharing validated JSON objects of any type.

MONARC Objects Sharing Platform Presentation MOSP is a platform for creating, editing and sharing validated JSON objects of any type. You can use any

CASES Luxembourg 72 Dec 14, 2022
jq for Python programmers Process JSON and HTML on the command-line with familiar syntax.

jq for Python programmers Process JSON and HTML on the command-line with familiar syntax.

Denis Volk 3 Jan 09, 2022
Json GUI for No Man's Sky save file

NMS-Save-Parser Json GUI for No Man's Sky save file GUI python NMS_SAVE_PARSER.py [optional|save.hg] converter only python convert.py usage: conver

2 Oct 19, 2022
Creates fake JSON files from a JSON schema

Use jsf along with fake data generators to provide consistent and meaningful fake data for your system.

Andy Challis 86 Jan 03, 2023
Package to Encode/Decode some common file formats to json

ZnJSON Package to Encode/Decode some common file formats to json Available via pip install znjson In comparison to pickle this allows having readable

ZINC 2 Feb 02, 2022
JSON for Modern C++ Release Scripts

JSON for Modern C++ Release Scripts Preparations Install required tools: make install_requirements. Add required keys to config.json (apparently not c

Niels Lohmann 4 Sep 19, 2022
JSONManipulator is a Python package to retrieve, add, delete, change and store objects in JSON files.

JSONManipulator JSONManipulator is a Python package to retrieve, add, delete, change and store objects in JSON files. Installation Use the package man

Andrew Polukhin 1 Jan 07, 2022
Convert your JSON data to a valid Python object to allow accessing keys with the member access operator(.)

JSONObjectMapper Allows you to transform JSON data into an object whose members can be queried using the member access operator. Unlike json.dumps in

Owen Trump 4 Jul 20, 2022
Small python wrapper around the valico rust library to provide fast JSON schema validation.

Small python wrapper around the valico rust library to provide fast JSON schema validation.

Simon J Knibbs 5 Jul 12, 2019
No more boilerplate to check and build a Python object from JSON.

JSONloader This module is for you if you're tired of writing boilerplate that: builds a straightforward Python object from loaded JSON. checks that yo

3 Feb 05, 2022
JSONx - Easy JSON wrapper packed with features.

🈷️ JSONx Easy JSON wrapper packed with features. This was made for small discord bots, for big bots you should not use this JSON wrapper. 📥 Usage Cl

2 Dec 25, 2022
RedisJSON - a JSON data type for Redis

RedisJSON is a Redis module that implements ECMA-404 The JSON Data Interchange Standard as a native data type. It allows storing, updating and fetching JSON values from Redis keys (documents).

3.4k Dec 29, 2022
Wikidot-forum-dump - Simple Python script that dumps a Wikidot wiki forum into JSON structures.

wikidot-forum-dump Script is partially based on 2stacks by bluesoul: https://github.com/scuttle/2stacks To dump a Wiki's forum, edit config.py and put

ZZYZX 1 Jun 29, 2022
Convert your subscriptions csv file into a valid json for Newpipe!

Newpipe-CSV-Fixer Convert your Google subscriptions CSV file into a valid JSON for Newpipe! Thanks to nikcorg for sharing how to convert the CSV into

Juanjo 44 Dec 29, 2022
import json files directly in your python scripts

Install Install from git repository pip install git+https://github.com/zaghaghi/direct-json-import.git Use With the following json in a file named inf

Hamed Zaghaghi 51 Dec 01, 2021
A Python application to transfer Zeek ASCII (not JSON) logs to Elastic/OpenSearch.

zeek2es.py This Python application translates Zeek's ASCII TSV logs into ElasticSearch's bulk load JSON format. For JSON logs, see Elastic's File Beat

Corelight, Inc. 28 Dec 22, 2022
A fast streaming JSON parser for Python that generates SAX-like events using yajl

json-streamer jsonstreamer provides a SAX-like push parser via the JSONStreamer class and a 'object' parser via the ObjectStreamer class which emits t

Kashif Razzaqui 196 Dec 15, 2022
Fileson - JSON File database tools

Fileson is a set of Python scripts to create JSON file databases

Joonas Pihlajamaa 2 Feb 02, 2022