zeek2es.py
This Python application translates Zeek's ASCII TSV logs into ElasticSearch's bulk load JSON format. For JSON logs, see Elastic's File Beats application.
This application will recognize gzip or uncompressed logs.
This application assumes you have Elasticsearch set up on your localhost at the default port.
Run this program on a system with the same timezone that was logged by Zeek originally, as zeek-cut -d -u translates the timestamps into UTC for Elasticsearch.
Command Line:
python zeek2es.py your_zeek_log.gz -i your_es_index_name
This script can be run in parallel on all connection logs, 10 at a time, with the following command:
find /some/dir -name “conn*.log.gz” | parallel -j 10 python zeek2es.py {1} :::: -
Command Line Options:
$ python zeek2es.py -h
usage: zeek2es.py [-h] [-i ESINDEX] [-u ESURL] [-l LINES] [-n NAME] [-c] [-s] [-b] filename
Process Zeek ASCII logs into Elasticsearch.
positional arguments:
filename The Zeek log in *.log or *.gz format. Include the full path.
optional arguments:
-h, --help show this help message and exit
-i ESINDEX, --esindex ESINDEX
The Elasticsearch index name.
-u ESURL, --esurl ESURL
The Elasticsearch URL. (default: http://localhost:9200/)
-l LINES, --lines LINES
Lines to buffer for RESTful operations. (default: 50,000)
-n NAME, --name NAME The name of the system to add to the index for uniqueness. (default: empty string)
-c, --checkindex Check for the ES index first, and if it exists exit this program.
-s, --stdout Print JSON to stdout instead of sending to Elasticsearch directly.
-b, --nobulk Remove the ES bulk JSON header. Requires --stdout.
Requirements:
- A Unix-like environment (MacOs works!)
- Python
- zeek-cut in your path
6 Dec 7, 2021
45 Nov 25, 2022
1 Dec 7, 2021
4 Jul 20, 2022
14 Dec 22, 2022
62 Sep 20, 2022
8 Dec 18, 2022
79 Nov 11, 2022
9 Dec 20, 2022
3 Jan 09, 2022
1 Jan 30, 2022
150 Nov 23, 2022
5 Dec 01, 2021
2 Dec 25, 2022
44 Dec 29, 2022
3 Jan 24, 2022
235 Dec 29, 2022
4 Aug 22, 2022
4 Sep 19, 2022
4 Jan 01, 2022
309 Jan 01, 2023
3 Aug 08, 2022
0 Jan 22, 2022
2 Feb 02, 2022
1.5k Jan 05, 2023
30 Dec 23, 2022
5 Jul 12, 2019