Python Service for MISP Feed Management

Overview

Python Service for MISP Feed Management

This set of scripts is designed to offer better reliability and more control over the fetching of feeds into MISP. For the moment, the schedule is broken up into multiple components, at the top of each plugin and in config.py:

  • MISP_TIMES: An array of times (24hr format) when enabled MISP feeds will be fetched and cached.
  • TEXT_TIMES: An array of times (24hr format) when enabled plaintext and CSV feeds will be fetched and cached.
  • HOURLY_FEEDS An array of the ID's of enabled feeds that you wish to run at the beginning of every hour.
  • FULL_EXPORT_TIME The time (24hr format) that you want to run a full text export of attributes.

In addition to this are "ENABLE" options for all external services. By default, Abuse.ch is configured to run every hour.

Am still working out the best way of going about granular scheduling.

Variable Notes:

  • MISP_ADMIN_KEY: MISP feeds must be fetched by a Site Admin user.
  • MISP_USER_KEY: This can be the key of an Org Admin, Sync User or your own custom role. They must be able to both manage and publish events, and hold the Tag Editor permission.

Installation:

  • Recommended: Ensure that the fetch_feeds and cache_feeds Scheduled Tasks are not enabled. Also, disable the default Abuse.ch feeds as this project includes a module that loads the data with more context and into a separate event each day.
  • SCP this folder to your MISP server.
  • Alter the paths in misp-feeds.service and start_worker.sh to point to where you've dropped the folder.
  • Correct the user in misp-feeds.service if it is not ubuntu.
  • Complete the variables at the top of the feed_manager.py, misp_export.py, otx_misp.py, twitter_misp.py and xforce_misp.py scripts.
  • Run the following (in the misp-feeds folder):
chmod +x start_worker.sh
apt install nodejs
pip3 install -r requirements.txt
sudo mv misp-feeds.service /etc/systemd/system
sudo chown root:root /etc/systemd/system/misp-feeds.service
sudo systemctl daemon-reload
sudo systemctl start misp-feeds.service
  • nodejs is required for cfscrape (used by the Twitter module to get Ghostbin pastes).
  • Check misp_feeds.log for errors. You can also run both of the Python scripts from the command line for standalone, ad-hoc operation.

Module Notes:

Export:

  • This is a rough script that I use for exporting a plaintext list of attributes for ingestion into external facilities. They're output to a subfolder of the MISP webroot, so ensure the script user has permission to write here and there's adequate access control in place.
  • A full export is run once a day for the number of days defined by EXPORT_DAYS. Incremental updates are made daily.
  • The sample values for EXPORT_TAGS and EXPORT_TYPES should give you an idea of how to configure this. 'domain' and 'hostname' can be output separately or together. Use EXPORT_MERGE_HOSTNAME to configure this.

Plugins:

At the top of each plugin are three variables which determine its operation:

  • PLUGIN_NAME: The friendly name of the Plugin. Only used for logging and ad-hoc operation.

  • PLUGIN_ENABLED: Boolean setting to enable/disable the plugin.

  • PLUGIN_TIMES: The times throughout the day to run the plugin. Also accepts 'hourly', which will run it on the hour every hour.

Default plugins are as follows:

  • Abuse.ch: Pulls URLhaus, Feodo Tracker, MalwareBazaar and ThreatFox into a single event per day. Attributes are tagged according to the feed tags and/or classification.
  • CleanMX: Virus and Phishing feeds are pulled into a single event per day. No tagging yet.
  • OTX: Individual pulses form a separate events in MISP. OTX tags can be spammy so are ignored, but Adversary, Malware and ATT&CK techniques are used. Galaxy tags are attempted, and if no appropriate tag can be found, the feed supplied tag is used.
  • RiskIQ: Individual articles form a separate events in MISP. The same method of tagging is employed as OTX.
  • Twitter: Pulls IOC's found on Twitter into a single event per day. GitHub, PasteBin and GhostBin links are followed and also scraped. Attributes are tagged with the hashtags included in the Tweet and the same method as OTX.
  • X-Force: Individual articles form a separate events in MISP. X-Force articles are not tagged, so the Title of the article is parsed to identify Galaxy tags that match Title keywords.
Owner
Chris
Security Architect / Malware Wrangler
Chris
Visual Python and C++ nanosecond profiler, logger, tests enabler

Look into Palanteer and get an omniscient view of your program Palanteer is a set of lean and efficient tools to improve the quality of software, for

Damien Feneyrou 1.9k Dec 26, 2022
Blender addons - A collection of Blender tools I've written for myself over the years.

gret A collection of Blender tools I've written for myself over the years. I use these daily so they should be bug-free, mostly. Feel free to take and

217 Jan 08, 2023
Xoroshiro-cairo - A xoroshiro128** pseudorandom number generator implementation in Cairo

xoroshiro-cairo A xoroshiro128** pseudorandom number generator implementation in

Milan Cermak 26 Oct 05, 2022
Check broken access control exists in the Java web application

javaEeAccessControlCheck Check broken access control exists in the Java web application. 检查 Java Web 应用程序中是否存在访问控制绕过问题。 使用 python3 javaEeAccessControl

kw0ng 3 May 04, 2022
A Classroom Engagement Platform

Project Introduction This is project introduction Setup Setting up Postgres This is the most tricky part when setting up the application. You will nee

Santosh Kumar Patro 1 Nov 18, 2021
A 3-line lisp implementation

Nanolisp The download page of many a language harbors deep senses of forboding, of evil lurking in its native lair. You feel that the language is not

5 Jun 17, 2022
GitHub saver for stargazers, forks, repos

GitHub backup repositories Save your repos and list of stargazers & list of forks for them. Pure python3 and git with no dependencies to install. GitH

Alexander Kapitanov 23 Aug 21, 2022
Sacred is a tool to help you configure, organize, log and reproduce experiments developed at IDSIA.

Sacred Every experiment is sacred Every experiment is great If an experiment is wasted God gets quite irate Sacred is a tool to help you configure, or

IDSIA 4k Jan 02, 2023
Simple web application, which has a single endpoint, dedicated to annotation parsing and convertion.

Simple web application, which has a single endpoint, dedicated to annotation parsing and conversion.

Pavel Paranin 1 Nov 01, 2021
Install JetBrains Toolbox

ansible-role-jetbrains-toolbox Install JetBrains Toolbox Example Playbook This example is taken from molecule/default/converge.yml and is tested on ea

Antoine Mace 2 Feb 04, 2022
C++ Environment InitiatorVisual Studio Code C / C++ Environment Initiator

Visual Studio Code C / C++ Environment Initiator Latest Version : v 1.0.1(2021/11/08) .exe link here About : Visual Studio Code에서 C/C++환경을 MinGW GCC/G

Junho Yoon 2 Dec 19, 2021
Random Programming Language Project

Crastle Random Programming Language Project Freedom of expression Are you a fan of curly brace languages? Then use curly braces! Not a fan of curly br

DevNugget 2 Dec 23, 2021
This is the improvised version of Dobot Magician which can be implemented for Dobot M1

pydobotM1 This is the edited driver for Dobot M1 version of the original pydobot library intended for use with the Dobot Magician. Here's what you nee

Shaik Abdullah 2 Jul 11, 2022
Tugas kelompok Struktur Data

Binary-Tree Tugas kelompok Struktur Data Silahkan jika ingin mengubah tipe data pada operasi binary tree *Boleh juga semua program kelompok bisa disat

Usmar manalu 2 Nov 28, 2022
Py4J enables Python programs to dynamically access arbitrary Java objects

Py4J Py4J enables Python programs running in a Python interpreter to dynamically access Java objects in a Java Virtual Machine. Methods are called as

Barthelemy Dagenais 1k Jan 02, 2023
Python script to autodetect a base set of swiftlint rules.

swiftlint-autodetect Python script to autodetect a base set of swiftlint rules. Installation brew install pipx

Jonathan Wight 24 Sep 20, 2022
A utility control surface for Ableton Live that makes the initialization of a Mixdown quick

Automate Mixdown initialization A script that transfers all the VSTs on your MIDI tracks to a new track so you can freeze your MIDI tracks and then co

Aarnav 0 Feb 23, 2022
Learning a Little about Containerlab

Learning a Little about Containerlab Hello all. This is the respository based on this blog post. Getting Started Feel free to use this example. You wi

10 Oct 16, 2022
switching computer? changing your setup? You need to automate the download of your current setup? This is the right tool for you :incoming_envelope:

🔮 setup_shift(SS.py) switching computer? changing your setup? You need to automate the download of your current setup? This is the right tool for you

Mohamed Elfaleh 15 Aug 26, 2022
Streamlit component to display topics from Streamlit's community forum related to any exception.

streamlit-forum Streamlit component to display topics from Streamlit's community forum related to any exception. Installation pip install streamlit-fo

Snehan Kekre 7 Jul 15, 2022