Anomaly detection on SQL data warehouses and databases

Overview

CueObserve Logo

Test Coverage License


With CueObserve, you can run anomaly detection on data in your SQL data warehouses and databases.

CueObserve

Getting Started

Install via Docker

docker run -p 3000:80 cuebook/cueobserve

Now visit http://localhost:3000 in your browser.

How it works

You write a SQL GROUP BY query, map its columns as dimensions and measures, and save it as a virtual Dataset.

Dataset SQL

Dataset Schema Map

You then define one or more anomaly detection jobs on the dataset.

Anomaly Definition

When an anomaly detection job runs, CueObserve does the following:

  1. Executes the SQL GROUP BY query on your data warehouse and stores the result as a Pandas dataframe.
  2. Generates one or more timeseries from the dataframe, as defined in your anomaly detection job.
  3. Generates a forecast for each timeseries using Prophet.
  4. Creates a visual card for each timeseries. Marks the card as an anomaly if the last data point is anomalous.

Features

  • Automated SQL to timeseries transformation.
  • Run anomaly detection on the aggregate metric or break it down by any dimension.
  • In-built Scheduler. CueObserve uses Celery as the executor and celery-beat as the scheduler.
  • Slack alerts when anomalies are detected. (coming soon)
  • Monitoring. Slack alert when a job fails. CueObserve maintains detailed logs. (coming soon)

Limitations

  • Currently supports Prophet for timeseries forecasting.
  • Not being built for real-time anomaly detection on streaming data.

Support

For general help using CueObserve, read the documentation, or go to Github Discussions.

To report a bug or request a feature, open an issue.

Contributing

We'd love contributions to CueObserve. Before you contribute, please first discuss the change you wish to make via an issue or a discussion. Contributors are expected to adhere to our code of conduct.

Comments
  • hourly error

    hourly error

    Discussed in https://github.com/cuebook/CueObserve/discussions/75

    Originally posted by jithendra945 August 5, 2021 image Im getting this error, when i am trying to run anomaly definition.

    Im not getting why it is having None in it.

    bug observe 
    opened by sachinkbansal 8
  • Clickhouse as datasource support

    Clickhouse as datasource support

    Describe the solution you'd like I'd love to give CueObserve a try but our warehouse is currently in MS SQL Server.

    Describe the solution you'd like Add ClickHouse as a supported data source.

    I will wait when https://github.com/cuebook/CueObserve/issues/52 will resolve and try to implements PR

    enhancement connection 
    opened by Slach 4
  • Support SQL Server Data Source

    Support SQL Server Data Source

    Is your feature request related to a problem? Please describe. I'd love to give CueObserve a try but our warehouse is currently in MS SQL Server.

    Describe the solution you'd like Add SQL Server as a supported data source.

    Additional context I'd be interested in making the necessary pull request, but I'd like some high level advice on what might be needed.

    Is it as simple as adding the necessary sqlserver.py in https://github.com/cuebook/CueObserve/tree/main/api/dbConnections ?

    enhancement connection 
    opened by adam133 4
  • Schedule Tasks more that or equal 6 are being struck lifetime in celery queue

    Schedule Tasks more that or equal 6 are being struck lifetime in celery queue

    Describe the bug Cueobserve scheduled tasks are getting being struck in celery queue and not completing(No error is thrown)

    image

    To Reproduce Steps to reproduce the behavior:

    1. create 6 different anomalies definitions
    2. create 5 min cron interval & wait for 5min to start the scheduled tasks

    th0se tasks are not completing even after 1 day.

    for debugging, I have tried executing 5 scheduled tasks and 1 Scheduled task separately(different cron intervals), its is working fine, but when the scheduled tasks are 6 with same cron interval those got struck and didn't finish

    Expected behavior It should complete the 6 accepted tasks and pull the next tasks

    Thanks

    bug observe 
    opened by itsmesrds 3
  • Docker Network Mode and port mapping

    Docker Network Mode and port mapping

    Is your feature request related to a problem? Please describe. The docker-compose files provided sets everything with network mode "host". For various reason this setting does not work in my environment.

    Describe the solution you'd like

    • Capacity to use port mapping within the docker-compose files.
    • Capacity to use a Docker Swarm
    observe 
    opened by adrien-ferrara 2
  • docker-compose start error

    docker-compose start error

    image

    got the result:

    /usr/local/lib/python2.7/dist-packages/paramiko/transport.py:33: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in the next release. from cryptography.hazmat.backends import default_backend Traceback (most recent call last): File "/usr/local/bin/docker-compose", line 7, in from compose.cli.main import main File "/usr/local/lib/python2.7/dist-packages/compose/cli/main.py", line 24, in from ..config import ConfigurationError File "/usr/local/lib/python2.7/dist-packages/compose/config/init.py", line 6, in from .config import ConfigurationError File "/usr/local/lib/python2.7/dist-packages/compose/config/config.py", line 51, in from .validation import match_named_volumes File "/usr/local/lib/python2.7/dist-packages/compose/config/validation.py", line 12, in from jsonschema import Draft4Validator File "/usr/local/lib/python2.7/dist-packages/jsonschema/init.py", line 21, in from jsonschema._types import TypeChecker File "/usr/local/lib/python2.7/dist-packages/jsonschema/_types.py", line 3, in from pyrsistent import pmap File "/usr/local/lib/python2.7/dist-packages/pyrsistent/init.py", line 3, in from pyrsistent._pmap import pmap, m, PMap File "/usr/local/lib/python2.7/dist-packages/pyrsistent/_pmap.py", line 98 ) from e ^ SyntaxError: invalid syntax

    opened by wdongdongde 1
  • Support generic rest end point for notification

    Support generic rest end point for notification

    Discussed in https://github.com/cuebook/CueObserve/discussions/129

    Originally posted by pjpringle August 30, 2021 Not everyone has slack especially in the work place environments. Provide support to plug in rest calls to notify of anomalies.

    • [x] Image bytes included in response json
    enhancement observe 
    opened by sachinkbansal 1
  • Failing task due to incomplete data

    Failing task due to incomplete data

    Sometimes Anomaly detection job fails or the RCA job fails due to fewer data in a few buckets in that case we don't even show the results for the other buckets.

        if not allTasksSucceeded:
            runStatusObj.status = ANOMALY_DETECTION_ERROR
    

    We have added such checks to the code. But I think this is not a good thing to do as the user is not aware of the reason for the failure. Better is to skip this bucket and show the result of the rest of the buckets.

    opened by AakarSharmaHME 3
  • Different Priority alerts on different slack channel

    Different Priority alerts on different slack channel

    Is your feature request related to a problem? Please describe. There are different kinds of datasets. Anomalies on some are a P0 alert and on some, they are not that urgent. On the same dataset also a certain threshold needs immediate attention while others do not that much.

    Describe the solution you'd like While configuring settings for the slack channel we should be able to configure the different priorities and their slack channel id. Then while creating anomaly definition we should mention the priority of the anomaly definition. Just like we do for schedules.

    opened by AakarSharmaHME 5
  • Docker Image Build fails on Dev compose

    Docker Image Build fails on Dev compose

    Describe the bug Failed to create a docker image using docker-compose -f docker-compose-dev.yml up -d . Build fails with error Service 'cueo-backend' failed to build : Build failed

    To Reproduce Steps to reproduce the behavior:

    1. Git clone this repo
    2. Run docker-compose -f docker-compose-dev.yml up -d

    Expected behavior CueO components image to be built successfully and CueObserve to start running on localhost:3000

    Additional context Will add the entire log in the file or snippet below.

    observe 
    opened by shreyas-dev 4
  • pyarrow not installed

    pyarrow not installed

    Discussed in https://github.com/cuebook/CueObserve/discussions/187

    Originally posted by ravi-jam October 16, 2021 Hi Folks,

    My team just deployed CueObserve to EC2. I Got this error when we tried to add a dataset

    {“log”:“2021-10-16 06:15:14,232 [dbConnections.bigquery:63] ERROR Can’t connect to db with this credentials The pyarrow library is not installed, please install pyarrow to use the to_arrow() function.\n”,“stream”:“stderr”,“time”:“2021-10-16T06:15:14.237031452Z”}

    Any help would be appreciated.

    bug 
    opened by sachinkbansal 0
  • Handle scenario where dataset's last data point is complete and must not be ignored

    Handle scenario where dataset's last data point is complete and must not be ignored

    Discussed in https://github.com/cuebook/CueObserve/discussions/172

    Originally posted by PhilippeDo October 8, 2021 I have now this error when I tried to use prophet

    {"d2b9be2a-6d27-4e61-a577-3d99892e94e0": {"dimVal": null, "error": "{\"message\": \"single positional indexer is out-of-bounds\", \"stackTrace\": \"Traceback (most recent call last):\\n File \\\"/code/ops/tasks/anomalyDetection.py\\\", line 56, in anomalyService\\n result = detect(df, granularity, detectionRuleType, anomalyDef)\\n File \\\"/code/ops/tasks/anomalyDetection.py\\\", line 29, in detect\\n return prophetDetect(df, granularity)\\n File \\\"/code/ops/tasks/detectionTypes/prophet.py\\\", line 47, in prophetDetect\\n lastISO = df.iloc[-1][\\\"ds\\\"]\\n File \\\"/opt/venv/lib/python3.7/site-packages/pandas/core/indexing.py\\\", line 895, in __getitem__\\n return self._getitem_axis(maybe_callable, axis=axis)\\n File \\\"/opt/venv/lib/python3.7/site-packages/pandas/core/indexing.py\\\", line 1501, in _getitem_axis\\n self._validate_integer(key, axis)\\n File \\\"/opt/venv/lib/python3.7/site-packages/pandas/core/indexing.py\\\", line 1444, in _validate_integer\\n raise IndexError(\\\"single positional indexer is out-of-bounds\\\")\\nIndexError: single positional indexer is out-of-bounds\\n\"}", "success": false}}

    I used the following in DATASET section

    select DATE_FORMAT(date, '%Y-%m-%d %H') as Date,
    pagelt from pageloadtime2
    

    and then when I run, the dataset looks liks this. It is a minimal dataset withou dimension, with only a measure and a timestamp

    image

    enhancement anomalyDefinition 
    opened by sachinkbansal 0
Releases(v0.3.2)
MySQLdb is a Python DB API-2.0 compliant library to interact with MySQL 3.23-5.1 (unofficial mirror)

==================== MySQLdb Installation ==================== .. contents:: .. Prerequisites ------------- + Python 2.3.4 or higher * http://ww

Sébastien Arnaud 17 Oct 10, 2021
Python MYSQL CheatSheet.

Python MYSQL CheatSheet Python mysql cheatsheet. Install Required Windows(WAMP) Download and Install from HERE Linux(LAMP) install packages. sudo apt

Mohammad Dori 4 Jul 15, 2022
Database connection pooler for Python

Nimue Strange women lying in ponds distributing swords is no basis for a system of government! --Dennis, Peasant Nimue is a database connection pool f

1 Nov 09, 2021
New generation PostgreSQL database adapter for the Python programming language

Psycopg 3 -- PostgreSQL database adapter for Python Psycopg 3 is a modern implementation of a PostgreSQL adapter for Python. Installation Quick versio

The Psycopg Team 880 Jan 08, 2023
Import entity definition document into SQLie3. Manage the entity. Also, create a "Create Table SQL file".

EntityDocumentMaker Version 1.00 After importing the entity definition (Excel file), store the data in sqlite3. エンティティ定義(Excelファイル)をインポートした後、データをsqlit

G-jon FujiYama 1 Jan 09, 2022
A framework based on tornado for easier development, scaling up and maintenance

turbo 中文文档 Turbo is a framework for fast building web site and RESTFul api, based on tornado. Easily scale up and maintain Rapid development for RESTF

133 Dec 06, 2022
An extension package of 🤗 Datasets that provides support for executing arbitrary SQL queries on HF datasets

datasets_sql A 🤗 Datasets extension package that provides support for executing arbitrary SQL queries on HF datasets. It uses DuckDB as a SQL engine

Mario Šaško 19 Dec 15, 2022
A simple password manager I typed with python using MongoDB .

Python with MongoDB A simple python code example using MongoDB. How do i run this code • First of all you need to have a python on your computer. If y

31 Dec 06, 2022
A Python DB-API and SQLAlchemy dialect to Google Spreasheets

Note: shillelagh is a drop-in replacement for gsheets-db-api, with many additional features. You should use it instead. If you're using SQLAlchemy all

Beto Dealmeida 185 Jan 01, 2023
A fast PostgreSQL Database Client Library for Python/asyncio.

asyncpg -- A fast PostgreSQL Database Client Library for Python/asyncio asyncpg is a database interface library designed specifically for PostgreSQL a

magicstack 5.8k Dec 31, 2022
aiosql - Simple SQL in Python

aiosql - Simple SQL in Python SQL is code. Write it, version control it, comment it, and run it using files. Writing your SQL code in Python programs

Will Vaughn 1.1k Jan 08, 2023
Example Python codes that works with MySQL and Excel files (.xlsx)

Python x MySQL x Excel by Zinglecode Example Python codes that do the processes between MySQL database and Excel spreadsheet files. YouTube videos MyS

Potchara Puttawanchai 1 Feb 07, 2022
edaSQL is a library to link SQL to Exploratory Data Analysis and further more in the Data Engineering.

edaSQL is a python library to bridge the SQL with Exploratory Data Analysis where you can connect to the Database and insert the queries. The query results can be passed to the EDA tool which can giv

Tamil Selvan 8 Dec 12, 2022
A Python Object-Document-Mapper for working with MongoDB

MongoEngine Info: MongoEngine is an ORM-like layer on top of PyMongo. Repository: https://github.com/MongoEngine/mongoengine Author: Harry Marr (http:

MongoEngine 3.9k Jan 08, 2023
Asynchronous interface for peewee ORM powered by asyncio

peewee-async Asynchronous interface for peewee ORM powered by asyncio. Important notes Since version 0.6.0a only peewee 3.5+ is supported If you still

05Bit 666 Dec 30, 2022
SQL for Humans™

Records: SQL for Humans™ Records is a very simple, but powerful, library for making raw SQL queries to most relational databases. Just write SQL. No b

Kenneth Reitz 6.9k Jan 07, 2023
dbd is a database prototyping tool that enables data analysts and engineers to quickly load and transform data in SQL databases.

dbd: database prototyping tool dbd is a database prototyping tool that enables data analysts and engineers to quickly load and transform data in SQL d

Zdenek Svoboda 47 Dec 07, 2022
google-cloud-bigtable Apache-2google-cloud-bigtable (🥈31 · ⭐ 3.5K) - Google Cloud Bigtable API client library. Apache-2

Python Client for Google Cloud Bigtable Google Cloud Bigtable is Google's NoSQL Big Data database service. It's the same database that powers many cor

Google APIs 39 Dec 03, 2022
A SQL linter and auto-formatter for Humans

The SQL Linter for Humans SQLFluff is a dialect-flexible and configurable SQL linter. Designed with ELT applications in mind, SQLFluff also works with

SQLFluff 5.5k Jan 08, 2023
Python client for InfluxDB

InfluxDB-Python InfluxDB-Python is a client for interacting with InfluxDB. Development of this library is maintained by: Github ID URL @aviau (https:/

InfluxData 1.6k Dec 24, 2022