Business Intelligence (BI) in Python, OLAP

Overview

Open Mining

Build Status - Circle CI https://coveralls.io/repos/github/mining/mining/badge.svg?branch=master Code Health

OpenMining

Business Intelligence (BI) Application Server written in Python

Requirements

  • Python 2.7 (Backend)
  • Lua 5.2 or LuaJIT 5.1 (OML backend)
  • MongoDB (Admin)
  • Redis (Queue and DataWarehouse)
  • Bower (Install frontend libs, NodeJS depends)

Install dependencies

$ sudo apt-get install mongodb-10gen redis-server nodejs nodejs-dev npm
$ npm install bower

If you use Mac OSX you can install all dependencies using HomeBrew.

Install Open Mining

Clone the repository

$ git clone [email protected]:mining/mining.git

Install python and bower dependencies using make command

$ make build

FAQ

If mongodb or redis-server problems

Install mongodb and redis-server, make sure it running

Supported databases

For example, to connect to a PostgreSQL database make sure you install a driver like psycopg2. OpenMining supports all databases that the underlying ORM SQLAlchemy supports.

See the SQLAlchemy documentation for more info about drivers and connection strings.

Run

python manage.py runserver
python manage.py celery
python manage.py scheduler

Running Demo

Make sure runserver still running when run 'build_demo' command.

python manage.py runserver
python manage.py build_demo

And now you can login with: username 'admin' and password 'admin'.

Screenshots

Dashboard OpenMining

Dashboard OpenMining

Dashboard Charts OpenMining

Dashboard Charts OpenMining

Dashboard Charts OpenMining

Dashboard Charts OpenMining

Dashboard Widgets OpenMining

Dashboard Widgets OpenMining

Late Scheduler and running Cubes OpenMining

Late Scheduler and running Cubes OpenMining

Contribute

Join us on IRC at #openmining on freenode (web access).

Credits

Authors: Avelino and UP! Essência

Many thanks to all the contributors!

License

Licensed under the MIT license (see the (LICENSE file).

Comments
  • bower install error: bower angular-ui-select2-sortable#~0.0.2

    bower install error: bower angular-ui-select2-sortable#~0.0.2

    bower angular-ui-select2-sortable#~0.0.2 EMALFORMED Failed to read /var/folders/r2/zjhxzwfx01gctj5bvnk3qw3w0000gn/T/leroy/bower/angular-ui-select2-sortable-60805-yH6IkR/bower.json

    opened by lerrua 10
  • mining.ini within installed package

    mining.ini within installed package

    When build the mining for production, it doesn't copy the mining.ini file within the package.

    Suggestions:

    • Under setup(), include a static session "data_files" and add "mining.ini" to this.
    • A different default place for the conf files for production installs, that doesn't depend from PROJECT_PATH variable
    • Other?
    opened by ayr-ton 4
  • MS Sql server not connect

    MS Sql server not connect

    File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/dialects/mssql/pymssql.py", line 72, in _get_server_version_info r"Microsoft SQL Server.*? - (\d+).(\d+).(\d+).(\d+)", vers) File "/usr/lib/python2.7/re.py", line 137, in match return _compile(pattern, flags).match(string) TypeError: expected string or buffer

    bug enhancement 
    opened by abuvanth 3
  • fatal error: Python.h: No such file or directory

    fatal error: Python.h: No such file or directory

    when i run make build shows

    mining/utils/_pandas.c:4:20: fatal error: Python.h: No such file or directory
     #include "Python.h"
                        ^
    compilation terminated.
    error: command 'gcc' failed with exit status 1
    make: *** [environment] Error 1
    

    what is meaning?

    opened by Susheldon 2
  • [WIP] Configure Dockerfile and docker-compose.yml

    [WIP] Configure Dockerfile and docker-compose.yml

    This is a work in progress

    I'm working on docker configuration of mining project. I used the Makefile as guide for building steps. We can't just RUN Makefile because that way Docker is unable to cache each step of building.

    I stumbled in some issues:

    • [ ] Two different ways of building the project (Makefile and Dockerfile)
    • [ ] Every time the code is changed setup.py will run again, which turn the development dull
    • [ ] python manage.py build_demo is not working
    • [ ] Add docker instructions in README

    Any idea how to solve this problems?

    I saw there is a separate repo for Dockerfile. Maybe I need to make this PR there?

    opened by gbmoretti 2
  • No such file or directory: 'mining/bin/demo/base.sql' on running build_demo command

    No such file or directory: 'mining/bin/demo/base.sql' on running build_demo command

    Guys,

    After following a first installation step-by-step I'm getting this error on running build_demo command.

    $ python manage.py build_demo
    OpenMining load demo system
    Traceback (most recent call last):
      File "manage.py", line 118, in <module>
        cmds(default_map=default_map)
      File "/Users/leroy/.virtualenvs/mining/lib/python2.7/site-packages/click/core.py", line 610, in __call__
        return self.main(*args, **kwargs)
      File "/Users/leroy/.virtualenvs/mining/lib/python2.7/site-packages/click/core.py", line 590, in main
        rv = self.invoke(ctx)
      File "/Users/leroy/.virtualenvs/mining/lib/python2.7/site-packages/click/core.py", line 936, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/Users/leroy/.virtualenvs/mining/lib/python2.7/site-packages/click/core.py", line 782, in invoke
        return ctx.invoke(self.callback, **ctx.params)
      File "/Users/leroy/.virtualenvs/mining/lib/python2.7/site-packages/click/core.py", line 416, in invoke
        return callback(*args, **kwargs)
      File "manage.py", line 112, in build_demo
        build(level)
      File "/Users/leroy/.virtualenvs/mining/lib/python2.7/site-packages/mining-0.2.0-py2.7-macosx-10.10-x86_64.egg/mining/bin/demo/build_admin.py", line 18, in build
        f = open('{}'.format(os.path.join(demo_path, 'base.sql')), 'r')
    IOError: [Errno 2] No such file or directory: '/Users/leroy/.virtualenvs/mining/lib/python2.7/site-packages/mining-0.2.0-py2.7-macosx-10.10-x86_64.egg/mining/bin/demo/base.sql'
    
    opened by lerrua 2
  • Remove use of the frontend

    Remove use of the frontend

    The Open Mining will have "N" interface, we need to remove the mandatory use of the frontend and the installer put an option to select the front-end will be used.

    feature backend 
    opened by avelino 2
  • Export to xls big data

    Export to xls big data

    Traceback (most recent call last):
      File "/home/mining/.virtualenvs/mining/local/lib/python2.7/site-packages/bottle.py", line 862, in _handle
        return route.call(**args)
      File "/home/mining/.virtualenvs/mining/local/lib/python2.7/site-packages/bottle_mongo.py", line 160, in wrapper
        rv = callback(*a, **ka)
      File "/home/mining/mining/controllers/export.py", line 72, in data
        df.to_excel(file_name)
      File "/home/mining/.virtualenvs/mining/local/lib/python2.7/site-packages/pandas/core/frame.py", line 1204, in to_excel
        startrow=startrow, startcol=startcol)
      File "/home/mining/.virtualenvs/mining/local/lib/python2.7/site-packages/pandas/io/excel.py", line 653, in write_cells
        val, style)
      File "/home/mining/.virtualenvs/mining/local/lib/python2.7/site-packages/xlwt/Worksheet.py", line 1030, in write
        self.row(r).write(c, label, style)
      File "/home/mining/.virtualenvs/mining/local/lib/python2.7/site-packages/xlwt/Worksheet.py", line 1078, in row
        self.__rows[indx] = self.Row(indx, self)
      File "/home/mining/.virtualenvs/mining/local/lib/python2.7/site-packages/xlwt/Row.py", line 42, in __init__
        raise ValueError("row index (%r) not an int in range(65536)" % rowx)
    ValueError: row index (65536) not an int in range(65536)
    
    bug frontend 
    opened by yuripiratello 2
  • python manage.py runserver

    python manage.py runserver

    Traceback (most recent call last): File "manage.py", line 24, in from mining.bin.scheduler import scheduler_app File "D:\Python27\Lib\site-packages\gevent\builtins.py", line 93, in import result = import(*args, **kwargs) File "C:\Users\Documents\Visual Studio 2012\Projects\PythonProject\mining\mining\bin\scheduler.py", line 13, in log_it("START", "bin-scheduler") File "C:\Users\Documents\Visual Studio 2012\Projects\PythonProject\mining\mining\utils_init.py", line 50, in log_it with open("/tmp/openmining-{}.log".format(name), "a") as log:IOError: [Errno 2] No such file or directory: '/tmp/openmining-bin-scheduler.log

    opened by MORYis 0
  • Installation error: IOError: [Errno 2] No such file or directory: 'requirements.txt'

    Installation error: IOError: [Errno 2] No such file or directory: 'requirements.txt'

    Hi,

    I am trying to install it in Mac OS. And, I am getting error. Following are my command line readings:

    pip install mining Collecting mining Using cached https://files.pythonhosted.org/packages/7a/b6/8288426c4aa010222619105c3649b31a7814dde17f4d5383a29046d24a10/mining-0.2.0.tar.gz Complete output from command python setup.py egg_info: Traceback (most recent call last): File "", line 1, in File "/private/var/folders/rz/vcghnnsn59q032827bpqnvvm0000gn/T/pip-install-oiZray/mining/setup.py", line 7, in REQUIREMENTS = [i.strip() for i in open("requirements.txt").readlines() IOError: [Errno 2] No such file or directory: 'requirements.txt'

    ----------------------------------------
    

    Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/rz/vcghnnsn59q032827bpqnvvm0000gn/T/pip-install-oiZray/mining/

    Please help.

    Regards

    opened by MiCodes2 3
  • Firefox can’t establish a connection to the server at 0.0.0.0:8888

    Firefox can’t establish a connection to the server at 0.0.0.0:8888

    [email protected]:~/work/mining# docker run 10f825960da2
    OpenMining start server at: 0.0.0.0:8888
    OpenMining start server at: 0.0.0.0:8888
    Bottle v0.12.7 server starting up (using GeventWebSocketServer())...
    Listening on http://0.0.0.0:8888/
    Hit Ctrl-C to quit.
    

    But I can't open the url:

    Unable to connect
    
    Firefox can’t establish a connection to the server at 0.0.0.0:8888.
    
        The site could be temporarily unavailable or too busy. Try again in a few moments.
        If you are unable to load any pages, check your computer’s network connection.
        If your computer or network is protected by a firewall or proxy, make sure that Firefox is permitted to access the Web.
    
    bug 
    opened by gclsoft 1
Releases(0.2.2)
Owner
Open Mining
Business Intelligence (BI) in Python (Pandas web interface)
Open Mining
A computer algebra system written in pure Python

SymPy See the AUTHORS file for the list of authors. And many more people helped on the SymPy mailing list, reported bugs, helped organize SymPy's part

SymPy 9.9k Dec 31, 2022
The official repository for ROOT: analyzing, storing and visualizing big data, scientifically

About The ROOT system provides a set of OO frameworks with all the functionality needed to handle and analyze large amounts of data in a very efficien

ROOT 2k Dec 29, 2022
Implementation in Python of the reliability measures such as Omega.

OmegaPy Summary Simple implementation in Python of the reliability measures: Omega Total, Omega Hierarchical and Omega Hierarchical Total. Name Link O

Rafael Valero Fernández 2 Apr 27, 2022
simple way to build the declarative and destributed data pipelines with python

unipipeline simple way to build the declarative and distributed data pipelines. Why you should use it Declarative strict config Scaffolding Fully type

aliaksandr-master 0 Jan 26, 2022
Full ELT process on GCP environment.

Rent Houses Germany - GCP Pipeline Project: The goal of the project is to extract data about house rentals in Germany, store, process and analyze it u

Felipe Demenech Vasconcelos 2 Jan 20, 2022
Created covid data pipeline using PySpark and MySQL that collected data stream from API and do some processing and store it into MYSQL database.

Created covid data pipeline using PySpark and MySQL that collected data stream from API and do some processing and store it into MYSQL database.

2 Nov 20, 2021
Python package for analyzing behavioral data for Brain Observatory: Visual Behavior

Allen Institute Visual Behavior Analysis package This repository contains code for analyzing behavioral data from the Allen Brain Observatory: Visual

Allen Institute 16 Nov 04, 2022
A CLI tool to reduce the friction between data scientists by reducing git conflicts removing notebook metadata and gracefully resolving git conflicts.

databooks is a package for reducing the friction data scientists while using Jupyter notebooks, by reducing the number of git conflicts between different notebooks and assisting in the resolution of

dataroots 86 Dec 25, 2022
Describing statistical models in Python using symbolic formulas

Patsy is a Python library for describing statistical models (especially linear models, or models that have a linear component) and building design mat

Python for Data 866 Dec 16, 2022
Tuplex is a parallel big data processing framework that runs data science pipelines written in Python at the speed of compiled code

Tuplex is a parallel big data processing framework that runs data science pipelines written in Python at the speed of compiled code. Tuplex has similar Python APIs to Apache Spark or Dask, but rather

Tuplex 791 Jan 04, 2023
Python package for analyzing sensor-collected human motion data

Python package for analyzing sensor-collected human motion data

Simon Ho 71 Nov 05, 2022
Scraping and analysis of leetcode-compensations page.

Leetcode compensations report Scraping and analysis of leetcode-compensations page.

utsav 96 Jan 01, 2023
Projeto para realizar o RPA Challenge . Utilizando Python e as bibliotecas Selenium e Pandas.

RPA Challenge in Python Projeto para realizar o RPA Challenge (www.rpachallenge.com), utilizando Python. O objetivo deste desafio é criar um fluxo de

Henrique A. Lourenço 1 Apr 12, 2022
Implementation in Python of the reliability measures such as Omega.

reliabiliPy Summary Simple implementation in Python of the [reliability](https://en.wikipedia.org/wiki/Reliability_(statistics) measures for surveys:

Rafael Valero Fernández 2 Apr 27, 2022
Airflow ETL With EKS EFS Sagemaker

Airflow ETL With EKS EFS & Sagemaker (en desarrollo) Diagrama de la solución Imp

1 Feb 14, 2022
Binance Kline Data With Python

Binance Kline Data by seunghan(gingerthorp) reference https://github.com/binance/binance-public-data/ All intervals are supported: 1m, 3m, 5m, 15m, 30

shquant 5 Jul 13, 2022
University Challenge 2021 With Python

University Challenge 2021 This repository contains: The TeX file of the technical write-up describing the University / HYPER Challenge 2021 under late

2 Nov 27, 2021
Get mutations in cluster by querying from LAPIS API

Cluster Mutation Script Get mutations appearing within user-defined clusters. Usage Clusters are defined in the clusters dict in main.py: clusters = {

neherlab 1 Oct 22, 2021
Udacity - Data Analyst Nanodegree - Project 4 - Wrangle and Analyze Data

WeRateDogs Twitter Data from 2015 to 2017 Udacity - Data Analyst Nanodegree - Project 4 - Wrangle and Analyze Data Table of Contents Introduction Proj

Keenan Cooper 1 Jan 12, 2022
Time ranges with python

timeranges Time ranges. Read the Docs Installation pip timeranges is available on pip: pip install timeranges GitHub You can also install the latest v

Micael Jarniac 2 Sep 01, 2022