Build a better understanding of your data in PostgreSQL.

Overview

Data Fluent for PostgreSQL

license

Build a better understanding of your data in PostgreSQL.

The following shows an example report generated by this tool. It gives the numbers of rows, columns, bytes as well as human-friendly size counts for each table within a given PostgreSQL database.

The Metrics Report

The following shows the row count for every column that represents a date grouped by year and month.

The Time Distribution Report

Installation

On Ubuntu 20:

$ wget -qO- \
    https://www.postgresql.org/media/keys/ACCC4CF8.asc \
        | sudo apt-key add -
$ echo "deb http://apt.postgresql.org/pub/repos/apt/ xenial-pgdg main" \
    | sudo tee /etc/apt/sources.list.d/pgdg.list

$ sudo apt update
$ sudo apt install \
    git \
    python3-pip \
    python3-virtualenv \
    postgresql-13 \
    postgresql-client-13 \
    postgresql-contrib

On macOS:

$ brew install \
    git \
    postgresql \
    virtualenv

Then, regardless of platform, setup a virtual environment and install the following Python-based dependencies.

$ virtualenv ~/.fluency
$ source ~/.fluency/bin/activate
$ python3 -m pip install \
    csvkit \
    humanfriendly \
    ipython \
    openpyxl \
    Pandas \
    psycopg2-binary \
    typer \
    xlsxwriter

Then clone this repo as it contains the datafluent_pg.py script:

$ git clone https://github.com/marklit/datafluent_pg.git ~/datafluent_pg
$ cd ~/datafluent_pg

Example Analysis

Clone fivethirtyeight's data repo. It has a large number of CSV-formatted datasets.

$ git clone https://github.com/fivethirtyeight/data.git ~/538data

Make sure you can access a PostgreSQL database on your machine. Here I'm creating an intel database for the mark user on my Ubuntu 20 machine.

$ sudo -u postgres \
    bash -c "psql -c \"CREATE USER mark
                       WITH PASSWORD 'test'
                       SUPERUSER;\""

With PostgreSQL access setup, create a database called intel.

$ createdb intel

I'll import one of the datasets within fivethirtyeight's repo. Note, because the dates within this dataset are not formatted in YYYY-MM-DD format, I needed to override the format so that the MM/DD/YYYY format would be read properly.

$ csvsql --db postgresql:///intel \
         --insert ~/538data/congress-generic-ballot/generic_topline_historical.csv \
         --datetime-format="%m/%d/%Y"

I'll run the Excel Report Generator:

$ python datafluent_pg.py

This will result in a fluency.xlsx file being produced with two worksheets: Metrics and Time Distributions.

If you need to override any parameters, please refer to the documentation:

$ python datafluent_pg.py --help
Usage: datafluent_pg.py [OPTIONS]

Options:
  --pg-dns TEXT                   [default: postgresql://localhost:5432/intel]
  --output TEXT                   [default: fluency.xlsx]
  --install-completion [bash|zsh|fish|powershell|pwsh]
                                  Install completion for the specified shell.
  --show-completion [bash|zsh|fish|powershell|pwsh]
                                  Show completion for the specified shell, to
                                  copy it or customize the installation.

  --help                          Show this message and exit.
Make a command interpreter that manages AirBnb objects

AirBnB Clone Project Description This is part 1 of our AirBnb Clone project. The purpose of this project is to make a command interpreter that manages

Firdaus H. Salim 1 Nov 14, 2021
Autofilter with imdb bot || broakcast , imdb poster and imdb rating

LuciferMoringstar_Robot How To Deploy Video Subscribe YouTube Channel Added Features Imdb posters for autofilter. Imdb rating for autofilter. Custom c

Muhammed 127 Dec 29, 2022
A Telegram Bot Plays With Words!!!

TheWordzBot ➠ I Can Turn Text Into Audio ➠ I Can Get Results From Dictionary ➠ I Can Make Google Search For You ➠ I Can Suggest Strong Passwords For Y

RAVEEN KUMAR 8 Feb 28, 2022
NekoRobot-2 - Neko is An Anime themed advance Telegram group management bot.

NekoRobot A modular telegram Python bot running on python3 with an sqlalchemy, mongodb database. ╒═══「 Status 」 Maintained Support Group Included Free

Lovely Boy 19 Nov 12, 2022
Análise de dados abertos do programa Taxigov.

Análise de dados do Taxigov Este repositório contém os cadernos Jupyter usados no projeto de análise de dados do Taxigov. Conjunto de dados O conjunto

Augusto Herrmann 1 Jan 10, 2022
SmartFile API Client (Python).

A SmartFile Open Source project. Read more about how SmartFile uses and contributes to Open Source software. Summary This library includes two API cli

SmartFile 19 Jan 11, 2022
Ein Bot, der Dokumente bei Studydrive möglichst immer online hält

studydrive-altklausuren-bot Ein Bot, der Dokumente bei Studydrive möglichst immer online hält geplante Features: Multiaccount supprt: Es gibt mehrere

henrydatei 1 Jun 04, 2022
Telegram anime bot that uses Anilist API

Telegram Bot Repo Capable of fetching the following Info via Anilist API inspired from AniFluid and Nepgear Anime Airing Manga Character Scheduled Top

Lucky Jain 71 Jan 03, 2023
Kang Sticker bot

Kang Sticker Bot A simple Telegram bot which creates sticker packs from other stickers, images, documents and URLs. Based on kangbot Deploy Credits: s

Hafitz Setya 11 Jan 02, 2023
Administration Panel for Control FiveM Servers From Discord

FiveM Discord Administration Panel Version 1.0.0 If you would like to report an issue or request a feature. Join our Discord or create an issue. Contr

NIma 9 Jun 17, 2022
Automate coin farming for dankmemer. Unlimited accounts at once. Uses a proxy

dankmemer-farm Simple script to farm Dankmemer coins with multiple accounts at once. Requires: Proxies, Discord Tokens Disclaimer I don't take respons

Scobra 12 Dec 20, 2022
This repository contains free labs for setting up an entire workflow and DevOps environment from a real-world perspective in AWS

DevOps-The-Hard-Way-AWS This tutorial contains a full, real-world solution for setting up an environment that is using DevOps technologies and practic

Mike Levan 1.6k Jan 05, 2023
Maubot azuracast - A maubot to fetch data from your radio station

Maubot Azuracast A maubot to fetch data from your radio station Setup Configure

3 Mar 14, 2022
An advanced QR Code telegram bot with more features.

QR Code Bot A telegram qr code encode and decode bot Advanced Features 1. Database ( MongoDB ) Support 2. Broadcast Support 3. Status Command 4. Setti

Fayas Noushad 16 Nov 12, 2022
Tools to download and aggregate feeds of vaccination clinic location information in the United States.

vaccine-feed-ingest Pipeline for ingesting nationwide feeds of vaccine facilities. Contributing How to Configure your environment (instructions on the

Call the Shots 26 Aug 05, 2022
Anti-corruption-bot - Anti corruption bot with python

anti-corruption-bot Test API (running via Flask) is currently hosted at https://

Richard Bankole 2 Feb 16, 2022
SpotPlay2YouPlay - Converts new additions to a Spotify playlist to a matching Youtube playlist

SpotPlay2YouPlay - Converts new additions to a Spotify playlist to a matching Youtube playlist, can also be configured to converting whole playlists with the refresh fun

9 Mar 06, 2022
Presentation and code files for the talk at PyCon Indonesia

pycon-indonesia Presentation and code files for the talk at PyCon Indonesia. Files used for the PyCon Indonesia presentation. [Directory Includes:] Be

Neeraj Pandey 2 Dec 04, 2021
HackZ-Token-Grabber-V2 - HackZ Token Grabber V2

HackZ-Token-Grabber-V2 was made by Love ❌ code ✅ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ 🌟

! ™NightMare 2 Mar 01, 2022
Automatically pick a winner who Retweeted, Commented, and Followed your Twitter account!

AutomaticTwitterGiveaways automates selecting winners for "Retweet, Comment, Follow" type Twitter giveaways.

1 Jan 13, 2022