A python app which aggregates and splits costs from multiple public cloud providers into a csv

Overview

Cloud Billing

This project aggregates the costs public cloud resources by accounts, services and tags by importing the invoices from public cloud providers and exporting their aggregates in CSV (or another format). It can be used for internal service brokers and allows for aggregating account / subscription / project costs for aws, azure and gcp and additionally allows for tag based aggregation.

It can run as a container and can schedule jobs through kubernetes cronjobs for monthly imports and exports.

It is not a graphical tool to get a drill down view over all costs (such as Azure Cost Management) but rather glue code between cloud provider invoices and internal invoicing systems (such as csv files based system, although other formats could added).

Sample Use Case

A team hosts a kubernetes service and an managed database on a public cloud provider, where multiple application teams run their apps. With this tool, cluster usage and databases can be imported and costs split between the app teams. Each app will be its own service and will receive its fair share of costs. Additionally, the platform team also receives an overview of their total costs. Billing setup is flexible and allows for fixed prices (e.g. each database price is X) and also flexible pricing based on tags and percentages (e.g. the cluster is invoiced for a resource with a tag plus a 10% fee for support).

Features

  • Multiple import options, amongst them azure, gcp and aws
  • Hashicorp Vault Integration, read secrets from Hashicorp Vault
  • Flexible billing with fixed prices and percentage-based prices
  • Allows invoicing of projects without tags
  • Informs service customers about service costs with a monthly email

Architecture

Architecture

Install & Run

Requirements

  • python3
  • pip
  • virtualenv
  • sqlite (or mysql for production)
  • mariadb-client

Install Requirements on Ubuntu

Some libraries for the DB connectors are required (mssql and mysql): sudo apt-get install mariadb-client libmariadbclient-dev unixodbc unixodbc-dev

Install Python Packages Requirements

pip install -r requirements.txt
pip install -r requirements-test.txt

Run locally with sqlite

# Import Sample Data from Azure from ./sample_files
python run.py -e env/dev.env import AzureImport -n azure

# Update Prices and Percentages
# Run command below and navigate to: http://localhost:5000/account/
python run.py -e ./env/dev.env run

# Export Accounts to local CSV and check the Azure.csv afterwards on the local filesystem
python run.py -e env/dev.env bill Azure -n bill_azure

# Update Services and attach them to a service customer
# Run below command and navigate to: http://localhost:5000/service/
# Set a a service customer in the service customer column and set a random string in the column left to it
python run.py -e ./env/dev.env run

# Now create an invoice csv file for the managed services and check the ManagedCloudServices.csv file on the local filesystem:
python run.py -e env/dev.env bill ManagedCloudServices -n bill_msc

Run Tests and Coverage

Tests are run with pytest and pymock. See /src/tests and /conftest.py for setup.

pytest
coverage run --source ./src -m pytest
coverage report

Configuration

Configuration is done within the env files in /env or through environment variables. Database settings are loaded in /src/settings.py.

Deployment

Ideally in a kubernetes environment where cronjobs can be created automatically from the interface. A Dockerfile can be found in the repository. Environment vars in ./env can also be added from configmaps within kubernetes.

See /k8s_manifests folder for sample deployment manifests. Setup is divided into:

  • _ns (Namespace)
  • _secret (Secrets for Connection to the Cloud Providers and the Mysql database)
  • _configmap (Configuration Details)
  • _deployment (Deployment of the app and ingresses and services)
Python USD rate in RUB parser

Python EUR and USD rate parser. Python USD and EUR rate in RUB parser. Parsing i

Andrew 2 Feb 17, 2022
Pyfunctools is a module that provides functions, methods and classes that help in the creation of projects in python

Pyfunctools Pyfunctools is a module that provides functions, methods and classes that help in the creation of projects in python, bringing functional

Natanael dos Santos Feitosa 5 Dec 22, 2022
Writing Alfred copy/paste scripts in Python

Writing Alfred copy/paste scripts in Python This repository shows how to create Alfred scripts in Python. It assumes that you using pyenv for Python v

Will Fitzgerald 2 Oct 26, 2021
A simple tool to extract python code from a Jupyter notebook, and then run pylint on it for static analysis.

Jupyter Pylinter A simple tool to extract python code from a Jupyter notebook, and then run pylint on it for static analysis. If you find this tool us

Edmund Goodman 10 Oct 13, 2022
🔩 Like builtins, but boltons. 250+ constructs, recipes, and snippets which extend (and rely on nothing but) the Python standard library. Nothing like Michael Bolton.

Boltons boltons should be builtins. Boltons is a set of over 230 BSD-licensed, pure-Python utilities in the same spirit as — and yet conspicuously mis

Mahmoud Hashemi 6k Jan 04, 2023
Go through a random file in your favourite open source projects!

Random Source Codes Never be bored again! Staring at your screen and just scrolling the great world wide web? Would you rather read through some code

Mridul Seth 1 Nov 03, 2022
A clock app, which helps you with routine tasks.

Clock This app helps you with routine tasks. Alarm Clock Timer Stop Watch World Time (Which city you want) About me Full name: Matin Ardestani Age: 14

Matin Ardestani 13 Jul 30, 2022
A Python class for checking the status of an enabled Minecraft server

mcstatus provides an easy way to query Minecraft servers for any information they can expose. It provides three modes of access (query, status and ping), the differences of which are listed below in

Nathan Adams 1.1k Jan 06, 2023
A simple package for handling variables in string.

A simple package for handling string variables. Welcome! This is a simple package for handling variables in string, You can add or remove variables wi

1 Dec 31, 2021
NFT-Generator is the best way to generate thousands of NFTs quick and easily with Python.

NFT-Generator is the best way to generate thousands of NFTs quick and easily with Python. Just add your files, set your configuration and run the scri

78 Dec 27, 2022
Abby's Left Hand Modifiers Dictionary

Abby's Left Hand Modifiers Dictionary Design This dictionary is inspired by and

12 Dec 08, 2022
.bvh to .mcfunction file converter.

bvh-to-mcf .bvh file to .mcfunction converter

Hanmin Kim 28 Nov 21, 2022
Basic loader is a small tool that will help you generating Cloudflare cookies

Basic Loader Cloudflare cookies loader This tool may help some people getting valide cloudflare cookies Installation 🔌 : pip install -r requirements.

IHateTomLrge 8 Mar 30, 2022
Tools for binary data on cassette

Micro Manchester Tape Storage Tools for storing binary data on cassette Includes: Python script for encoding Arduino sketch for decoding Eagle CAD fil

Zack Nelson 28 Dec 25, 2022
A collection of resources/tools and analyses for the angr binary analysis framework.

Awesome angr A collection of resources/tools and analyses for the angr binary analysis framework. This page does not only collect links and external r

105 Jan 02, 2023
Genart - Generate random art to sell as nfts

Genart - Generate random art to sell as nfts Usage git clone

Will 13 Mar 17, 2022
🌲 A simple BST (Binary Search Tree) generator written in python

Tree-Traversals (BST) 🌲 A simple BST (Binary Search Tree) generator written in python Installation Use the package manager pip to install BST. Usage

Jan Kupczyk 1 Dec 12, 2021
A set of Python scripts to surpass human limits in accomplishing simple tasks.

Human benchmark fooler Summary A set of Python scripts with Selenium designed to surpass human limits in accomplishing simple tasks available on https

Bohdan Dudchenko 3 Feb 10, 2022
A script copies movie and TV files to your GD drive, or create Hard Link in a seperate dir, in Emby-happy struct.

torcp A script copies movie and TV files to your GD drive, or create Hard Link in a seperate dir, in Emby-happy struct. Usage: python3 torcp.py -h Exa

ccf2012 105 Dec 22, 2022
Program to extract signatures from documents.

Extracting Signatures from Bank Checks Introduction Ahmed et al. [1] suggest a connected components-based method for segmenting signatures in document

Muhammad Saif Ullah Khan 9 Jan 26, 2022