Python function to construct an ODS spreadsheet on the fly - without having to store the entire file in memory or disk

Overview

stream-write-ods CircleCI Test Coverage

Python function to construct an ODS (OpenDocument Spreadsheet) on the fly - without having to store the entire file in memory or disk.

Can be used to convert CSV, SQLite, or JSON to ODS format.

Installation

pip install stream-write-ods

Usage

In general, pass a nested iterable to stream_write_ods and it will return an interable of bytes of the ODS file, as follows.

from stream_write_ods import stream_write_ods

def get_sheets():
    def get_rows_of_sheet_1():
        yield 'Value A', 'Value B'
        yield 'Value C', 'Value D'

    yield 'Sheet 1 name', ('col_1_name', 'col_2_name'), get_rows_of_sheet_1()

    def get_rows_of_sheet_2():
        yield 'col_1_value',

    yield 'Sheet 2 name', ('col_1_name',), get_rows_of_sheet_2()

ods_chunks = stream_write_ods(get_sheets())

Usage: Convert CSV to ODS

The following recipe converts a CSV to ODS.

import codecs
import csv
from stream_write_ods import stream_write_ods

# Any iterable that yields the bytes of a CSV file
# Hard coded for the purposes of this example
csv_bytes_iter = (
    b'col_1,col_2\n',
    b'1,"value"\n',
)

def get_sheets(sheet_name, csv_reader):
    yield sheet_name, next(csv_reader), csv_reader

csv_str_iter = codecs.iterdecode(csv_bytes_iter, 'utf-8')
csv_reader = csv.reader(csv_str_iter, csv.QUOTE_NONNUMERIC)
ods_chunks = stream_write_ods(get_sheets('Sheet 1', csv_reader))

Usage: Convert JSON to ODS

Using ijson to stream-parse a JSON file, it's possible to convert JSON data to ODS on the fly:

import ijson
import itertools
from stream_write_ods import stream_write_ods

# Any iterable that yields the bytes of a JSON file
# Hard coded for the purposes of this example
json_bytes_iter = (b'''{
  "data": [
      {"id": 1, "name": "Foo"},
      {"id": 2, "name": "Bar"}
  ]
}''',)

# ijson requires a file-like object
def to_file_like_obj(bytes_iter):
    chunk = b''
    offset = 0
    it = iter(bytes_iter)

    def up_to_iter(num):
        nonlocal chunk, offset

        while num:
            if offset == len(chunk):
                try:
                    chunk = next(it)
                except StopIteration:
                    break
                else:
                    offset = 0
            to_yield = min(num, len(chunk) - offset)
            offset = offset + to_yield
            num -= to_yield
            yield chunk[offset - to_yield:offset]

    class FileLikeObj:
        def read(self, n):
            return b''.join(up_to_iter(n))

    return FileLikeObj()

def get_sheets(json_file):
    columns = None

    def rows():
        nonlocal columns
        for item in ijson.items(json_file, 'data.item'):
            if columns is None:
                columns = list(item.keys())
            yield tuple(item[column] for column in columns)

    # Ensure columns populated
    rows_it = rows()
    first_row = next(rows_it)

    yield 'Sheet 1', columns, itertools.chain((first_row,), rows_it)

json_file = to_file_like_obj(json_bytes_iter)
ods_chunks = stream_write_ods(get_sheets(json_file))

Usage: Convert SQLite to ODS

SQLite isn't particularly streaming-friendly since typically you need random access to the file. But it's still possible to use stream-write-ods to convert SQLite to ODS.

import contextlib
import sqlite3
import tempfile
from stream_write_ods import stream_write_ods

@contextlib.contextmanager
def get_db():
    # Hard coded in memory database for the purposes of this example
    with sqlite3.connect(':memory:') as con:
        cur = con.cursor()
        cur.execute("CREATE TABLE my_table_a (my_col text);")
        cur.execute("CREATE TABLE my_table_b (my_col text);")
        cur.execute("INSERT INTO my_table_a VALUES ('Value A')")
        cur.execute("INSERT INTO my_table_b VALUES ('Value B')")
        yield con

def quote_identifier(value):
    return '"' + value.replace('"', '""') + '"'

def get_sheets(db):
    cur_table = db.cursor()
    cur_table.execute('''
        SELECT name FROM sqlite_master
        WHERE type = "table" AND name NOT LIKE 'sqlite\\_%' ESCAPE '\\'
    ''')
    cur_data = db.cursor()
    for table, in cur_table:
        cur_data.execute(f'SELECT * FROM {quote_identifier(table)} ORDER BY rowid')
        yield table, tuple(col[0] for col in cur_data.description), cur_data

with get_db() as db:
    ods_chunks = stream_write_ods(get_sheets(db))

Types

There are 8 possible data types in an Open Document Spreadsheet: boolean, currency, date, float, percentage, string, time, and void. 4 of these can be output by stream-write-ods, chosen automatically according to the following table.

Python type ODS type
boolean boolean
date date - without time component
datetime date - with time component
int float
float float
str string
NoneType string - as #NA

Limitations

ODS spreadsheets are essentially ZIP archives containing several member files. While in general ZIP archives can be up to 16EiB (exbibyte) in size using ZIP64, LibreOffice does not support ZIP64, and so ODS files are de-facto limited to 4GiB (gibibyte). This limit applies to the size of the entire compressed archive, the compressed size of each member file, and the uncompressed size of each member file.

Owner
Department for International Trade
Department for International Trade
API Basica per a synologys Active Backup For Buissiness

Synology Active Backup for Business API-NPP Informació Per executar el programa

Nil Pujol 0 May 13, 2022
Bancos de Dados Relacionais (SQL) na AWS com Amazon RDS.

Bancos de Dados Relacionais (SQL) na AWS com Amazon RDS Explorando o Amazon RDS, um serviço de provisionamente e gerenciamento de banco de dados relac

Lucas Magalhães 1 Dec 05, 2021
Auto Join: A GitHub action script to automatically invite everyone to the organization who comment at the issue page.

Auto Invite To Org By Issue Comment A GitHub action script to automatically invite everyone to the organization who comment at the issue page. What is

Max Base 6 Jun 08, 2022
A python crypto trading bot on Binance using RSI in 25 Lines 🚀

RSI Crypto Trading Bot - Binance A Crypto Trading Bot on Binance trading BTCUSDT and ETHUSDT using RSI in 25 Lines of Code Getting Started Note Python

Blankly Finance 10 Dec 26, 2022
A script written in python3 for bruteforcing Gmail accounts.

GmailBruteforce Made for bruteforcing gmail accounts. It needs Less Secure Apps setting turned on in order to work. Installation For windows git clone

Shinero 4 Sep 16, 2022
Automatic SystemVerilog linting in github actions with the help of Verible

Verible Lint Action Usage See action.yml This is a GitHub Action used to lint Verilog and SystemVerilog source files and comment erroneous lines of co

CHIPS Alliance 10 Dec 26, 2022
Easy-apply-bot - A LinkedIn Easy Apply bot to help with my job search.

easy-apply-bot A LinkedIn Easy Apply bot to help with my job search. Getting Started First, clone the repository somewhere onto your computer, or down

Matthew Alunni 5 Dec 09, 2022
A self-bot for discord, written in Python, which will send you notifications to your desktop if it detects an intruder on your discord server

A self-bot for discord, written in Python, which will send you notifications to your desktop if it detects an intruder on your discord server

LevPrav 1 Jan 11, 2022
A basic template for Creating Odoo Module

Odoo ERP Boilerplate A basic template for Creating Odoo Module. Folders inside this repository consist of snippet code and a module example. Folders w

Altela Eleviansyah Pramardhika 1 Feb 06, 2022
This is a telegram bot built using the Oxford Dictionary API

Oxford Dictionaries Telegram Bot This is a telegram bot built using the Oxford Dictionary API Source: Oxford Dictionaries API Documentation Install En

Abhijith N T 2 Mar 18, 2022
👾 Telegram Smart Group Assistant 🤖

DarkHelper 🌖 Features ⚡️ Smart anti-apam & anti-NFSW message checker Tag Members , Entertain facility , Welcommer ban , unban , mute , unmute , lock

amirali rajabi 38 Dec 18, 2022
A python SDK for interacting with quantum devices on Amazon Braket

Amazon Braket Python SDK The Amazon Braket Python SDK is an open source library that provides a framework that you can use to interact with quantum co

Amazon Web Services 213 Dec 14, 2022
Visualize size of directories, s3 buckets.

Dir Sizer This is a work in progress, right now consider this an Alpha or Proof of Concept level. dir_sizer is a utility to visualize the size of a di

Scott Seligman 13 Dec 08, 2022
Custom bot I've made to host events on my personal Discord server.

discord_events Custom bot I've made to host events on my personal Discord server. You can try the bot out in my personal server here: https://discord.

AlexFlipnote 5 Mar 16, 2022
Download videos from Youtube and other platforms through a Telegram Bot

ytdl-bot Download videos from YouTube and other platforms through a Telegram Bot Usage: https://t.me/benny_ytdlbot Send link from YouTube directly to

Telegram Bot Collection 289 Jan 03, 2023
Bot developed in python, 100% open-source, compatible with Windows and Linux.

Bombcrypto Bot [Family JOW] Bot desenvolvido em python, 100% do código é aberto, para aqueles que tenham conhecimento validarem que não existe nenhum

Renato Maia 71 Dec 20, 2022
Telegram Bot Repo Capable of fetching the following Info via Anilist API inspired from AniFluid and Nepgear

Telegram Bot Repo Capable of fetching the following Info via Anilist API inspired from AniFluid and Nepgear Anime Airing Manga Character Scheduled Top

Rikka-Chan 2 Apr 01, 2022
A Python script that exports users from one Telegram group to another using one or more concurrent user bots.

ExportTelegramUsers A Python script that exports users from one Telegram group to another using one or more concurrent user bots. Make sure to set all

Fasil Minale 17 Jun 26, 2022
Select random winners for a Twitter giveaway

twitter_picker Select random winners for a Twitter giveaway Once the Twitter giveaway (or airdrop) is closed, assign a number to each participant. The

Michael Rawner 1 Dec 11, 2021
An interactive App to play with Spotify data, both from the Spotify Web API and from CSV datasets.

An interactive App to play with Spotify data, both from the Spotify Web API and from CSV datasets.

Caio Lang 3 Jan 24, 2022