Python function to construct an ODS spreadsheet on the fly - without having to store the entire file in memory or disk

Overview

stream-write-ods CircleCI Test Coverage

Python function to construct an ODS (OpenDocument Spreadsheet) on the fly - without having to store the entire file in memory or disk.

Can be used to convert CSV, SQLite, or JSON to ODS format.

Installation

pip install stream-write-ods

Usage

In general, pass a nested iterable to stream_write_ods and it will return an interable of bytes of the ODS file, as follows.

from stream_write_ods import stream_write_ods

def get_sheets():
    def get_rows_of_sheet_1():
        yield 'Value A', 'Value B'
        yield 'Value C', 'Value D'

    yield 'Sheet 1 name', ('col_1_name', 'col_2_name'), get_rows_of_sheet_1()

    def get_rows_of_sheet_2():
        yield 'col_1_value',

    yield 'Sheet 2 name', ('col_1_name',), get_rows_of_sheet_2()

ods_chunks = stream_write_ods(get_sheets())

Usage: Convert CSV to ODS

The following recipe converts a CSV to ODS.

import codecs
import csv
from stream_write_ods import stream_write_ods

# Any iterable that yields the bytes of a CSV file
# Hard coded for the purposes of this example
csv_bytes_iter = (
    b'col_1,col_2\n',
    b'1,"value"\n',
)

def get_sheets(sheet_name, csv_reader):
    yield sheet_name, next(csv_reader), csv_reader

csv_str_iter = codecs.iterdecode(csv_bytes_iter, 'utf-8')
csv_reader = csv.reader(csv_str_iter, csv.QUOTE_NONNUMERIC)
ods_chunks = stream_write_ods(get_sheets('Sheet 1', csv_reader))

Usage: Convert JSON to ODS

Using ijson to stream-parse a JSON file, it's possible to convert JSON data to ODS on the fly:

import ijson
import itertools
from stream_write_ods import stream_write_ods

# Any iterable that yields the bytes of a JSON file
# Hard coded for the purposes of this example
json_bytes_iter = (b'''{
  "data": [
      {"id": 1, "name": "Foo"},
      {"id": 2, "name": "Bar"}
  ]
}''',)

# ijson requires a file-like object
def to_file_like_obj(bytes_iter):
    chunk = b''
    offset = 0
    it = iter(bytes_iter)

    def up_to_iter(num):
        nonlocal chunk, offset

        while num:
            if offset == len(chunk):
                try:
                    chunk = next(it)
                except StopIteration:
                    break
                else:
                    offset = 0
            to_yield = min(num, len(chunk) - offset)
            offset = offset + to_yield
            num -= to_yield
            yield chunk[offset - to_yield:offset]

    class FileLikeObj:
        def read(self, n):
            return b''.join(up_to_iter(n))

    return FileLikeObj()

def get_sheets(json_file):
    columns = None

    def rows():
        nonlocal columns
        for item in ijson.items(json_file, 'data.item'):
            if columns is None:
                columns = list(item.keys())
            yield tuple(item[column] for column in columns)

    # Ensure columns populated
    rows_it = rows()
    first_row = next(rows_it)

    yield 'Sheet 1', columns, itertools.chain((first_row,), rows_it)

json_file = to_file_like_obj(json_bytes_iter)
ods_chunks = stream_write_ods(get_sheets(json_file))

Usage: Convert SQLite to ODS

SQLite isn't particularly streaming-friendly since typically you need random access to the file. But it's still possible to use stream-write-ods to convert SQLite to ODS.

import contextlib
import sqlite3
import tempfile
from stream_write_ods import stream_write_ods

@contextlib.contextmanager
def get_db():
    # Hard coded in memory database for the purposes of this example
    with sqlite3.connect(':memory:') as con:
        cur = con.cursor()
        cur.execute("CREATE TABLE my_table_a (my_col text);")
        cur.execute("CREATE TABLE my_table_b (my_col text);")
        cur.execute("INSERT INTO my_table_a VALUES ('Value A')")
        cur.execute("INSERT INTO my_table_b VALUES ('Value B')")
        yield con

def quote_identifier(value):
    return '"' + value.replace('"', '""') + '"'

def get_sheets(db):
    cur_table = db.cursor()
    cur_table.execute('''
        SELECT name FROM sqlite_master
        WHERE type = "table" AND name NOT LIKE 'sqlite\\_%' ESCAPE '\\'
    ''')
    cur_data = db.cursor()
    for table, in cur_table:
        cur_data.execute(f'SELECT * FROM {quote_identifier(table)} ORDER BY rowid')
        yield table, tuple(col[0] for col in cur_data.description), cur_data

with get_db() as db:
    ods_chunks = stream_write_ods(get_sheets(db))

Types

There are 8 possible data types in an Open Document Spreadsheet: boolean, currency, date, float, percentage, string, time, and void. 4 of these can be output by stream-write-ods, chosen automatically according to the following table.

Python type ODS type
boolean boolean
date date - without time component
datetime date - with time component
int float
float float
str string
NoneType string - as #NA

Limitations

ODS spreadsheets are essentially ZIP archives containing several member files. While in general ZIP archives can be up to 16EiB (exbibyte) in size using ZIP64, LibreOffice does not support ZIP64, and so ODS files are de-facto limited to 4GiB (gibibyte). This limit applies to the size of the entire compressed archive, the compressed size of each member file, and the uncompressed size of each member file.

Owner
Department for International Trade
Department for International Trade
Want to get your driver's license? Can't get a appointment because of COVID? Well I got a solution for you.

NJDMV-appoitment-alert Want to get your driver's license? Can't get a appointment because of COVID? Well I got a solution for you. We'll get you one i

Harris Spahic 3 Feb 04, 2022
E0 AI Bot is based on the message, it prints the answer with the highest probability using probability from the database.

E0 AI Chat Bot Based on the message, it prints the answer with the highest probability using probability from the database. Install on linux (Arch,Deb

Error 27 Dec 03, 2022
RDMAss - A Python Discord bot creating an interaction with RDM API

RDMAss A Python Discord bot creating an interaction with RDM API. Features Assig

5 Sep 21, 2022
Discord bot template.py

discord_bot_template.py A minimal and open-source discord.py boilerplate for kick-starting bot projects. I spend a lot of time developing bots for dif

Tarran Prior 1 Feb 24, 2022
Telegram Reporter

[Telegram Reporter v.3 ] 🇮🇷 AliCybeRR 🇮🇷 [ AliCybeRR.Reporter feature ] Login Your Telegram account 👽 support Termux ❕ No Limits ⚡ Secure 🔐 Free

AliCybeRR 1 Jun 08, 2022
A Python wrapper for the Yelp API v2

python-yelp-v2 A Python wrapper for the Yelp API v2. The structure for this was inspired by the python-twitter library, and some internal methods are

Matthew Conlen 12 Oct 24, 2017
Official python API for Phish.AI public and private API to detect zero-day phishing websites

phish-ai-api Summary Official python API for Phish.AI public and private API to detect zero-day phishing websites How it Works (TLDR) Essentially we h

Phish.AI 168 May 17, 2022
Discord Mafia Game Bot using nextcord

Mafia-Bot Discord Mafia Game Bot using nextcord Features Mafia Game Game Replays Installation Run the following command to install required modules: p

Nian 6 Nov 19, 2022
Python based Discord Bot with a simple music player

C32 Discord Bot Discord bot that plays music Table Of Contents About the Project Built With Acknowledgements About The Project Play music using the !p

Christopher Burwell 2 Oct 17, 2021
SaltConf21: Adding Workflow Approval to Salt

SaltConf21: Adding Workflow Approval to Salt Running To run the example, install Docker and docker-compose and run the following commands: docker-comp

SSYS Sistemas 4 Nov 24, 2021
A discord webhook client written in Python.

DiscordWebhook A discord webhook client written in Python. Installation pip install webhook-client Example from webhook_client import WebhookClient, E

Elijah 4 Nov 28, 2022
Drover is a command-line utility for deploying Python packages to Lambda functions.

drover drover: a command-line utility for deploying Python packages to Lambda functions. Background This utility aims to provide a simple, repeatable,

Jeffrey Wilges 4 May 19, 2021
Create Basic ERC20 token with Solidity, Brownie and Python

Create Basic ERC20 token with Solidity, Brownie and Python Demo Check out Cornell Token on Rinnkeby network with Etherscan. Installation Install brown

Ethan Huang 2 Feb 16, 2022
Scuttlecrab.py - Python Version of Scuttle Crab Bot

____ _ _ _ ____ _ / ___| ___ _ _| |_|

Fabrizo 4 Jul 08, 2022
My homeserver setup. Everything managed securely using Portainer.

homeserver-traefik-portainer Features: access all services with free TLS from letsencrypt using your own domain running a side project is super simple

Tomasz Wójcik 44 Jan 03, 2023
Its Is A Telegram Maths Basic Calculator Bot

Its Is A Telegram Maths Basic Calculator Bot

ANKIT KUMAR 1 Dec 26, 2021
✨ A Telegram mirror/leech bot By SparkXcloud Group ✨

SparkXcloud-Gdrive-MirrorBot SparkXcloud-Gdrive-MirrorBot is a multipurpose Telegram Bot writen in Python for mirroring files on the Internet to our b

119 Oct 23, 2022
Python script to download WAX transactions

WAXtax Python script to download WAX transactions WAXtax uses the CoinGecko API and the WAX Blockchain History API to download csvs for each account y

SixPM Software 11 Oct 09, 2022
Discord.py-Bot-Template - Discord Bot Template with Python 3.x

Discord Bot Template with Python 3.x This is a template for creating a custom Di

Keagan Landfried 3 Jul 17, 2022
Grocy-create-product - A script supports the batch creation of new products in Grocy

grocy-create-product This script supports the batch creation of new products in

André Heuer 6 Jul 28, 2022