Python function to construct an ODS spreadsheet on the fly - without having to store the entire file in memory or disk

Overview

stream-write-ods CircleCI Test Coverage

Python function to construct an ODS (OpenDocument Spreadsheet) on the fly - without having to store the entire file in memory or disk.

Can be used to convert CSV, SQLite, or JSON to ODS format.

Installation

pip install stream-write-ods

Usage

In general, pass a nested iterable to stream_write_ods and it will return an interable of bytes of the ODS file, as follows.

from stream_write_ods import stream_write_ods

def get_sheets():
    def get_rows_of_sheet_1():
        yield 'Value A', 'Value B'
        yield 'Value C', 'Value D'

    yield 'Sheet 1 name', ('col_1_name', 'col_2_name'), get_rows_of_sheet_1()

    def get_rows_of_sheet_2():
        yield 'col_1_value',

    yield 'Sheet 2 name', ('col_1_name',), get_rows_of_sheet_2()

ods_chunks = stream_write_ods(get_sheets())

Usage: Convert CSV to ODS

The following recipe converts a CSV to ODS.

import codecs
import csv
from stream_write_ods import stream_write_ods

# Any iterable that yields the bytes of a CSV file
# Hard coded for the purposes of this example
csv_bytes_iter = (
    b'col_1,col_2\n',
    b'1,"value"\n',
)

def get_sheets(sheet_name, csv_reader):
    yield sheet_name, next(csv_reader), csv_reader

csv_str_iter = codecs.iterdecode(csv_bytes_iter, 'utf-8')
csv_reader = csv.reader(csv_str_iter, csv.QUOTE_NONNUMERIC)
ods_chunks = stream_write_ods(get_sheets('Sheet 1', csv_reader))

Usage: Convert JSON to ODS

Using ijson to stream-parse a JSON file, it's possible to convert JSON data to ODS on the fly:

import ijson
import itertools
from stream_write_ods import stream_write_ods

# Any iterable that yields the bytes of a JSON file
# Hard coded for the purposes of this example
json_bytes_iter = (b'''{
  "data": [
      {"id": 1, "name": "Foo"},
      {"id": 2, "name": "Bar"}
  ]
}''',)

# ijson requires a file-like object
def to_file_like_obj(bytes_iter):
    chunk = b''
    offset = 0
    it = iter(bytes_iter)

    def up_to_iter(num):
        nonlocal chunk, offset

        while num:
            if offset == len(chunk):
                try:
                    chunk = next(it)
                except StopIteration:
                    break
                else:
                    offset = 0
            to_yield = min(num, len(chunk) - offset)
            offset = offset + to_yield
            num -= to_yield
            yield chunk[offset - to_yield:offset]

    class FileLikeObj:
        def read(self, n):
            return b''.join(up_to_iter(n))

    return FileLikeObj()

def get_sheets(json_file):
    columns = None

    def rows():
        nonlocal columns
        for item in ijson.items(json_file, 'data.item'):
            if columns is None:
                columns = list(item.keys())
            yield tuple(item[column] for column in columns)

    # Ensure columns populated
    rows_it = rows()
    first_row = next(rows_it)

    yield 'Sheet 1', columns, itertools.chain((first_row,), rows_it)

json_file = to_file_like_obj(json_bytes_iter)
ods_chunks = stream_write_ods(get_sheets(json_file))

Usage: Convert SQLite to ODS

SQLite isn't particularly streaming-friendly since typically you need random access to the file. But it's still possible to use stream-write-ods to convert SQLite to ODS.

import contextlib
import sqlite3
import tempfile
from stream_write_ods import stream_write_ods

@contextlib.contextmanager
def get_db():
    # Hard coded in memory database for the purposes of this example
    with sqlite3.connect(':memory:') as con:
        cur = con.cursor()
        cur.execute("CREATE TABLE my_table_a (my_col text);")
        cur.execute("CREATE TABLE my_table_b (my_col text);")
        cur.execute("INSERT INTO my_table_a VALUES ('Value A')")
        cur.execute("INSERT INTO my_table_b VALUES ('Value B')")
        yield con

def quote_identifier(value):
    return '"' + value.replace('"', '""') + '"'

def get_sheets(db):
    cur_table = db.cursor()
    cur_table.execute('''
        SELECT name FROM sqlite_master
        WHERE type = "table" AND name NOT LIKE 'sqlite\\_%' ESCAPE '\\'
    ''')
    cur_data = db.cursor()
    for table, in cur_table:
        cur_data.execute(f'SELECT * FROM {quote_identifier(table)} ORDER BY rowid')
        yield table, tuple(col[0] for col in cur_data.description), cur_data

with get_db() as db:
    ods_chunks = stream_write_ods(get_sheets(db))

Types

There are 8 possible data types in an Open Document Spreadsheet: boolean, currency, date, float, percentage, string, time, and void. 4 of these can be output by stream-write-ods, chosen automatically according to the following table.

Python type ODS type
boolean boolean
date date - without time component
datetime date - with time component
int float
float float
str string
NoneType string - as #NA

Limitations

ODS spreadsheets are essentially ZIP archives containing several member files. While in general ZIP archives can be up to 16EiB (exbibyte) in size using ZIP64, LibreOffice does not support ZIP64, and so ODS files are de-facto limited to 4GiB (gibibyte). This limit applies to the size of the entire compressed archive, the compressed size of each member file, and the uncompressed size of each member file.

Owner
Department for International Trade
Department for International Trade
An Advanced Python Playing Card Module that makes creating playing card games simple and easy!

playingcards.py An Advanced Python Playing Card Module that makes creating playing card games simple and easy! Features Easy to Understand Class Objec

Blake Potvin 5 Aug 30, 2022
UP It is a script to notify of a new update of your project, done in python and using GitHub, to modify the versions to notify users.

UP-Updater UP It is a script to notify of a new update of your project, done in python and using GitHub, to modify the versions to notify users. Requi

Made in 4 Oct 28, 2021
Discord-selfbot - Very basic discord self bot

discord-selfbot Very basic discord self bot still being actively developed requi

nana 4 Apr 07, 2022
โ๐“๐ก๐ž ๐Œ๐จ๐ฌ๐ญ ๐๐จ๐ฐ๐ž๐ซ๐Ÿ๐ฎ๐ฅ๐ฅ ๐†๐ซ๐จ๐ฎ๐ฉ ๐Œ๐š๐ง๐š๐ ๐ž๐ฆ๐ž๐ง๐ญ ๐๐จ๐ญโž

โ๐“๐ก๐ž ๐Œ๐จ๐ฌ๐ญ ๐๐จ๐ฐ๐ž๐ซ๐Ÿ๐ฎ๐ฅ๐ฅ ๐†๐ซ๐จ๐ฎ๐ฉ ๐Œ๐š๐ง๐š๐ ๐ž๐ฆ๐ž๐ง๐ญ ๐๐จ๐ญโž

Abdisamad Omar Mohamed 5 Jun 24, 2022
A demo without ๐Ÿš€ science, just simple UTXO spending logic.

Stuck TX Demo Docker container that runs 4 dogecoind to demonstrate "the stuck tx problem". Scenario A wallet sends out 3 transactions to a recipient

Patrick Lodder 2 Nov 16, 2021
Busty - A bot for the Busty Discord server

Busty Discord bot used for the Busty server. Install You'll need at least Python

Andrew Morgan 7 Dec 05, 2022
Companion "receiver" to matrix-appservice-webhooks for [matrix].

Matrix Webhook Receiver Companion "receiver" to matrix-appservice-webhooks for [matrix]. The purpose of this app is to listen for generic webhook mess

Kim Brose 13 Sep 29, 2022
Soundcloud Music Downloader

Soundcloud Music Downloader Description This script is able to download music from SoundCloud and set id3tag to the downloaded music. Compatible with

Ronan 2.6k Jan 01, 2023
SquireBot is a Discord bot designed to run and manage tournaments entirely within a Discord.

Overview SquireBot is a Discord bot designed to run and manage tournaments entirely within a Discord. The current intended usecase is Magic: the Gathe

7 Nov 29, 2022
The Research PACS on AWS solution facilitates researchers' access medical images stored in the clinical PACS in a secure and seamless manner

Research PACS on AWS Challenge to solve Solution presentation Deploy the solution Further reading Releases License Challenge to solve The rise of new

AWS Samples 23 Sep 09, 2022
Mega.nz to GDrive uploader

Mega.nz to GDrive uploader With this telegram bot you can download files from mega.nz and upload those files or telegram uploaded files to GDrive. You

30 Nov 13, 2022
A python package that allows you to place automated trades using the TD Ameritrade API.

Template Repo Table of Contents Overview Setup Usage Support These Projects Overview Setup Setup - Requirements Install:* For this particular project,

Alex Reed 4 Jan 25, 2022
First Party data integration solution built for marketing teams to enable audience and conversion onboarding into Google Marketing products (Google Ads, Campaign Manager, Google Analytics).

Megalista Sample integration code for onboarding offline/CRM data from BigQuery as custom audiences or offline conversions in Google Ads, Google Analy

Google 76 Dec 29, 2022
Male' Map Telegram Bot

Male' Map TelegramBot A simple TelegramBot to fetch residential addresses in Male', Maldives. The bot can be queried inline or directly. sample .env f

Naail Abdul Rahman 12 Nov 25, 2022
This discord bot will help you to control your target through PickleC2

PickleC2-Bot This discord bot will help you to control your target through PickleC2 WHAT's a PickleC2? PickleC2 is a simple C2 framework written in py

4 Jun 25, 2022
Discord Voice Call DoS

VC DoS Simple, effective Discord DM/GC voice call Denial of Service. How to Use & FAQ 1. Download the script (obviously). 2. In CMD prompt, find the l

Roover 4 Feb 28, 2022
TuShare is a utility for crawling historical data of China stocks

TuShare Tushare Pro็‰ˆๅทฒๅ‘ๅธƒ๏ผŒ่ฏท่ฎฟ้—ฎๆ–ฐ็š„ๅฎ˜็ฝ‘ไบ†่งฃๅ’ŒๆŸฅ่ฏขๆ•ฐๆฎๆŽฅๅฃ๏ผ https://tushare.pro TuShareๆ˜ฏๅฎž็Žฐๅฏน่‚ก็ฅจ/ๆœŸ่ดง็ญ‰้‡‘่žๆ•ฐๆฎไปŽๆ•ฐๆฎ้‡‡้›†ใ€ๆธ…ๆด—ๅŠ ๅทฅ ๅˆฐ ๆ•ฐๆฎๅญ˜ๅ‚จ่ฟ‡็จ‹็š„ๅทฅๅ…ท๏ผŒๆปก่ถณ้‡‘่ž้‡ๅŒ–ๅˆ†ๆžๅธˆๅ’Œๅญฆไน ๆ•ฐๆฎๅˆ†ๆž็š„ไบบๅœจๆ•ฐๆฎ่Žทๅ–ๆ–น้ข็š„้œ€ๆฑ‚๏ผŒๅฎƒ็š„็‰น็‚นๆ˜ฏๆ•ฐๆฎ่ฆ†็›–่Œƒๅ›ดๅนฟ๏ผŒๆŽฅๅฃ

ๆŒ–ๅœฐๅ…” 11.9k Dec 30, 2022
IACR Events Scraper

IACR Events Scraper This scrapes https://iacr.org/events/ and exports it as a calendar file. I host a version of this for myself under https://arrrr.c

Karolin Varner 6 May 28, 2022
ClassesMD5-64 - Get whatsapp md5 code using python

Hello Installation Clone Repo & install bash $ git clone https://github.com/Pito

PitoDev 1 Jan 03, 2022
Fully undetected auto skillcheck hack for dead by daylight that works decently well

Auto-skillcheck was made by Love โŒ code โœ… โ” ใƒปHow to use Start off by installing python ofc Open cmd in the same directory and type pip install -r requ

Rdimo 10 Aug 13, 2022