Python function to construct an ODS spreadsheet on the fly - without having to store the entire file in memory or disk

Overview

stream-write-ods CircleCI Test Coverage

Python function to construct an ODS (OpenDocument Spreadsheet) on the fly - without having to store the entire file in memory or disk.

Can be used to convert CSV, SQLite, or JSON to ODS format.

Installation

pip install stream-write-ods

Usage

In general, pass a nested iterable to stream_write_ods and it will return an interable of bytes of the ODS file, as follows.

from stream_write_ods import stream_write_ods

def get_sheets():
    def get_rows_of_sheet_1():
        yield 'Value A', 'Value B'
        yield 'Value C', 'Value D'

    yield 'Sheet 1 name', ('col_1_name', 'col_2_name'), get_rows_of_sheet_1()

    def get_rows_of_sheet_2():
        yield 'col_1_value',

    yield 'Sheet 2 name', ('col_1_name',), get_rows_of_sheet_2()

ods_chunks = stream_write_ods(get_sheets())

Usage: Convert CSV to ODS

The following recipe converts a CSV to ODS.

import codecs
import csv
from stream_write_ods import stream_write_ods

# Any iterable that yields the bytes of a CSV file
# Hard coded for the purposes of this example
csv_bytes_iter = (
    b'col_1,col_2\n',
    b'1,"value"\n',
)

def get_sheets(sheet_name, csv_reader):
    yield sheet_name, next(csv_reader), csv_reader

csv_str_iter = codecs.iterdecode(csv_bytes_iter, 'utf-8')
csv_reader = csv.reader(csv_str_iter, csv.QUOTE_NONNUMERIC)
ods_chunks = stream_write_ods(get_sheets('Sheet 1', csv_reader))

Usage: Convert JSON to ODS

Using ijson to stream-parse a JSON file, it's possible to convert JSON data to ODS on the fly:

import ijson
import itertools
from stream_write_ods import stream_write_ods

# Any iterable that yields the bytes of a JSON file
# Hard coded for the purposes of this example
json_bytes_iter = (b'''{
  "data": [
      {"id": 1, "name": "Foo"},
      {"id": 2, "name": "Bar"}
  ]
}''',)

# ijson requires a file-like object
def to_file_like_obj(bytes_iter):
    chunk = b''
    offset = 0
    it = iter(bytes_iter)

    def up_to_iter(num):
        nonlocal chunk, offset

        while num:
            if offset == len(chunk):
                try:
                    chunk = next(it)
                except StopIteration:
                    break
                else:
                    offset = 0
            to_yield = min(num, len(chunk) - offset)
            offset = offset + to_yield
            num -= to_yield
            yield chunk[offset - to_yield:offset]

    class FileLikeObj:
        def read(self, n):
            return b''.join(up_to_iter(n))

    return FileLikeObj()

def get_sheets(json_file):
    columns = None

    def rows():
        nonlocal columns
        for item in ijson.items(json_file, 'data.item'):
            if columns is None:
                columns = list(item.keys())
            yield tuple(item[column] for column in columns)

    # Ensure columns populated
    rows_it = rows()
    first_row = next(rows_it)

    yield 'Sheet 1', columns, itertools.chain((first_row,), rows_it)

json_file = to_file_like_obj(json_bytes_iter)
ods_chunks = stream_write_ods(get_sheets(json_file))

Usage: Convert SQLite to ODS

SQLite isn't particularly streaming-friendly since typically you need random access to the file. But it's still possible to use stream-write-ods to convert SQLite to ODS.

import contextlib
import sqlite3
import tempfile
from stream_write_ods import stream_write_ods

@contextlib.contextmanager
def get_db():
    # Hard coded in memory database for the purposes of this example
    with sqlite3.connect(':memory:') as con:
        cur = con.cursor()
        cur.execute("CREATE TABLE my_table_a (my_col text);")
        cur.execute("CREATE TABLE my_table_b (my_col text);")
        cur.execute("INSERT INTO my_table_a VALUES ('Value A')")
        cur.execute("INSERT INTO my_table_b VALUES ('Value B')")
        yield con

def quote_identifier(value):
    return '"' + value.replace('"', '""') + '"'

def get_sheets(db):
    cur_table = db.cursor()
    cur_table.execute('''
        SELECT name FROM sqlite_master
        WHERE type = "table" AND name NOT LIKE 'sqlite\\_%' ESCAPE '\\'
    ''')
    cur_data = db.cursor()
    for table, in cur_table:
        cur_data.execute(f'SELECT * FROM {quote_identifier(table)} ORDER BY rowid')
        yield table, tuple(col[0] for col in cur_data.description), cur_data

with get_db() as db:
    ods_chunks = stream_write_ods(get_sheets(db))

Types

There are 8 possible data types in an Open Document Spreadsheet: boolean, currency, date, float, percentage, string, time, and void. 4 of these can be output by stream-write-ods, chosen automatically according to the following table.

Python type ODS type
boolean boolean
date date - without time component
datetime date - with time component
int float
float float
str string
NoneType string - as #NA

Limitations

ODS spreadsheets are essentially ZIP archives containing several member files. While in general ZIP archives can be up to 16EiB (exbibyte) in size using ZIP64, LibreOffice does not support ZIP64, and so ODS files are de-facto limited to 4GiB (gibibyte). This limit applies to the size of the entire compressed archive, the compressed size of each member file, and the uncompressed size of each member file.

Owner
Department for International Trade
Department for International Trade
OAN Music - Highly advanced User Music Bot

ཧᜰ꙰ꦿ➢𝐎𝐀𝐍༒☛ 🎧 Advanced 𝐎𝐀𝐍 Music bot. 🔗 𝐏𝐨𝐰𝐞𝐫𝐞𝐝 𝐛𝐲 : ➢𝐀ttitude

Attitude king 5 Feb 25, 2022
Tracks twitter spaces and sends it to a discord webhook.

Tracks twitter spaces and sends it to a discord webhook. Uses the twitter api to find twitter spaces and then the m3u8 url for the space is found using selenium and will have it posted using a discor

Sam Phung 20 Dec 17, 2022
Library to manage your own custom RPC on your desktop

Info I don't recommend novices setting this up yourself. It requires Redis, a server to host the API on, and a bit of understanding of Windows & Pytho

Isaac K 1 Apr 16, 2022
A cool discord bot, called Fifi

Fifi A cool discord bot, called Fifi This bot is the official server bot of Meme Studios discord server. This github repo is the code we use for the b

Fifi Discord Bot 3 Jun 08, 2021
Telegram vc userbot

Telegram Vc Userbot Available Commands /ping :- To check whether userbot is up or not /joinvc :- To join vc /leavevc :- To leave vc /join_group :- To

NandyDark 7 Nov 18, 2022
A free sniper bot built to work with PancakeSwap: Router V2

Pancakeswap Sniper Bot PancakeSwap sniper bot. Automated sniping bot to snipe crypto coin launches. How it works The sniping bot can be used in three

89 Aug 06, 2022
Cedric Owens 16 Sep 27, 2022
Poll-Bot Repo For Telegram #telegram

Intro This Is A Simple Bot To Create Poll In Channel and Groups And Also This Is our First Project Too.. Enter you tokens at these are very important

BotsUniverse 6 Oct 21, 2022
A Discord bot made by QwertyIsCoding

QwertyBot QwertyBot A Discord bot made by QwertyIsCoding Explore the docs » View Demo . Report Bug . Request Feature About The Project This Discord bo

4 Oct 08, 2022
this is an op music pyrogram music bot.

amanrajputpytgcallmusic this is an op music pyrogram music bot..... this bot user music bot can play music without being admin...... TG-MusicPlayer A

2 Dec 27, 2021
Add Reactions to your Channel Posts!

• Shaaak - Post Reaction Bot Simple and Minimalistic telegram bot to add Reactions and Comments to your Channel Posts! - What's Unique About it?

Harsh Raj 4 Jan 31, 2022
PyMusic Player is a music player written in python3.

PyMusic Player is a music player written in python3. It harvests r

PythonSerious 2 Jan 30, 2022
Телеграм бот решающий задания ЦДЗ, написанный на библиотеке libmesh.

MESHBot-Telegram Телеграм бот решающий задания ЦДЗ. Описание: Бот написан с использованием библиотеки libmesh. Для начала работы отправьте ему ссылку

2 Jun 19, 2022
Python functions to run WASS stereo wave processing executables, and load and post process WASS output files.

wass-pyfuns Python functions to run the WASS stereo wave processing executables, and load and post process the WASS output files. General WASS (Waves

Mika Malila 3 May 13, 2022
Checks instagram names to see if they're available

How to install You must have python 3.7.6 installed and make sure you click the 'ADD TO PATH' option when installing Open cmd and type pip install aio

2 Oct 20, 2021
Discord Mass Report script that uses multiple tokens

Discord-Mass-Report Discord Mass Report script that uses multiple tokens, full credits to https://github.com/hoki0/Discord-mass-report who made it in

cChimney 4 Jun 08, 2022
A bot to playing music in telegram vcg

Idzeroid Music|| Idzeroid Music adalah sebuah repository music bot telegram untuk memainkan suara di voice chat group anda. Fyi This repo im using for

idzeroid 1 Oct 26, 2021
Fetch fund data from avanza.se using Python and some web scraping with bs4

Py(A)vanza Fetch fund data from avanza.se using Python and some web scraping with bs4. The default way is to display the data in the terminal, apply -

dunderrrrrr 1 Jan 27, 2022
This package allows interactions with the BuyCoins API.

The BuyCoins Python library allows interactions with the BuyCoins API from applications written in Python.

Abdulazeez Abdulazeez Adeshina 45 May 23, 2022
修改自SharpNoPSExec的基于python的横移工具 A Lateral Movement Tool Learned From SharpNoPSExec -- Twitter: @juliourena

PyNoPSExec A Lateral Movement Tool Learned From SharpNoPSExec -- Twitter: @juliourena 根据@juliourena大神的SharpNOPsExec项目改写的横向移动工具 Platform(平台): Windows 1

<a href=[email protected]"> 23 Nov 09, 2022