BREP : Binary Search in plaintext and gzip files

Last update: Dec 24, 2021

Related tags

Overview

BREP : Binary Search in plaintext and gzip files

Search large files in O(log n) time using binary search.
We support plaintext and Gzipped files.

Benchmark : 8x faster than `grep` on a 2GB dataset !

brep is usually faster than grep for >1GB datasets.

Check tests/benchmark.py to reproduce the results.

grep ^777 test.txt : 1.594 s (15 runs)
brep 777 test.txt : 206.8 ms (15 runs)

Installation

pip install brep or pip install . from this repo

Index your file

In order to conduct binary search, your file needs to be sorted.
We recommend GNU sort, as it's multithreaded and supports large files.
LC_ALL=C sort -u -o output_file input_file

BREP supports compressed file in the GZIP format.
We recommend pigz for quick multicore compression : pigz file

Usage

Provide 1 prefix search term and 1 filepath
brep 77777 test/large.gz

You can also search from our Python class

from brep import Search

for result in Search("77777", "test/large.gz"):
    print(result)

Contribute

PRs are welcome!

Install dev dependencies: pip install -e .[dev]
Test and lint before submitting: pytest && flake8

Todo

Reimplement in Rust
Faster gz size estimation
Search multiple strings at once

dotsend is a web application which helps you to upload your large files and share file via link

0 Dec 3, 2022

A wrapper for DVD file structure and ISO files.

vs-parsedvd DVDs were an error. A wrapper for DVD file structure and ISO files. You can find me in the IEW Discord server

7 Nov 17, 2022

Python Fstab Generator is a small Python script to write and generate /etc/fstab files based on yaml file on Unix-like systems.

PyFstab Generator PyFstab Generator is a small Python script to write and generate /etc/fstab files based on yaml file on Unix-like systems. NOTE : Th

2 Nov 9, 2021

A tool for batch processing large fasta files and accompanying metadata table to upload to repositories via API

Fasta Uploader A tool for batch processing large fasta files and accompanying metadata table to repositories via API The python fasta_uploader.py scri

Centre for Infectious Disease and One Health

1 Dec 9, 2021

Python interface for reading and appending tar files

Python interface for reading and appending tar files, while keeping a fast index for finding and reading files in the archive. This interface has been

1 Nov 12, 2021

A simple file module for creating, editing and saving files.

1 Nov 25, 2021

This program can help you to move and rename many files at once

This program can help you to rename and save many files in a folder in seconds, but don't give the same name to files, it can delete both files.

1 Oct 10, 2022

Fast Python reader and editor for ASAM MDF / MF4 (Measurement Data Format) files

asammdf is a fast parser and editor for ASAM (Association for Standardization of Automation and Measuring Systems) MDF (Measurement Data Format) files

440 Dec 31, 2022

Here is some Python code that allows you to read in SVG files and approximate their paths using a Fourier series.

Here is some Python code that allows you to read in SVG files and approximate their paths using a Fourier series. The Fourier series can be animated and visualized, the function can be output as a two dimensional vector for Desmos and there is a method to output the coefficients as LaTeX code.

12 Jan 1, 2023

BREP : Binary Search in plaintext and gzip files

Related tags

Overview

BREP : Binary Search in plaintext and gzip files

Benchmark : 8x faster than grep on a 2GB dataset !

Installation

Index your file

Usage

Contribute

Todo

You might also like...

dotsend is a web application which helps you to upload your large files and share file via link

A wrapper for DVD file structure and ISO files.

Python Fstab Generator is a small Python script to write and generate /etc/fstab files based on yaml file on Unix-like systems.

A tool for batch processing large fasta files and accompanying metadata table to upload to repositories via API

Python interface for reading and appending tar files

A simple file module for creating, editing and saving files.

This program can help you to move and rename many files at once

Fast Python reader and editor for ASAM MDF / MF4 (Measurement Data Format) files

Here is some Python code that allows you to read in SVG files and approximate their paths using a Fourier series.

Releases(v1.0.1)

v1.0.1(Nov 7, 2021)

Owner

Arnaud de Saint Meloir

A tiny Python library for writing multi-channel TIFF stacks.

Publicly Open Amazon AWS S3 Bucket Viewer

Nmap XML output to CSV and HTTP/HTTPS URLS.

A python script to convert an ucompressed Gnucash XML file to a text file for Ledger and hledger.

CSV To VCF (Multiples en un archivo)

An object-oriented approach to Python file/directory operations.

Python's Filesystem abstraction layer

File support for asyncio

Swiss army knife for Apple's .tbd file manipulation

Python function to construct a ZIP archive with on the fly - without having to store the entire ZIP in memory or disk

Python's Filesystem abstraction layer

A tool for batch processing large fasta files and accompanying metadata table to upload to repositories via API

Python Fstab Generator is a small Python script to write and generate /etc/fstab files based on yaml file on Unix-like systems.

Two scripts help you to convert csv file to md file by template

Remove [x]_ from StudIP zip Archives and archive_filelist.csv completely

This is a junk file creator tool which creates junk files in Internal Storage

Sheet Data Image/PDF-to-CSV Converter

Various technical documentation, in electronically parseable format

Ini adalah program python untuk mengubah background foto dalam 1 folder, tidak perlu satu satu

Better directory iterator and faster os.walk(), now in the Python 3.5 stdlib

Benchmark : 8x faster than `grep` on a 2GB dataset !