Python Package for DataHerb: create, search, and load datasets.

Last update: Feb 11, 2022

Overview

The Python Package for DataHerb

A DataHerb Core Service to Create and Load Datasets.

Install

pip install dataherb

Documentation: dataherb.github.io/dataherb-python

The DataHerb Command-Line Tool

Requires Python 3

The DataHerb cli provides tools to create dataset metadata, validate metadata, search dataset in flora, and download dataset.

Search and Download

Search by keyword

dataherb search covid19
# Shows the minimal metadata

Search by dataherb id

dataherb search -i covid19_eu_data
# Shows the full metadata

Download dataset by dataherb id

dataherb download covid19_eu_data
# Downloads this dataset: http://dataherb.io/flora/covid19_eu_data

Create Dataset Using Command Line Tool

We provide a template for dataset creation.

Within a dataset folder where the data files are located, use the following command line tool to create the metadata template.

dataherb create

Upload dataset to remote

Within the dataset folder, run

dataherb upload

UI for all the datasets in a flora

dataherb serve

Use DataHerb in Your Code

Load Data into DataFrame

# Load the package
from dataherb.flora import Flora

# Initialize Flora service
# The Flora service holds all the dataset metadata
use_flora = "path/to/my/flora.json"
dataherb = Flora(flora=use_flora)

# Search datasets with keyword(s)
geo_datasets = dataherb.search("geo")
print(geo_datasets)

# Get a specific file from a dataset and load as DataFrame
tz_df = pd.read_csv(
  dataherb.herb(
      "geonames_timezone"
  ).get_resource(
      "dataset/geonames_timezone.csv"
  )
)
print(tz_df)

The DataHerb Project

What is DataHerb

DataHerb is an open-source data discovery and management tool.

A DataHerb or Herb is a dataset. A dataset comes with the data files, and the metadata of the data files.
A Herb Resource or Resource is a data file in the DataHerb.
A Flora is the combination of all the DataHerbs.

In many data projects, finding the right datasets to enhance your data is one of the most time consuming part. DataHerb adds flavor to your data project. By creating metadata and manage the datasets systematically, locating an dataset is much easier.

Currently, dataherb supports sync dataset between local and S3/git. Each dataset can have its own remote location.

What is DataHerb Flora

We desigined the following workflow to share and index open datasets.

The repo dataherb-flora is a demo flora that lists some datasets and demonstrated on the website https://dataherb.github.io. At this moment, the whole system is being renovated.

Development

Create a conda environment.
Install requirements: pip install -r requirements.txt

Documentation

The source of the documentation for this package is located at docs.

References and Acknolwedgement

dataherb uses datapackage in the core. datapackage is a python library for the data-package standard. The core schema of the dataset is essentially the data-package standard.

Comments

would you like to take a look at our api?

I come across this repo and found it very similar to our API, though much more mature. https://github.com/Glacier-Ice/data-sci-api

we have problems in creating a standard of dataset collection and API documentation for end-users

is there a way we can collaborate?

opened by Stockard 4
Format search results for better ux

The current search result shows too much information. It would be good to format the result into a way that is easier to read and get the id if needed.
enhancement

opened by emptymalei 1
use rapidfuzz instead of fuzzywuzzy

FuzzyWuzzy is GPLv2 licensed which would force you to licence the whole project under GPLv2. I had the same problem on one of my projects and so I wrote rapidfuzz which is implementing the same algorithm but is based on a version of fuzzywuzzy that was MIT Licensed and is therefor MIT Licensed aswell, so it can be used in here without forcing a License change. As a nice bonus it is fully implemented in C++ and comes with a few Algorithmic improvements making it faster than FuzzyWuzzy.

opened by maxbachmann 1
Use One File for Each Herb in Flora
Is it better to have one file for each herb in flora?

Situition

Currently, the flora is defined in a single json file.

It becomes hard to read. This is not fitting into the human-readable principle.

It becomes hard to manage. We are currently sorting everything in the big file. When we have a problem, the whole flora will be unusable.

Solution

Use separate files for herbs.

Simply Copy dataherb.json

Copy dataherb.json to workdir/{id}/dataherb.json or {id}.json will work.

Using folders allows us to put in more files. For example, we can take datapackage content out to make it more managable.

Build the flora from all these files.

[x] Implement this new structure.

Ready for a Demo repo of flora

In this way, we can put up a repo for open datasets easily and allow users to add more easily.

Possible creating process

Create package directly on GitHub by uploading the dataherb.json file.

But there should be a validation process to avoid duplicate id.

[ ] Setup a demo repo as demo flora.

enhancement
opened by emptymalei 0
Overhaul: New Core Management, Local Indexing Webpage, Flexible Flora Database
This is a completely new era of Dataherb.

New Stuff

Supporting S3 as source

Serve whole flora as webpages with search

User config for flora

Multiple flora on one machine

We also redesigned the core.
opened by emptymalei 0
Add dataset using the URL of a remote repo
We don't only upload datasets, we might also want to load datasets from remote.

Here we propose to add the option to add datasets using the URL.

Build a Herb from remote data

Option to add metadata only or download everything.

Adding metadata only will only add data to the flora

Thus we can not find the dataset folder with the corresponding id.

This can be used to decide if a dataset is metadata only or fully downloaded.
opened by emptymalei 0
Sync Flora Metafolder
Managing flora using command line

Version control of the flora is not really hard. We just get into the folder and use git.

But it would be much easier if we can simply run dataherb sync flora

Approaches:

Invoking command line: ref cookiecutter

enhancement
opened by emptymalei 0

Releases(0.1.6)

0.1.6(Feb 10, 2022)
Fixed

Command line tool dataherb configure -l now only opens the folder.

Command line too dataherb download will also display where the dataset is downloaded to. This makes it easier for the user to find the downloaded dataset.

Source code(tar.gz)
Source code(zip)
0.1.5(Aug 12, 2021)

Using Dedicated Folders for Herbs

In the previous versions, we can only use a single file to host all the flora metadata. It will become unmanageable and hard to read as the number of herbs grows. (#14)

In this version, we introduce a new structure for the flora metadata. Each herb is getting its own folder! This structure makes it easier for us to read and manage by hand. It is also better for version-controling your flora.

(🌱 Best wishes to your herbs in their own pots. )
Source code(tar.gz)
Source code(zip)
0.1.4(Aug 7, 2021)
Added

🎉 Better search result formatting in terminal (See docs for a screenshot.)

📺 Show config using dataherb configure --show

Changed

Better config management. Config has been promoted to a class.

Source code(tar.gz)
Source code(zip)
0.1.3(Aug 7, 2021)
Added

Server to serve flora as a website

Configuration system

Remove herb from flora

Add herb to flora

and more

Source code(tar.gz)
Source code(zip)
0.0.5(Mar 14, 2020)
Now we can use

dataherb validate

to validate the metadata file.
Source code(tar.gz)
Source code(zip)
0.0.3(Feb 23, 2020)

dataherb command line tool now automatically finds the data files and generate part of the metadata based on the files. CSV files are automatically parsed.
Source code(tar.gz)
Source code(zip)
0.0.2(Feb 16, 2020)

Source code(tar.gz)
Source code(zip)

Owner

DataHerb

Get datasets in a blink of an eye | Experimenting with simple modular small dataset discovery

GitHub Repository https://dataherb.github.io/dataherb-python

Very basic but functional Kakuro solver written in Python.

kakuro.py Very basic but functional Kakuro solver written in Python. It uses a reduction to exact set cover and Ali Assaf's elegant implementation of

4 Jan 15, 2022

Scraping and analysis of leetcode-compensations page.

Leetcode compensations report Scraping and analysis of leetcode-compensations page.

96 Jan 01, 2023

Hydrogen (or other pure gas phase species) depressurization calculations

HydDown Hydrogen (or other pure gas phase species) depressurization calculations This code is published under an MIT license. Install as simple as: pi

13 Nov 26, 2022

A multi-platform GUI for bit-based analysis, processing, and visualization

529 Dec 19, 2022

2019 Data Science Bowl

Kaggle-2019-Data-Science-Bowl-Solution - Here i present my solution to kaggle 2019 data science bowl and how i improved it to win a silver medal in that competition.

1 Jan 01, 2022

A Python module for clustering creators of social media content into networks

sm_content_clustering A Python module for clustering creators of social media content into networks. Currently supports identifying potential networks

72 Dec 30, 2022

A library to create multi-page Streamlit applications with ease.

107 Jan 04, 2023

Option Pricing Calculator using the Binomial Pricing Method (No Libraries Required)

Binomial Option Pricing Calculator Option Pricing Calculator using the Binomial Pricing Method (No Libraries Required) Background A derivative is a fi

1 Nov 29, 2021

Import, connect and transform data into Excel

xlwings_query Import, connect and transform data into Excel. Description The concept is to apply data transformations to a main query object. When the

1 Jan 19, 2022

Titanic data analysis for python

Titanic-data-analysis This Repo is an analysis on Titanic_mod.csv This csv file contains some assumed data of the Titanic ship after sinking This full

1 Dec 26, 2021

Pipeline to convert a haploid assembly into diploid

HapDup (haplotype duplicator) is a pipeline to convert a haploid long read assembly into a dual diploid assembly. The reconstructed haplotypes

50 Jan 05, 2023

BinTuner is a cost-efficient auto-tuning framework, which can deliver a near-optimal binary code that reveals much more differences than -Ox settings.

BinTuner is a cost-efficient auto-tuning framework, which can deliver a near-optimal binary code that reveals much more differences than -Ox settings. it also can assist the binary code analysis rese

42 Dec 16, 2022

Create HTML profiling reports from pandas DataFrame objects

Pandas Profiling Documentation | Slack | Stack Overflow Generates profile reports from a pandas DataFrame. The pandas df.describe() function is great

10k Jan 01, 2023

SNV calling pipeline developed explicitly to process individual or trio vcf files obtained from Illumina based pipeline (grch37/grch38).

SNV Pipeline SNV calling pipeline developed explicitly to process individual or trio vcf files obtained from Illumina based pipeline (grch37/grch38).

1 Nov 02, 2021

Python Package for DataHerb: create, search, and load datasets.

Related tags

Overview

The Python Package for DataHerb

A DataHerb Core Service to Create and Load Datasets.

Install

The DataHerb Command-Line Tool

Search and Download

Create Dataset Using Command Line Tool

Upload dataset to remote

UI for all the datasets in a flora

Use DataHerb in Your Code

Load Data into DataFrame

The DataHerb Project

What is DataHerb

What is DataHerb Flora

Development

Documentation

References and Acknolwedgement

Comments

would you like to take a look at our api?

Format search results for better ux

use rapidfuzz instead of fuzzywuzzy

Use One File for Each Herb in Flora

Situition

Solution

Simply Copy dataherb.json

Ready for a Demo repo of flora

Overhaul: New Core Management, Local Indexing Webpage, Flexible Flora Database

New Stuff

Add dataset using the URL of a remote repo

Sync Flora Metafolder

Managing flora using command line

Releases(0.1.6)

0.1.6(Feb 10, 2022)

Fixed

0.1.5(Aug 12, 2021)

Using Dedicated Folders for Herbs

0.1.4(Aug 7, 2021)

Added

Changed

0.1.3(Aug 7, 2021)

Added

0.0.5(Mar 14, 2020)

0.0.3(Feb 23, 2020)

0.0.2(Feb 16, 2020)