Cached file system for online resources in Python

Overview

Minato

Actions Status Python version License pypi version

Cache & file system for online resources in Python

Features

Minato enables you to:

  • Download & cache online recsources
    • minato supports the following protocols: HTTP(S) / AWS S3 / Google Cloud Storage
    • You can manage cached files via command line interface
  • Automatically update cached files based on ETag
    • minato downloads new versions if available when you access cached files
  • Open online files super easily
    • By using minato.open, you can read/write online resources like the built-in open method

Installation

pip install minato[all]

Usage

Python

import minato

# Read / write files on online storage
with minato.open("s3://your_bucket/path/to/file", "w") as f:
    f.write("Create a new file on AWS S3!")

# Cache & manage online resources in local storage
local_filename = minato.cached_path("http://example.com/path/to/archive.zip!inner/path/to/file")

CLI

❯ poetry run minato --help
usage: minato

positional arguments:
  {cache,list,remove,update}
    cache               cache remote file and return cached local file path
    list                show list of cached files
    remove              remove cached files
    update              update cached files

optional arguments:
  -h, --help            show this help message and exit
  --version             show program's version number and exit
You might also like...
Exploit Discord's cache system to remote upload payloads on Discord users machines
Exploit Discord's cache system to remote upload payloads on Discord users machines

Exploit Discord's cache system to hide payloads PoC Remote upload embedded payload from image using EOF to Discord users machines through cache. Depen

A generic system for filtering Django QuerySets based on user selections

Django Filter Django-filter is a reusable Django application allowing users to declaratively add dynamic QuerySet filtering from URL parameters. Full

A Django based shop system

django-SHOP Django-SHOP aims to be a the easy, fun and fast e-commerce counterpart to django-CMS. Here you can find the full documentation for django-

Login System Django

Login-System-Django Login System Using Django Tech Used Django Python Html Run Locally Clone project git clone https://link-to-project Get project for

A Blog Management System Built with django
A Blog Management System Built with django

Blog Management System Backend use: Django Features Enhanced Ui

This repository contains django library management system project.

Library Management System Django ** INSTALLATION** First of all install python on your system. Then run pip install -r requirements.txt to required se

Djang Referral System
Djang Referral System

Djang Referral System About | Features | Technologies | Requirements | Starting | License | Author 🎯 About I created django referral system and I wan

Advanced school management system written in Django :)
Advanced school management system written in Django :)

Advanced school management system written in Django :) ⚙️ Config the project First you should make venv for this project. So in the main root of proje

A visual indicator of what environment/system you're using in django

A visual indicator of what environment/system you're using in django

Comments
  • Bump protobuf from 4.21.5 to 4.21.6

    Bump protobuf from 4.21.5 to 4.21.6

    Bumps protobuf from 4.21.5 to 4.21.6.

    Release notes

    Sourced from protobuf's releases.

    Protocol Buffers v3.0.0-beta-4

    Version 3.0.0-beta-4

    General

    • Added a deterministic serialization API for C++. The deterministic serialization guarantees that given a binary, equal messages will be serialized to the same bytes. This allows applications like MapReduce to group equal messages based on the serialized bytes. The deterministic serialization is, however, NOT canonical across languages; it is also unstable across different builds with schema changes due to unknown fields. Users who need canonical serialization, e.g. persistent storage in a canonical form, fingerprinting, etc, should define their own canonicalization specification and implement the serializer using reflection APIs rather than relying on this API.

    • Added OneofOptions. You can now define custom options for oneof groups.

      import "google/protobuf/descriptor.proto";
      extend google.protobuf.OneofOptions {
        optional int32 my_oneof_extension = 12345;
      }
      message Foo {
        oneof oneof_group {
          (my_oneof_extension) = 54321;
          ...
        }
      }
      

    C++ (beta)

    • Introduced a deterministic serialization API in CodedOutputStream::SetSerializationDeterministic(bool). See the notes about deterministic serialization in the General section.
    • Added google::protobuf::Map::swap() to swap two map fields.
    • Fixed a memory leak when calling Reflection::ReleaseMessage() on a message allocated on arena.
    • Improved error reporting when parsing text format protos.
    • JSON
      • Added a new parser option to ignore unknown fields when parsing JSON.
      • Added convenient methods for message to/from JSON conversion.
    • Various performance optimizations.

    Java (beta)

    • File option "java_generate_equals_and_hash" is now deprecated. equals() and hashCode() methods are generated by default.
    • Added a new JSON printer option "omittingInsignificantWhitespace" to produce a more compact JSON output. The printer will pretty-print by default.
    • Updated Java runtime to be compatible with 2.5.0/2.6.1 generated protos.

    Python (beta)

    ... (truncated)

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    dependencies 
    opened by dependabot[bot] 0
  • Bump ujson from 5.3.0 to 5.4.0

    Bump ujson from 5.3.0 to 5.4.0

    Bumps ujson from 5.3.0 to 5.4.0.

    Release notes

    Sourced from ujson's releases.

    5.4.0

    Added

    Fixed

    Commits
    • 9c20de0 Merge pull request from GHSA-fm67-cv37-96ff
    • b21da40 Fix double free on string decoding if realloc fails
    • 67ec071 Merge pull request #555 from JustAnotherArchivist/fix-decode-surrogates-2
    • bc7bdff Replace wchar_t string decoding implementation with a uint32_t-based one
    • cc70119 Merge pull request #548 from JustAnotherArchivist/arbitrary-ints
    • 4b5cccc Merge pull request #553 from bwoodsend/pypy-ci
    • abe26fc Merge pull request #551 from bwoodsend/bye-bye-travis
    • 3efb5cc Delete old TravisCI workflow and references.
    • 404de1a xfail test_decode_surrogate_characters() on Windows PyPy.
    • f7e66dc Switch to musl docker base images.
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    dependencies 
    opened by dependabot[bot] 0
  • CVE-2007-4559 Patch

    CVE-2007-4559 Patch

    Patching CVE-2007-4559

    Hi, we are security researchers from the Advanced Research Center at Trellix. We have began a campaign to patch a widespread bug named CVE-2007-4559. CVE-2007-4559 is a 15 year old bug in the Python tarfile package. By using extract() or extractall() on a tarfile object without sanitizing input, a maliciously crafted .tar file could perform a directory path traversal attack. We found at least one unsantized extractall() in your codebase and are providing a patch for you via pull request. The patch essentially checks to see if all tarfile members will be extracted safely and throws an exception otherwise. We encourage you to use this patch or your own solution to secure against CVE-2007-4559. Further technical information about the vulnerability can be found in this blog.

    If you have further questions you may contact us through this projects lead researcher Kasimir Schulz.

    opened by TrellixVulnTeam 0
Releases(v0.10.0)
  • v0.10.0(Aug 11, 2022)

    What's Changed

    • Bump ujson from 5.3.0 to 5.4.0 by @dependabot in https://github.com/altescy/minato/pull/36
    • Table supports full-width characters by @altescy in https://github.com/altescy/minato/pull/37
    • Modify auto-format settings by @altescy in https://github.com/altescy/minato/pull/38
    • Add progress & remove tqdm by @altescy in https://github.com/altescy/minato/pull/39
    • Add disable-progressbar option by @altescy in https://github.com/altescy/minato/pull/40
    • Add interactive selector by @altescy in https://github.com/altescy/minato/pull/41
    • Bump version number to v0.10.0 by @altescy in https://github.com/altescy/minato/pull/42

    New Contributors

    • @dependabot made their first contribution in https://github.com/altescy/minato/pull/36

    Full Changelog: https://github.com/altescy/minato/compare/v0.9.0...v0.10.0

    Source code(tar.gz)
    Source code(zip)
  • v0.9.0(Jul 2, 2022)

    What's Changed

    • Use options for open_file function by @altescy in https://github.com/altescy/minato/pull/33
    • Support Python 3.9 and 3.10 by @altescy in https://github.com/altescy/minato/pull/34
    • Bump version number to v0.9.0 by @altescy in https://github.com/altescy/minato/pull/35

    Full Changelog: https://github.com/altescy/minato/compare/v0.8.2...v0.9.0

    Source code(tar.gz)
    Source code(zip)
  • v0.8.2(Feb 24, 2022)

    What's Changed

    • Modify log level by @altescy in https://github.com/altescy/minato/pull/30
    • Upgrade dependent packages by @altescy in https://github.com/altescy/minato/pull/31
    • Bump version number to v0.8.2 by @altescy in https://github.com/altescy/minato/pull/32

    Full Changelog: https://github.com/altescy/minato/compare/v0.8.1...v0.8.2

    Source code(tar.gz)
    Source code(zip)
  • v0.8.1(Feb 11, 2022)

    What's Changed

    • Fix s3 progress by @altescy in https://github.com/altescy/minato/pull/27
    • Add all command to Makefile by @altescy in https://github.com/altescy/minato/pull/28
    • Bump version number to v0.8.1 by @altescy in https://github.com/altescy/minato/pull/29

    Full Changelog: https://github.com/altescy/minato/compare/v0.8.0...v0.8.1

    Source code(tar.gz)
    Source code(zip)
  • v0.8.0(Feb 8, 2022)

    What's Changed

    • Modify tqdm progress by @altescy in https://github.com/altescy/minato/pull/22
    • Upgrade dependent packages by @altescy in https://github.com/altescy/minato/pull/23
    • Split github workflows into ci and pypi-publish by @altescy in https://github.com/altescy/minato/pull/24
    • Modify type annotations by @altescy in https://github.com/altescy/minato/pull/25
    • Bump version number to v0.8.0 by @altescy in https://github.com/altescy/minato/pull/26

    Full Changelog: https://github.com/altescy/minato/compare/v0.7.0...v0.8.0

    Source code(tar.gz)
    Source code(zip)
  • v0.7.0(Nov 19, 2021)

    • Shrink table column width automatically (#16)
    • Modify S3 download (#17)
    • Move boto3-stubs to dev-dependencies (#18)
    • Add upload method to S3FileSystem (#19)
    • Add upload method to GCSFileSystem (#20)
    • Bump version number to v0.7.0 (#21)
    Source code(tar.gz)
    Source code(zip)
  • v0.6.2(Oct 23, 2021)

  • v0.6.1(Oct 1, 2021)

  • v0.6.0(Sep 30, 2021)

    • Remove quotations of S3 Etags (#2)
    • Modify CI settings (#3)
    • Update dependent packages (#4)
    • Make filesystems optional (#5)
    • Add opening options (#6)
    • Add helps to each command line options (#7)
    • Add feature description to readme (#8)
    • Bump version number to v0.6.0 (#9)
    Source code(tar.gz)
    Source code(zip)
  • v0.5.0(Jul 31, 2021)

  • v0.4.0(Jun 25, 2021)

    • Support file-level expiration
    • Enable automatic cache updating based on HTTP ETag
    • Show download progress of S3 and GCS
    • Deprecate upload/download commands
    Source code(tar.gz)
    Source code(zip)
  • v0.3.0(Jun 23, 2021)

Owner
Yasuhiro Yamaguchi
Research Engineer / NLP / ML
Yasuhiro Yamaguchi
An extremely fast JavaScript and CSS bundler and minifier

Website | Getting started | Documentation | Plugins | FAQ Why? Our current build tools for the web are 10-100x slower than they could be: The main goa

Evan Wallace 34.2k Jan 04, 2023
A package to handle images in django

Django Image Tools Django Image Tools is a small app that will allow you to manage your project's images without worrying much about image sizes, how

The Bonsai Studio 42 Jun 02, 2022
Djang Referral System

Djang Referral System About | Features | Technologies | Requirements | Starting | License | Author 🎯 About I created django referral system and I wan

Alex Kotov 5 Oct 25, 2022
Running in outer Django project folder (cd django_project)

Django Running in outer Django project folder (cd django_project) Make Migrations python manage.py makemigrations Migrate to Database python manage.py

1 Feb 07, 2022
Sistema de tratamento e análise de grandes volumes de dados através de técnicas de Data Science

Sistema de tratamento e análise de grandes volumes de dados através de técnicas de data science Todos os scripts, gráficos e relatórios de todas as at

Arthur Quintanilha Neto 1 Sep 05, 2022
Improved Django model inheritance with automatic downcasting

Polymorphic Models for Django Django-polymorphic simplifies using inherited models in Django projects. When a query is made at the base model, the inh

1.4k Jan 03, 2023
Redia Cache implementation in django.

django-redis Recipe APP Simple Recipe app which shows different kinds off recipe to the user. Why Cache ? Accessing data from cache is much faster tha

Avinash Alanjkar 1 Sep 21, 2022
Packs a bunch of smaller CSS files together from 1 folder.

Packs a bunch of smaller CSS files together from 1 folder.

1 Dec 09, 2021
Django API creation with signed requests utilizing forms for validation.

django-formapi Create JSON API:s with HMAC authentication and Django form-validation. Version compatibility See Travis-CI page for actual test results

5 Monkeys 34 Apr 04, 2022
a little task queue for python

a lightweight alternative. huey is: a task queue (2019-04-01: version 2.0 released) written in python (2.7+, 3.4+) clean and simple API redis, sqlite,

Charles Leifer 4.3k Dec 29, 2022
Bleach is an allowed-list-based HTML sanitizing library that escapes or strips markup and attributes

Bleach Bleach is an allowed-list-based HTML sanitizing library that escapes or strips markup and attributes. Bleach can also linkify text safely, appl

Mozilla 2.5k Dec 29, 2022
A CBV to handle multiple forms in one view

django-shapeshifter A common problem in Django is how to have a view, especially a class-based view that can display and process multiple forms at onc

Kenneth Love 167 Nov 26, 2022
Basic implementation of Razorpay payment gateway 💳 with Django

Razorpay Payment Integration in Django 💥 In this project Razorpay payment gateway 💳 is integrated with Django by breaking down the whole process int

ScaleReal 12 Dec 12, 2022
A collection of models, views, middlewares, and forms to help secure a Django project.

Django-Security This package offers a number of models, views, middlewares and forms to facilitate security hardening of Django applications. Full doc

SD Elements 258 Jan 03, 2023
A app for managing lessons with Django

Course Notes A app for managing lessons with Django Some Ideas

Motahhar.Mokfi 6 Jan 28, 2022
Streamlining Django forms to provide all the wins of single-page-applications without the pain.

nango Streamlining Django forms to provide all the wins of single-page-applications without the pain. Key features Available to all Django deployments

Nick Farrell 107 Dec 12, 2022
Django Rest Framework + React application.

Django Rest Framework + React application.

2 Dec 19, 2022
Customize the behavior of django.contrib.auth permissions.

Customizando o comportamento do django.contrib.auth. O que queremos? Não criar as permissões padrões automaticamente (add, delete, view, read). Criar

Henrique Bastos 7 Nov 26, 2022
Imparare Django ricreando un sito facsimile a quello Flask

SitoPBG-Django Imparare Django ricreando un sito facsimile a quello Flask Note di utilizzo Necessita la valorizzazione delle seguenti variabili di amb

Mario Nardi 1 Dec 08, 2021
Thumbnails for Django

Thumbnails for Django. Features at a glance Support for Django 2.2, 3.0 and 3.1 following the Django supported versions policy Python 3 support Storag

Jazzband 1.6k Jan 03, 2023