Python's Filesystem abstraction layer

Last update: Jan 02, 2023

Overview

PyFilesystem2

Python's Filesystem abstraction layer.

Documentation

Introduction

Think of PyFilesystem's FS objects as the next logical step to Python's file objects. In the same way that file objects abstract a single file, FS objects abstract an entire filesystem.

Let's look at a simple piece of code as an example. The following function uses the PyFilesystem API to count the number of non-blank lines of Python code in a directory. It works recursively, so it will find .py files in all sub-directories.

def count_python_loc(fs):
    """Count non-blank lines of Python code."""
    count = 0
    for path in fs.walk.files(filter=['*.py']):
        with fs.open(path) as python_file:
            count += sum(1 for line in python_file if line.strip())
    return count

We can call count_python_loc as follows:

from fs import open_fs
projects_fs = open_fs('~/projects')
print(count_python_loc(projects_fs))

The line project_fs = open_fs('~/projects') opens an FS object that maps to the projects directory in your home folder. That object is used by count_python_loc when counting lines of code.

To count the lines of Python code in a zip file, we can make the following change:

projects_fs = open_fs('zip://projects.zip')

Or to count the Python lines on an FTP server:

projects_fs = open_fs('ftp://ftp.example.org/projects')

No changes to count_python_loc are necessary, because PyFileystem provides a simple consistent interface to anything that resembles a collection of files and directories. Essentially, it allows you to write code that is independent of where and how the files are physically stored.

Contrast that with a version that purely uses the standard library:

def count_py_loc(path):
    count = 0
    for root, dirs, files in os.walk(path):
        for name in files:
            if name.endswith('.py'):
                with open(os.path.join(root, name), 'rt') as python_file:
                    count += sum(1 for line in python_file if line.strip())
    return count

This version is similar to the PyFilesystem code above, but would only work with the OS filesystem. Any other filesystem would require an entirely different API, and you would likely have to re-implement the directory walking functionality of os.walk.

Credits

The following developers have contributed code and their time to this projects:

See CONTRIBUTORS.md for a full list of contributors.

PyFilesystem2 owes a massive debt of gratitude to the following developers who contributed code and ideas to the original version.

Ryan Kelly
Andrew Scheller
Ben Timby

Apologies if I missed anyone, feel free to prompt me if your name is missing here.

Support

If commercial support is required, please contact Will McGugan.

Comments

How to get a path pointing to io.BytesIO?

I am finding lot of librarys that use the filename for opening the file. However, in my Flask app I would like to get a filename that points to IO.BytesIO without the need o using temporal files

opened by WaterKnight1998 32
fs.listdir and UnicodeError

On my system for whatever reason I have a file whith wrong encoding in the / dir. Always if I want
for item in sorted(self.fs.listdir(_sel_dir)):

I have to encapsulate this by an exception for UnicodeDecodeError. I would prefer to not crash but just ignore this file.

(I am still looking on why that file anyway is there)
bug accepted

opened by ReimarBauer 29
Add copy_if_newer parameter to functions in copy.py

Added copy_if_newer parameter to all functions in copy.py. Files are copied only if destination files don't exist or sources are newer then destination files. Directories are always copied. In case time of modification for files cannot be determined files are safely copied.

Moreover I added geturl method to osfs returning "file://" + self.getsyspath(path).

opened by gpcimino 28

Bad namespacing, imports breaks setuptools

fs.dropboxfs is implemented using fs as a namespace package. This causes it to misbehave when importing from fs for several reasons:

fs declares itself as a namespace package. According to the setuptools documentation here,

A) only namespace packages are supposed to contain the __import__('pkg_resources')... B) they must not contain executable code aside from that import.

We violate both of those rules, thus resulting in weird exceptions like this:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/dargueta/dev/goodrx-data-scripts/airflow_dags/grx/third_party/dropbox.py", line 6, in <module>
    from fs.dropboxfs import DropboxFS
  File "/Users/dargueta/.pyenv/versions/2.7.15/envs/gds/lib/python2.7/site-packages/fs/dropboxfs/__init__.py", line 1, in <module>
    from .dropboxfs import DropboxFile
  File "/Users/dargueta/.pyenv/versions/2.7.15/envs/gds/lib/python2.7/site-packages/fs/dropboxfs/dropboxfs.py", line 12, in <module>
    from fs.base import FS
  File "/Users/dargueta/.pyenv/versions/2.7.15/envs/gds/lib/python2.7/site-packages/fs/base.py", line 24, in <module>
    from . import copy, errors, fsencode, iotools, move, tools, walk, wildcard
ImportError: cannot import name fsencode

cannot import name fsencode

Changing that line to these two fixes the problem:

from . import copy, errors, iotools, move, tools, walk, wildcard
from ._fscompat import fsencode

There's a similar issue in fs.test:

# Broken
from fs import ResourceType, Seek

# Works
from fs.enums import ResourceType, Seek

It appears these are the only two places in the repo we do this. Fixing these imports gets us a separate weird error in code using fs.dropboxfs but that may be unrelated.

opened by dargueta 23

Update tests, and fix discovered bugs
Hi Will, I updated the tests in fs.test with some missing checks, and discovered some new bugs I will fix gradually in this PR.

New tests

FS.copy, FS.move, FS.copydir and FS.movedir should raise ResourceNotFound when the source is nowhere to be found

FS.copy and FS.move should probably raise FileExpected when the source is a directory (not listed in the documentation, but makes sense I guess ?...)

FS.movedir and FS.copydir should raise DirectoryExpected when the source is a file

If a filesystem is advertising itself as case insensitive, check that's true

If a filesystem is advertising itself as supporting unicode paths, check that's true

Check that file-like objects returned by FS.openbin:

have a seek method that returns the new absolute position when called

have a write method that returns the number of bytes (in binary mode) / characters (in text mode) written

have a truncate method that returns the new length of the file

Discovered bugs

~FTPFile.seek, FTPFile.truncate and FTPFile.write do not return anything~ fixed

~FTPFS crashes on unicode paths (/földér)~ fixed, see below

~MemoryFile.write and MemoryFile.truncate do not return anything~ fixed

~many filesystem's move and copy do not raise FileExpected when given a directory~ fixed by fixing FS.copy and FS.move

About FTPFS

FTP servers supporting UTF-8 encoded paths display the feature UTF8. But the ftplib.FTP objects require to set the encoding before connecting to the server, or else the server and the client encoding will not be synchronised (client will send utf-8 paths but server will treat them as latin-1).

To fix this, the encoding is checked when accessing the FTPFS.ftp property for the first time. A mock connection is established to check for the UTF8 feature ; then, a proper connection is established, with utf-8 as the encoding if available, and latin-1 otherwise. Then, any new ftp connection will use the previous encoding, so the overhead is limited.

About test_ftpfs

I rewrote the class so that a new server is created once by class instead of before each test, which was a lot longer for nothing. I also made use of pyftpdlib.test.FTPd instead of externally calling the file and then task-killing the thread, which was a lot hackier.

I still think there is an issue with the timeout of FTPFS objects, since the test_connection_error tests take a really long time, longer than what they should take given the timeout.
opened by althonos 22
backslash in windows path definition - need a forward slash
Hi I ran in a problem of \ used in pathes. This was in the past different. Also the documentations shows:

This is broadly similar to the standard os.path module but works with paths in the canonical format expected by all FS objects (that is, separated by forward slashes and with an optional leading slash).

from fs.tempfs import TempFS ROOT_FS = TempFS() ROOT_DIR = ROOT_FS.root_path print(ROOT_DIR) 'C:\\Users\\Local...'
question
opened by ReimarBauer 21
Fix units in `fs.filesize`

According to Wikipedia, the binary filesize units should be KiB, MiB, ..., and the decimal units should be kB, MB, ...

Previously, traditional would use the SI prefixes, and decimal would display bits instead of bytes.

Also, I'm not sure about this, but once again functions should probably be raising TypeError when given an argument that cannot be coerced to a numerical value.

opened by althonos 19
Migrate tests to Pytest
Closes #327

Type of changes

[x] Bug fix

[x] New feature

[ ] Documentation / docstrings

[x] Tests

[ ] Other

Checklist

[x] I've run the latest black with default args on new code.

[x] I've updated CHANGELOG.md and CONTRIBUTORS.md where appropriate.

[ ] I've added tests for new code.

[x] I accept that @willmcgugan may be pedantic in the code review.

Description

Migrate test suite from nose to pytest

Randomized execution order of tests to expose interdependencies

Overhauled Travis config to use tox environments instead of passing them in multiple places

Bugs / Cleanup

Fix a few import mock bugs that imported mock on Python 3 instead of the native module

Remove unused imports in a few test files

Remove requirements.txt, as it's out of date and no longer needed.

Fixed abstract class imports from collections which will break on Python 3.8
opened by dargueta 18

Is there a way to check whether two paths (or filesystems) are referencing the same file?

I'd like to know if there is a way to check whether two pathes reference the same file. I found that not even the filesystems can be compared:

>>> import fs
>>> fs1 = fs.open_fs("~/Desktop")
>>> fs2 = fs.open_fs("~/Desktop")
>>> fs1 == fs2
False

At the moment I'm resorting to this, which is not nice and complete at all.

from typing import NamedTuple

class Resource(NamedTuple):
    fs: FS
    path: str

def is_same(a: Resource, b: Resource):
    from fs.osfs import OSFS
    from fs.zipfs import ZipFS
    from fs.tarfs import TarFS

    if isinstance(a.fs, OSFS) and isinstance(b.fs, OSFS):
        return a.fs.getospath(a.path) == b.fs.getospath(b.path)
    elif isinstance(a.fs, ZipFS) and isinstance(b.fs, ZipFS):
        return a.fs._file == b.fs._file and a.path == b.path
    elif isinstance(a.fs, TarFS) and isinstance(b.fs, TarFS):
        return a.fs._file == b.fs._file and a.path == b.path
    return False

Is there a nicer way to check this? Would you be interested in a PR that fixes the __eq__ method for some filesystems?

question

opened by tfeldmann 17

splitext removes a forward slash from url
Why does fs.path.splitext(url) remove one of the forward slashes?

url = 's3://simonm3/_testdata/file1.pkl' fs.path.splitext(url), os.path.splitext(url)

(('s3:/simonm3/_testdata/file1', '.pkl'), ('s3://simonm3/_testdata/file1', '.pkl'))

Also what is the purpose of the second forward slash? The standard format of a url is that // denotes the start what urllib.urlparse calls "netloc" and includes username etc.. The pyfilesystem url instead uses // as part of the filesystem. Why not have "s3:/somefolder/somefile" rather than "s3://somefolder/somefile" ?

https://en.wikipedia.org/wiki/Uniform_Resource_Identifier#Generic_syntax
opened by simonm3 17
Implement TarFS.geturl and ZipFS.geturl and Fix #329, #333, #340
Type of changes

[x] Bug fix

[x] New feature

[ ] Documentation / docstrings

[x] Tests

[ ] Other

Checklist

[x] I've run the latest black with default args on new code.

[x] I've updated CHANGELOG.md and CONTRIBUTORS.md where appropriate.

[x] I've added tests for new code.

[x] I accept that @willmcgugan may be pedantic in the code review.

Description

Fix #329, #333, #340

And deliver geturl for ZipFS and TarFS, and my use case is to get url for a file inside a zip, deposit the url, and read it later.

with file_system.open_fs('zip://test.zip') as fs_handle: template_file_exists = fs_handle.exists( template ) and fs_handle.isfile(template) if template_file_exists: return fs_handle.geturl(template)

The existing code in ZipFS and TarFS did not imagine an download url is really needed for a file inside a zip or tar. Hence, I would like to present my use case and get it supported.
opened by chfw 17
Fix file move time preservation (#558)
Type of changes

Bug fix

Checklist

[x] I've run the latest black with default args on new code.

[x] I've updated CHANGELOG.md and CONTRIBUTORS.md where appropriate.

[x] I've added tests for new code.

[ ] I've updated the Documentation.

[x] I accept that @PyFilesystem/maintainers may be pedantic in the code review.

Description

This PR fixes a bug in the methods FS.move and MemoryFS.move, where preserve_time=True resulted in an ResourceNotFound error (see #558).

The new functions to separate copy.copy_modified_time are currently residing in copy.py. One could argue that reading and updating a files meta info are more suitable for info.py, but this would introduce further changes. Additionally, we could modify the original one to utilize both new functions. This would avoid code duplication but introduce further function calls. Opinions on this matter and reviews are welcomed!

Lastly, I would update the documentation for the new functions accordingly.
opened by mj0nez 1

ResourceNotFound - moving files on the same filesystem with preserve_time=True

Hi, while experimenting with the FS lib, I’ve encountered an unexpected ResourceNotFound error. The error is raised if the param preserve_time=True and we try to move a file on the same OS or in-memory filesystem.

This block represents a reproduceable example for a MemoryFS:

    mem_fs = memoryfs.MemoryFS()

    mem_fs.makedir("foo")
    mem_fs.writetext("foo/README.md", "Tetris clone")

    move_file(
        src_fs=mem_fs,
        src_path="foo/README.md",
        dst_fs=mem_fs,
        dst_path="foo/README_2.md",
        preserve_time=True,
    )

The raised error message:

./tests/test_resource_error.py::test_move_between_same_fs_mem Failed: [undefined]fs.errors.ResourceNotFound: resource 'foo/README.md' not found
def test_move_between_same_fs_mem():
        mem_fs = memoryfs.MemoryFS()
    
        mem_fs.makedir("foo")
        mem_fs.writetext("foo/README.md", "Tetris clone")
    
>       move_file(
            src_fs=mem_fs,
            src_path="foo/README.md",
            dst_fs=mem_fs,
            dst_path="foo/README_2.md",
            preserve_time=True,
        )

tests\test_resource_error.py:31: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
fs\move.py:68: in move_file
    _src_fs.move(
fs\memoryfs.py:478: in move
    copy_modified_time(self, src_path, self, dst_path)
fs\copy.py:532: in copy_modified_time
    src_meta = _src_fs.getinfo(src_path, namespaces)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = MemoryFS(), path = 'foo/README.md', namespaces = ('details',)

    def getinfo(self, path, namespaces=None):
        # type: (Text, Optional[Collection[Text]]) -> Info
        _path = self.validatepath(path)
        dir_entry = self._get_dir_entry(_path)
        if dir_entry is None:
>           raise errors.ResourceNotFound(path)
E           fs.errors.ResourceNotFound: resource 'foo/README.md' not found

fs\memoryfs.py:394: ResourceNotFound

I have modified the test case test_move_file_same_fs and found similar errors for OS- and Temp-filesystems. For my understanding, the current implementation wants to copy the file’s metadata, after moving it or renaming its path. Therefore, the copy_modified_time function encounters a missing resource under these conditions:

moving a file on the same filesystem
preserve_time=True

AND either

copy_modified_time is called after moving the file, see memoryfs.py
or FS.move of base.py is used and the filesystem supports renaming, see base.py

Is this a limitation of my system (Win11 x64)? I could not find similar issues or information in the docs.

If this is a not yet supported edge case, I would suggest that we read the file’s meta data first, then move it and update the meta info afterwards.

Any review and comments are welcome! Regards

opened by mj0nez 2

#556 Fallback to alternate mtime correctly
Type of changes

Bug fix

Checklist

[X] I've run the latest black with default args on new code.

[x] I've updated CHANGELOG.md and CONTRIBUTORS.md where appropriate.

[x] I've added tests for new code.

[X] I accept that @PyFilesystem/maintainers may be pedantic in the code review.

Description

This addresses an issues where write_zip() assumed that the details namespace was always available. The stat namespace will be checked first, followed by details, and finally the current time will be used if st_mtime is still None.

I'm willing to author a test against this, but I'm only aware of this being an issue with the s3 filesystem (and only when it encounters a directory) and I'm not certain how best to test that here.

https://github.com/PyFilesystem/pyfilesystem2/issues/556
bug
opened by james-emerton 2

Unable to use compress.write_zip() with s3fs source

I believe this ticket should be for this package, but the issue is caused by implementation details of the fs-s3fs package. Attempting to create an archive of files from s3 using write_zip() fails with the following error:

Traceback (most recent call last):
  File "/Users/james/dev/spire/mount-s3-archive/test.py", line 7, in <module>
    write_zip(source, sys.argv[2])
  File "/Users/james/dev/spire/mount-s3-archive/.venv/lib/python3.10/site-packages/fs/compress.py", line 78, in write_zip
    mt = info.modified or datetime.utcnow()
  File "/Users/james/dev/spire/mount-s3-archive/.venv/lib/python3.10/site-packages/fs/info.py", line 322, in modified
    self._require_namespace("details")
  File "/Users/james/dev/spire/mount-s3-archive/.venv/lib/python3.10/site-packages/fs/info.py", line 128, in _require_namespace
    raise MissingInfoNamespace(namespace)
fs.errors.MissingInfoNamespace: namespace 'details' is required for this attribute

Code:

import sys

import fs
from fs.compress import write_zip

source = fs.open_fs(sys.argv[1])
write_zip(source, sys.argv[2])

Executing the above code as test.py s3://some-bucket/ test.zip will fail with the above error while a local source directory will compress nicely.

The issue seems to be that S3FS directory entries don't expose a details namespace. It's unclear if that's correct or not, but the code in write_zip() seems to assume that details is always available (but makes allowance for Info.modified to be a falsy value).

opened by james-emerton 0

Invalid characters in the path
I apologize if my question sounds silly but I just started using this library and can't find the answer to my question. I'm trying to copy a file from my disk system to a memory file system:

mem_fs = fs.open_fs('mem://') zip_path = 'C:\\project\\test.zip" fs.copy.copy_file('/', zip_path.format(os.getcwd()), mem_fs, 'test.zip')

I have the following exception: fs.errors.InvalidCharsInPath because the path contains ":" and "\". If I replace all these "invalid" characters with "/" the resource can't be found. The OS: Win7. How can this issue be solved? And what is the cross-platform solution (I will run my app on Linux too)? Thanks!
opened by kwantdev 11
Drop usage of `pkg_resources`
Type of changes

Bug fix

Checklist

[ ] I've run the latest black with default args on new code.

[ ] I've updated CHANGELOG.md and CONTRIBUTORS.md where appropriate.

[x] I've added tests for new code.

[x] I accept that @PyFilesystem/maintainers may be pedantic in the code review.

Description

Drop pkg_resources in favor of the importlib.metadata API.

Closes https://github.com/PyFilesystem/pyfilesystem2/issues/356
opened by edgarrmondragon 1

Releases(2.4.16)

2.4.16(May 2, 2022)
Changed

Make fs.zipfs._ZipExtFile use the seeking mechanism implemented in the Python standard library in Python version 3.7 and later (#527).

Mark fs.zipfs.ReadZipFS as a case-sensitive filesystem (#527).

Optimized moving files between filesystems with syspaths. (#523).

Fixed fs.move.move_file to clean up the copy on the destination in case of errors.

fs.opener.manage_fs with writeable=True will now raise a ResourceReadOnly exception if the managed filesystem is not writeable.

Marked filesystems wrapped with fs.wrap.WrapReadOnly as read-only.

Source code(tar.gz)
Source code(zip)
2.4.15(Feb 7, 2022)
Changed

Support more lenient usernames and group names in FTP servers (#507). Closes #506.

Fixed

Fixed MemoryFS.move and MemoryFS.movedir not updating the name of moved resources, causing MemoryFS.scandir to use the old name. (#510). Closes #509.

Make WrapFS.move and WrapFS.movedir use the delegate FS methods instead of fs.move functions, which was causing optimized implementation of movedir to be always skipped. (#511).

Source code(tar.gz)
Source code(zip)
v2.4.14(Nov 16, 2021)
Added

Added fs.copy.copy_file_if, fs.copy.copy_dir_if, and fs.copy.copy_fs_if. Closes #458.

Added fs.base.FS.getmodified.

Changed

FTP servers that do not support the MLST command now try to use the MDTM command to retrieve the last modification timestamp of a resource. Closes #456.

Fixed

Fixed performance bugs in fs.copy.copy_dir_if_newer. Test cases were adapted to catch those bugs in the future.

Fixed precision bug for timestamps in fs.OSFS.setinfo.

Source code(tar.gz)
Source code(zip)
v2.4.13(Mar 27, 2021)
Added

Added FTP over TLS (FTPS) support to FTPFS. Closes #437, #449.

PathError now supports wrapping an exception using the exc argument. Closes #453.

Better documentation of the writable parameter of fs.open_fs, and hint about using fs.wrap.read_only when a read-only filesystem is required. Closes #441.

Changed

Make FS.upload explicit about the expected error when the parent directory of the destination does not exist. Closes #445.

Migrate continuous integration from Travis-CI to GitHub Actions and introduce several linters again in the build steps (#448). Closes #446.

Stop requiring pytest to run tests, allowing any test runner supporting unittest-style test suites.

FSTestCases now builds the large data required for upload and download tests only once in order to reduce the total testing time.

MemoryFS.move and MemoryFS.movedir will now avoid copying data. Closes #452.

FS.removetree("/") behaviour has been standardized in all filesystems, and is expected to clear the contents of the root folder without deleting it. Closes #471.

FS.getbasic is now deprecated, as it is redundant with FS.getinfo, and FS.getinfo is now explicitly expected to return the basic info namespace unconditionally. Closes #469.

Fixed

Make FTPFile, MemoryFile and RawWrapper accept array.array arguments for the write and writelines methods, as expected by their base class io.RawIOBase.

Various documentation issues, including MemoryFS docstring not rendering properly.

Avoid creating a new connection on every call of FTPFS.upload. Closes #455.

WrapReadOnly.removetree not raising a ResourceReadOnly when called. Closes #468.

WrapCachedDir.isdir and WrapCachedDir.isfile raising a ResourceNotFound error on non-existing path (#470).

FTPFS not listing certain entries with sticky/SUID/SGID permissions set by Linux server (#473). Closes #451.

scandir iterator not being closed explicitly in OSFS.scandir, occasionally causing a ResourceWarning to be thrown. Closes #311.

Incomplete type annotations for the temp_fs parameter of WriteTarFS and WriteZipFS. Closes #410.

Source code(tar.gz)
Source code(zip)
v2.4.12(Jan 14, 2021)
Added

Missing mode attribute to _MemoryFile objects returned by MemoryFS.openbin.

Missing readinto method for MemoryFS and FTPFS file objects. Closes #380.

Added compatibility if a Windows FTP server returns file information to the LIST command with 24-hour times. Closes #438.

Changed

Start testing on PyPy. Due to #342 we have to treat PyPy builds specially and allow them to fail, but at least we'll be able to see if we break something aside from known issues with FTP tests.

Include docs in source distributions as well as the whole tests folder, ensuring conftest.py is present, fixes #364.

Stop patching copy with Python 3.8+ because it already uses sendfile.

Fixed

Fixed crash when CPython's -OO flag is used

Fixed error when parsing timestamps from a FTP directory served from a WindowsNT FTP Server, fixes #395.

Fixed documentation of Mode.to_platform_bin. Closes #382.

Fixed the code example in the "Testing Filesystems" section of the "Implementing Filesystems" guide. Closes #407.

Fixed FTPFS.openbin not implicitly opening files in binary mode like expected from openbin. Closes #406.

Source code(tar.gz)
Source code(zip)
v2.4.11(Sep 7, 2019)
[2.4.11] - 2019-09-07

Added

Added geturl for TarFS and ZipFS for 'fs' purpose. NoURL for 'download' purpose.

Added helpful root path in CreateFailed exception #340

Added Python 3.8 support

Fixed

Fixed tests leaving tmp files

Fixed typing issues

Fixed link namespace returning bytes

Fixed broken FSURL in windows #329

Fixed hidden exception at fs.close() when opening an absent zip/tar file URL #333

Fixed abstract class import from collections which would break on Python 3.8

Fixed incorrect imports of mock on Python 3

Removed some unused imports and unused requirements.txt file

Added mypy checks to Travis. Closes #332.

Fixed missing errno.ENOTSUP on PyPy. Closes #338.

Fixed bug in a decorator that would trigger an AttributeError when a class was created that implemented a deprecated method and had no docstring of its own.

Changed

Entire test suite has been migrated to pytest. Closes #327.

Style checking is now enforced using flake8; this involved some code cleanup such as removing unused imports.

Source code(tar.gz)
Source code(zip)
v2.4.10(Jul 29, 2019)
[2.4.10] - 2019-07-29

Fixed

Fixed broken WrapFS.movedir #322

Source code(tar.gz)
Source code(zip)
v2.4.9(Jul 28, 2019)
[2.4.9] - 2019-07-28

Fixed

Restored fs.path import

Fixed potential race condition in makedirs. Fixes #310

Added missing methods to WrapFS. Fixed #294

Changed

MemFS now immediately releases all memory it holds when close() is called, rather than when it gets garbage collected. Closes issue #308.

FTPFS now translates EOFError into RemoteConnectionError. Closes #292

Added automatic close for filesystems that go out of scope. Fixes #298

Source code(tar.gz)
Source code(zip)
v2.4.8(Jun 12, 2019)
[2.4.8] - 2019-06-12

Changed

geturl will return URL with user/password if needed @zmej-serow

Source code(tar.gz)
Source code(zip)
v2.4.7(Jun 8, 2019)
[2.4.7] - 2019-06-08

Added

Flag to OSFS to disable env var expansion

Source code(tar.gz)
Source code(zip)
v2.4.6(Jun 8, 2019)
[2.4.6] - 2019-06-08

Added

Implemented geturl in FTPFS @zmej-serow

Fixed

Fixed FTP test suite when time is not UTC-0 @mrg0029

Fixed issues with paths in tarfs https://github.com/PyFilesystem/pyfilesystem2/issues/284

Changed

Dropped Python3.3 support

Source code(tar.gz)
Source code(zip)
v2.4.5(May 5, 2019)
[2.4.5] - 2019-05-05

Fixed

Restored deprecated setfile method with deprecation warning to change to writefile

Fixed exception when a tarfile contains a path called '.' https://github.com/PyFilesystem/pyfilesystem2/issues/275

Made TarFS directory loading lazy

Changed

Detect case insensitivity using by writing temp file

Source code(tar.gz)
Source code(zip)
v2.4.4(Feb 23, 2019)
[2.4.4] - 2019-02-23

Fixed

OSFS fail in nfs mounts

Source code(tar.gz)
Source code(zip)
v2.4.3(Feb 23, 2019)
[2.4.3] - 2019-02-23

Fixed

Fixed broken "case_insensitive" check

Fixed Windows test fails

Source code(tar.gz)
Source code(zip)
v2.4.2(Feb 22, 2019)
[2.4.2] - 2019-02-22

Fixed

Fixed exception when Python runs with -OO

Source code(tar.gz)
Source code(zip)
v2.4.1(Feb 20, 2019)
[2.4.1] - 2019-02-20

Fixed

Fixed hash method missing from WrapFS

Source code(tar.gz)
Source code(zip)
v2.4.0(Feb 15, 2019)
[2.4.0] - 2019-02-15

Added

Added exclude and filter_dirs arguments to walk

Micro-optimizations to walk

Source code(tar.gz)
Source code(zip)
v2.3.1(Feb 10, 2019)
[2.3.1] - 2019-02-10

Fixed

Add encoding check in OSFS.validatepath

Source code(tar.gz)
Source code(zip)
v2.3.0(Jan 30, 2019)
[2.3.0] - 2019-01-30

Fixed

IllegalBackReference had mangled error message

Added

FS.hash method

Source code(tar.gz)
Source code(zip)
v2.2.1(Jan 6, 2019)
[2.2.1] - 2019-01-06

Fixed

Registry.install returns its argument.

Source code(tar.gz)
Source code(zip)
v2.2.0(Jan 2, 2019)
[2.2.0] - 2019-01-01

A few methods have been renamed for greater clarity (but functionality remains the same).

The old methods are now aliases and will continue to work, but will issue a deprecation warning via the warnings module. Please update your code accordingly.

getbytes -> readbytes

getfile -> download

gettext -> readtext

setbytes -> writebytes

setbinfile -> upload

settext -> writetext

Changed

Changed default chunk size in copy_file_data to 1MB

Added chunk_size and options to FS.upload

Source code(tar.gz)
Source code(zip)
v2.1.3(Dec 24, 2018)
[2.1.3] - 2018-12-24

Fixed

Incomplete FTPFile.write when using workers @geoffjukes

Fixed AppFS not creating directory

Added

Added load_extern switch to opener, fixes #228 @althanos

Source code(tar.gz)
Source code(zip)
v2.1.2(Nov 10, 2018)
[2.1.2] - 20180-11-10

Added

Support for Windows NT FTP servers @sspross

Fixed

Root dir of MemoryFS accessible as a file

Packaging issues @televi

Deprecation warning re collections.Mapping

Source code(tar.gz)
Source code(zip)
v2.1.1(Oct 3, 2018)
[2.1.1] - 2018-10-03

Added

Added PEP 561 py.typed files

Use sendfile for faster copies @althonos

Atomic exclusive mode in Py2.7 @sqwishy

Fixed

Fixed lstat @kamomil

Source code(tar.gz)
Source code(zip)
v2.1.0(Aug 12, 2018)
[2.1.0] - 2018-08-12

Added

fs.glob support

Source code(tar.gz)
Source code(zip)
v2.0.27(Aug 5, 2018)
[2.0.27] - 2018-08-05

Fixed

Fixed for Winows paths #152

Fixed ftp dir parsing (@dhirschfeld)

Source code(tar.gz)
Source code(zip)
v2.0.26(Jul 26, 2018)
Fixed

fs.copy and fs.move disable workers if not thread-safe

fs.match detects case insensitivity

Open in exclusive mode is atomic (@squishy)

Exceptions can be pickleabe (@Spacerat)

Source code(tar.gz)
Source code(zip)
v2.0.25(Jul 20, 2018)
Added

workers parameter to fs.copy, fs.move, and fs.mirror for concurrent copies

Source code(tar.gz)
Source code(zip)
v2.0.24(Jun 28, 2018)

[2.0.24] - 2018-06-28

Added timeout to FTP opener
Source code(tar.gz)
Source code(zip)
v2.0.23(May 19, 2018)
[2.0.23] - 2018-05-02

Fix for Markdown on PyPi, no code changes

[2.0.22] - 2018-05-02

Fixed

Handling of broken unicode on Python2.7

Added

Added fs.getospath

Source code(tar.gz)
Source code(zip)

Owner

pyFilesystem

GitHub Repository https://www.pyfilesystem.org

Python virtual filesystem for SQLite to read from and write to S3

70 Jan 04, 2023

Object-oriented file system path manipulation

path (aka path pie, formerly path.py) implements path objects as first-class entities, allowing common operations on files to be invoked on those path

1k Dec 28, 2022

Organize the files into the relevant sub-folders

This program can be used to organize files in a directory by their file extension. And move duplicate files to a duplicates folder.

2 Dec 15, 2021

Various converters to convert value sets from CSV to JSON, etc.

ValueSet Converters Tools for converting value sets in different formats. Such as converting extensional value sets in CSV format to JSON format able

4 Sep 08, 2022

Convert CSV files into a SQLite database

csvs-to-sqlite Convert CSV files into a SQLite database. Browse and publish that SQLite database with Datasette. Basic usage: csvs-to-sqlite myfile.cs

731 Dec 27, 2022

Import Python modules from any file system path

pathimp Import Python modules from any file system path. Installation pip3 install pathimp Usage import pathimp

2 Nov 29, 2021

Python function to stream unzip all the files in a ZIP archive: without loading the entire ZIP file or any of its files into memory at once

206 Jan 02, 2023

CSV To VCF (Multiples en un archivo)

CSV To VCF Convierte archivo CSV a Tarjeta VCF (varias en una) How to use En main.py debes reemplazar CONTACTOS.csv por tu archivo csv, y debes respet

2 Jan 12, 2022

Find potentially sensitive files

find_files Find potentially sensitive files This script searchs for potentially sensitive files based off of file name or string contained in the file

4 Aug 20, 2022

Nintendo Game Boy music assembly files parser into musicxml format

GBMusicParser Nintendo Game Boy music assembly files parser into musicxml format This python code will get an file.asm from the disassembly of a Game

1 Dec 11, 2021

A python wrapper for libmagic

python-magic python-magic is a Python interface to the libmagic file type identification library. libmagic identifies file types by checking their hea

2.3k Dec 29, 2022

LightCSV - This CSV reader is implemented in just pure Python.

LightCSV Simple light CSV reader This CSV reader is implemented in just pure Python. It allows to specify a separator, a quote char and column titles

6 Mar 05, 2022

QSynthesis is a Python3 API to perform I/O based program synthesis of bitvector expressions.

QSynthesis is a Python3 API to perform I/O based program synthesis of bitvector expressions. It aims at facilitating code deobfuscation. The algorithm is greybox approach combining both a blackbox I/

103 Dec 30, 2022

ZipFly is a zip archive generator based on zipfile.py

ZipFly is a zip archive generator based on zipfile.py. It was created by Buzon.io to generate very large ZIP archives for immediate sending out to clients, or for writing large ZIP archives without m

506 Jan 04, 2023

Extract longest transcript or longest CDS transcript from GTF annotation file or gencode transcripts fasta file.

13 Nov 23, 2022

This python project contains a class FileProcessor which allows one to grab a file and get some meta data and header information from it

This python project contains a class FileProcessor which allows one to grab a file and get some meta data and header information from it. In the current state, it outputs a PrettyTable to txt file as

1 Nov 09, 2021

Python's Filesystem abstraction layer

Related tags

Overview

PyFilesystem2

Documentation

Introduction

Credits

Support

Comments

New tests

Discovered bugs

About FTPFS

About test_ftpfs

Type of changes

Checklist

Description

Bugs / Cleanup

Type of changes

Checklist

Description

Type of changes

Checklist

Description

Type of changes

Checklist

Description

Type of changes

Checklist

Description

Releases(2.4.16)

2.4.16(May 2, 2022)

Changed

2.4.15(Feb 7, 2022)

Changed

Fixed

v2.4.14(Nov 16, 2021)

Added

Changed

Fixed

v2.4.13(Mar 27, 2021)

Added

Changed

Fixed

v2.4.12(Jan 14, 2021)

Added

Changed

Fixed

v2.4.11(Sep 7, 2019)

[2.4.11] - 2019-09-07

Added

Fixed

Changed

v2.4.10(Jul 29, 2019)

[2.4.10] - 2019-07-29

Fixed

v2.4.9(Jul 28, 2019)

[2.4.9] - 2019-07-28

Fixed

Changed

v2.4.8(Jun 12, 2019)

[2.4.8] - 2019-06-12

Changed

v2.4.7(Jun 8, 2019)

[2.4.7] - 2019-06-08

Added

v2.4.6(Jun 8, 2019)

[2.4.6] - 2019-06-08

Added

Fixed

Changed

v2.4.5(May 5, 2019)

[2.4.5] - 2019-05-05

Fixed

Changed

v2.4.4(Feb 23, 2019)

[2.4.4] - 2019-02-23

Fixed

v2.4.3(Feb 23, 2019)

[2.4.3] - 2019-02-23

Fixed