Hi, here's a HTML Table Set importer for messytables.

It's not fantastic yet; but it's a pretty good start

• Supports rowspan/colspan - currently by inserting blank cells.
• Supports multiple TABLE elements - but may have unexpected behaviour where there are nested tables.
• Doesn't attempt to handle tables that aren't using TABLE, TR, TD, TH.
• Not enormously well tested, but seems to work on the tables I've fed it so far.
• Requires lxml.

It's the first time I've ever made a pull request; let us know if there's anything we can do to improve it for you.

json-table-schema is a broken dependency as of yesterday. This affects current and previous releases on pypi.

To fix this at this end we've changed the dep https://github.com/okfn/messytables/pull/143 and now messytables installs from source again, but it needs a release to pypi. I don't have permission for this.

(test)[email protected]:/tmp$ pip install messytables
Downloading/unpacking messytables
  Downloading messytables-0.15.0.tar.gz
  Running setup.py egg_info for package messytables

Downloading/unpacking xlrd>=0.8.0 (from messytables)
  Downloading xlrd-0.9.4.tar.gz (322Kb): 322Kb downloaded
  Running setup.py egg_info for package xlrd

Downloading/unpacking python-magic>=0.4.6 (from messytables)
  Downloading python-magic-0.4.10.tar.gz
  Running setup.py egg_info for package python-magic

    no previously-included directories found matching 'test'
Downloading/unpacking chardet>=2.3.0 (from messytables)
  Downloading chardet-2.3.0.tar.gz (164Kb): 164Kb downloaded
  Running setup.py egg_info for package chardet

    warning: no files found matching 'COPYING'
    warning: no files found matching '*.html' under directory 'docs'
    warning: no files found matching '*.css' under directory 'docs'
    warning: no files found matching '*.png' under directory 'docs'
    warning: no files found matching '*.gif' under directory 'docs'
Downloading/unpacking python-dateutil>=2.4.2 (from messytables)
  Downloading python-dateutil-2.4.2.tar.gz (209Kb): 209Kb downloaded
  Running setup.py egg_info for package python-dateutil

Downloading/unpacking lxml>=3.2 (from messytables)
  Downloading lxml-3.5.0b1.tar.gz (3.8Mb): 3.8Mb downloaded
  Running setup.py egg_info for package lxml
    Building lxml version 3.5.0b1.
    Building without Cython.
    Using build configuration of libxslt 1.1.26
    Building against libxml2/libxslt in the following directory: /usr/lib/x86_64-linux-gnu

    warning: no previously-included files found matching '*.py'
Downloading/unpacking requests (from messytables)
  Downloading requests-2.8.1.tar.gz (480Kb): 480Kb downloaded
  Running setup.py egg_info for package requests

Downloading/unpacking html5lib (from messytables)
  Downloading html5lib-1.0b8.tar.gz (889Kb): 889Kb downloaded
  Running setup.py egg_info for package html5lib

Downloading/unpacking json-table-schema>=0.2 (from messytables)
  Downloading json-table-schema-0.5.0.tar.gz
  Running setup.py egg_info for package json-table-schema
    json-table-schema has been replaced by jsontableschema. See https://github.com/okfn/json-table-schema-py-old for details.
    Traceback (most recent call last):
      File "<string>", line 14, in <module>
      File "/tmp/test/build/json-table-schema/setup.py", line 16, in <module>
        with io.open(README_PATH, mode='r+t', encoding='utf-8') as stream:
    IOError: [Errno 2] No such file or directory: '/tmp/test/build/json-table-schema/README.md'
    Complete output from command python setup.py egg_info:
    json-table-schema has been replaced by jsontableschema. See https://github.com/okfn/json-table-schema-py-old for details.

Traceback (most recent call last):

  File "<string>", line 14, in <module>

  File "/tmp/test/build/json-table-schema/setup.py", line 16, in <module>

    with io.open(README_PATH, mode='r+t', encoding='utf-8') as stream:

IOError: [Errno 2] No such file or directory: '/tmp/test/build/json-table-schema/README.md'

----------------------------------------
Command python setup.py egg_info failed with error code 1 in /tmp/test/build/json-table-schema
Storing complete log in /home/co/.pip/pip.log

Does any know, informally or otherwise, what it will take to get messytables running on Python 3?

I'm keen to use various functions and modules from messytables, but I'm trying to maintain 2.7/3.3/3.4 support in my own libraries.

Hey all. This repository seems to be semi-inactive, and it unclear to me what the path to merging a PR like #171 is (who would have to approve?). I use messytables in production code day to day, and this lack of clarity on process makes the library a liability. My understanding is that okfn's resources and interest is focussed on goodtables and the frictionlessdata toolchain.

I would therefore like to apply to become the maintainer for messytables, merge #171 & co., and generally make sure that changes in this thing are handled and bugs are actively tracked.

Thoughts, @pwalsh, @davidread, @rufuspollock? Please let me know.

I could trace this back to #141 where len() is being used in the test() method of DateUtilType.

I think there should be a try/except block around that, that catches this TypeError. But I'm not too familiar with the code, so I'm basically asking if you agree, or if I'm missing something.

I'm happy to provide the PR.

BTW: I'm getting this error via datapusher on some Excel sheet that is being parsed with the default parameters. The excel sheet has indeed a lot of float values in it.

Messytables doesn't work well in a lot of situations when the provided fileobj is a socket.

The BufferedFile object attempts to resolve this, but in a lot of cases it will force a read(-1) and cause a complete download of the file (into ram) anyway. This is particularly true of anything that that wants to seek within the file (such as zip and xls) or the buffer passed to magic.from_buffer (which is inadequate in some cases and from_file would be more accurate).

Downloading the content to temporary storage isn't an onerous task, and if the interface was modified to use filenames instead of file-objects it could even transparently download the content when a url is provided (which is is destined to do anyway at some point).

We've been exploring different options for parsing PDFs. Currently we're using an (alpha) in-house library called pdftables (we blogged about it here)

This pull request integrates pdftables into messytables. It is an optional requirement - if pdftables is not installed, messytables will work as usual and the PDF tests will be skipped.

We're looking into other ways of extracting tables from PDFs, but either way we'll need the messytables integration.

A reworked reader for ODS files that doesn't use any broken third-party libraries. Reads the .xml directly from the zipfile and performs much better on larger spreadsheets.

I'm based off of http://messytables.readthedocs.org/en/latest/ but have also looked at the GitHub readme, etc. Couldn't find any actual install instructions anywhere, but here's what I did.

Environment: Mac OS X latest, up to date homebrew

1. pip install messytables

2. brew install libmagic

3. The following Python:

   % python                
   Python 2.7.6 (default, Nov 14 2013, 09:55:56) 
   [GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.2.79)] on darwin
   Type "help", "copyright", "credits" or "license" for more information.
   >>> import messytables
   >>> messytables.any_tableset(open('README.txt', 'rb'))
   Traceback (most recent call last):
     File "<stdin>", line 1, in <module>
     File "/usr/local/lib/python2.7/site-packages/messytables/any.py", line 138, in any_tableset
       magic_mime = get_mime(fileobj)
     File "/usr/local/lib/python2.7/site-packages/messytables/any.py", line 38, in get_mime
       mimetype = magic.from_buffer(header, mime=True)
     File "build/bdist.macosx-10.9-x86_64/egg/magic.py", line 103, in from_buffer
       def __init__(self, ms):
     File "build/bdist.macosx-10.9-x86_64/egg/magic.py", line 94, in _get_magic_type
       _list = _libraries['magic'].magic_list
     File "build/bdist.macosx-10.9-x86_64/egg/magic.py", line 83, in _get_magic_mime
       _load.restype = c_int
     File "build/bdist.macosx-10.9-x86_64/egg/magic.py", line 51, in __init__
       magic_set._fields_ = []
     File "build/bdist.macosx-10.9-x86_64/egg/magic.py", line 138, in errorcheck
       except:
   magic.MagicException: no magic files loaded
   

4. README.txt:

   ============
   README
   ============
   
   A single-line README file.
   

Phase 1 of 2 for completely removing openpyxl and using XLSTableSet instead. (Phase 2 will actually remove the dependency and excelx.py, then you won't be able to reference XLSXTableSet)

If you always use any_tableset it'll just work correctly - you'll now get back an XLSTableSet instead of an XLSXTableSet.

I've left the latter in with a DeprecationWarning (and test) in order to remain compatible with code written with explicity XLSXTableSet.

I'm feeling like we should encourage people towards only using any_tableset (perhaps with an argument to override force the type detection). It's quite awkward that currently our users are needlessly coupling to our class naming convention. Unless I've missed a use-case - any compelling reasons to allow that?

Not ready to merge yet I suspect. Closes #83

We were having problems with any.py, so I rewrote it.

Features:

• new extension detection function (you can pass a whole filename/URL)
• nice lists of mimetypes/extensions parsed
• special pleading for XLS/XLSX files :(
• tests for autodetection
• various fixes

Attempting to use messytables with Python 3.10 results in the following error:

  File "/layers/google.python.pip/pip/lib/python3.10/site-packages/messytables/core.py", line 2, in <module>
    from collections import Mapping
ImportError: cannot import name 'Mapping' from 'collections' (/opt/python3.10/lib/python3.10/collections/__init__.py)

This is due to Mapping moving to package collections.abc in Python 3.10.

core.py should be updated to take account of this.

Bumps lxml from 4.3.4 to 4.9.1.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Describe the bug Messytables should guess decimals correctly respecting the locale configuration. For example: In germany the , is used as decimal dot but a value 1,200 is guessed as type "text".

This issue was initially reported as ckan issue https://github.com/ckan/ckan/issues/5769 where I recognized it.

The type guessing seems to happen here: https://github.com/okfn/messytables/blob/51b736892a48e420ab313675f54901c77b446dec/messytables/types.py and seems to happen locale specific. (I think the magic happens in line 100: value = locale.atof(value)

Unfortunately python seems to recognizes a dot as decimal point even if a german locale is set, which I could reproduce in my local environment:

>>> locale.getlocale()
('de_DE', 'cp1252')
>>> locale.atof('1,200')

Traceback (most recent call last):
  File "<pyshell#35>", line 1, in <module>
    locale.atof('1,200')
  File "C:\Program Files\Python27\lib\locale.py", line 318, in atof
    return func(string)
ValueError: invalid literal for float(): 1,200
>>> locale.localeconv()
{'mon_decimal_point': '', 'int_frac_digits': 127, 'p_sep_by_space': 127, 'frac_digits': 127, 'thousands_sep': '', 'n_sign_posn': 127, 'decimal_point': '.', 'int_curr_symbol': '', 'n_cs_precedes': 127, 'p_sign_posn': 127, 'mon_thousands_sep': '', 'negative_sign': '', 'currency_symbol': '', 'n_sep_by_space': 127, 'mon_grouping': [], 'p_cs_precedes': 127, 'positive_sign': '', 'grouping': []}

This line specifies an error message. In the test, the text of the exception caused by the code under test is expected to match exactly.

errmsg = "Can't read Excel file: XLRDError('Workbook is encrypted',)"

When running tests on Python 3.7 and 3.8 this fails, because their outputs do not contain the comma (probably due to this change in Python 3.7, I'm guessing).

This version of xlrd is currently pinned for testing on Travis in requirements-test.txt.

Prior to v1.2.0, xlrd used the time.clock() function inside book.py and this was removed in Python 3.8.