Summary

This PR adds complete numpy backend support for the StandardSet of types. The type implementation is fully compatible with the pandas equivalent implementations with the exception of the object type.

Caveats

Whereas pandas provides support for Optional[int] and Optional[bool] numpy doesn't - in order to support those types by default I was forced to make object completely disjoint from any other concrete type. A similar story plays out for datetime objects with timezones which also default to object in numpy.

(This is part of the review in openjournals/joss-reviews#2145)

Hi @sbrugman,

I am currently going through the package and I found it a very interesting project. The type inference that already exists in built-in Python is frequently not enough, and I often find myself writing my own functions for it on a case-by-case basis. So, it is nice to see that this is being done.

However, I do find myself having quite a bit of trouble using the package effectively. Funnily enough, this is mostly caused by the (in my opinion) confusing naming schemes and the structure of the namespace. It may require a bit of effort to solve (not to mention that it might create incompatibilities with previous versions), but I believe that it will greatly improve the user experience when fixed.

So, the main problem in my opinion is that almost all definitions are stored in their own subpackages in the visions.core subpackage, often separated as well over an implementations and a model subpackage. This means that in order to access a definition, let's say VisionsTypeset, I have to import this from visions.core.model.typeset, while pre-defined typesets I have to import from visions.core.implementations.typesets. In my opinion, it is incredibly confusing that these are stored in different subpackages/submodules, as they are related to the same thing, namely typesets, and I would expect to find all of these definitions in a visions.typesets subpackage. Preferably, all definitions the average user would use, should be available either at root (visions) or a single level deep (visions.xxx).

I also noticed that almost all definitions have the word visions in their name. I get the feeling that the reason for this is to avoid namespace clashes when someone uses a wildcard import (from visions import *). However, as wildcard imports are heavily discouraged in the community, this leads to the user writing the word visions at least twice for every definition used (for example, using the visions_integer type requires me to write visions.visions_integer, which could be simplified to visions.integer or even visions.int).

Finally, I am not entirely sure if this has to do with the online documentation being outdated as mentioned in #21, but according to the example here, a visions_standard_set object has a method called detect_type_series. In v0.2.3, this object neither has a method called detect_type_series nor type_detect_series (the name that the stand-alone function has in visions.core.functional), but instead it is called detect_series_type. If possible, could you check and make sure that the methods and stand-alone functions use consistent naming schemes?

Please let me know if you have any questions.

I have a question about how to use the library. I considered opening an issue, but I see in the documentation that you recommend asking questions about how to use the package on Stack Overflow. Is there a tag that you'd suggest people use when asking questions there? I don't see anything with visions as a tag, but maybe I'm just the first person to ask a question over there.

If you think visions would be a good tag choice it would make sense to update the stack overflow ask a question link to pre-populate the question with the tag (https://stackoverflow.com/questions/ask?tags=visions).

Thanks!

The documentation should rebuilt at every merge. The differences caused by the documentation convolute code reviews. The steps to build the documentation are simple and can be automated.

Suggested solution via Github Actions (e.g.): https://github.com/marketplace/actions/sphinx-build https://github.com/ammaraskar/sphinx-action-test/blob/master/.github/workflows/default.yml

Describe the bug visions uses the @attr.s decorator which comes from the attrs module, not the attr module. The master version of visions has the correct dependency, but the pypi versions do not.

Additional context When using pandas_profiling which depends on visions, I got the following error:

AttributeError: module 'attr' has no attribute 's'

which led me to post this issue.

0.7.5 Includes:

• Fixes to numpy backend for complex, object, email_address, URL, boolean
• Support for new versions of pandas ABCIndex class (previously called ABCIndexClass)
• Updated tests for numpy backend
• Automated Github Actions unit tests on PR
• Additional documentation

Describe the bug The tests failed with 0.6.1 release. To Reproduce Steps to reproduce the behavior:

python setup.py build
PYTHONPATH=build/lib pytest -v

Expected behavior pass all tests

Additional context error log:

=================================== FAILURES ===================================
_____________________ test_contains[file_mixed_ext x File] _____________________

series = 0    /build/python-visions/src/visions-0.6.1/build/...
1    /build/python-visions/src/visions-0.6.1/build/...
2    /build/python-visions/src/visions-0.6.1/build/...
Name: file_mixed_ext, dtype: object
type = File, member = True

    @pytest.mark.parametrize(**get_contains_cases(series, contains_map, typeset))
    def test_contains(series, type, member):
        """Test the generated combinations for "series in type"
    
        Args:
            series: the series to test
            type: the type to test against
            member: the result
        """
        result, message = contains(series, type, member)
>       assert result, message
E       AssertionError: file_mixed_ext in File; expected True, got False
E       assert False

tests/typesets/test_complete_set.py:190: AssertionError
_______________________ test_contains[image_png x File] ________________________

series = 0    /build/python-visions/src/visions-0.6.1/build/...
1    /build/python-visions/src/visions-0.6.1/build/...
2    /build/python-visions/src/visions-0.6.1/build/...
Name: image_png, dtype: object
type = File, member = True

    @pytest.mark.parametrize(**get_contains_cases(series, contains_map, typeset))
    def test_contains(series, type, member):
        """Test the generated combinations for "series in type"
    
        Args:
            series: the series to test
            type: the type to test against
            member: the result
        """
        result, message = contains(series, type, member)
>       assert result, message
E       AssertionError: image_png in File; expected True, got False
E       assert False

tests/typesets/test_complete_set.py:190: AssertionError
_______________________ test_contains[image_png x Image] _______________________

series = 0    /build/python-visions/src/visions-0.6.1/build/...
1    /build/python-visions/src/visions-0.6.1/build/...
2    /build/python-visions/src/visions-0.6.1/build/...
Name: image_png, dtype: object
type = Image, member = True

    @pytest.mark.parametrize(**get_contains_cases(series, contains_map, typeset))
    def test_contains(series, type, member):
        """Test the generated combinations for "series in type"
    
        Args:
            series: the series to test
            type: the type to test against
            member: the result
        """
        result, message = contains(series, type, member)
>       assert result, message
E       AssertionError: image_png in Image; expected True, got False
E       assert False

tests/typesets/test_complete_set.py:190: AssertionError
___________________ test_contains[image_png_missing x File] ____________________

series = 0    /build/python-visions/src/visions-0.6.1/build/...
1    /build/python-visions/src/visions-0.6.1/build/...
2       ...c/visions-0.6.1/build/...
4                                                 None
Name: image_png_missing, dtype: object
type = File, member = True

    @pytest.mark.parametrize(**get_contains_cases(series, contains_map, typeset))
    def test_contains(series, type, member):
        """Test the generated combinations for "series in type"
    
        Args:
            series: the series to test
            type: the type to test against
            member: the result
        """
        result, message = contains(series, type, member)
>       assert result, message
E       AssertionError: image_png_missing in File; expected True, got False
E       assert False

tests/typesets/test_complete_set.py:190: AssertionError
___________________ test_contains[image_png_missing x Image] ___________________

series = 0    /build/python-visions/src/visions-0.6.1/build/...
1    /build/python-visions/src/visions-0.6.1/build/...
2       ...c/visions-0.6.1/build/...
4                                                 None
Name: image_png_missing, dtype: object
type = Image, member = True

    @pytest.mark.parametrize(**get_contains_cases(series, contains_map, typeset))
    def test_contains(series, type, member):
        """Test the generated combinations for "series in type"
    
        Args:
            series: the series to test
            type: the type to test against
            member: the result
        """
        result, message = contains(series, type, member)
>       assert result, message
E       AssertionError: image_png_missing in Image; expected True, got False
E       assert False

tests/typesets/test_complete_set.py:190: AssertionError
_____________ test_inference[file_mixed_ext x File expected True] ______________

series = 0    /build/python-visions/src/visions-0.6.1/build/...
1    /build/python-visions/src/visions-0.6.1/build/...
2    /build/python-visions/src/visions-0.6.1/build/...
Name: file_mixed_ext, dtype: object
type = File, typeset = CompleteSet, difference = False

    @pytest.mark.parametrize(**get_inference_cases(series, inference_map, typeset))
    def test_inference(series, type, typeset, difference):
        """Test the generated combinations for "inference(series) == type"
    
        Args:
            series: the series to test
            type: the type to test against
        """
        result, message = infers(series, type, typeset, difference)
>       assert result, message
E       AssertionError: inference of file_mixed_ext expected File to be True (typeset=CompleteSet)
E       assert False

tests/typesets/test_complete_set.py:317: AssertionError
_____________ test_inference[file_mixed_ext x Path expected False] _____________

series = 0    /build/python-visions/src/visions-0.6.1/build/...
1    /build/python-visions/src/visions-0.6.1/build/...
2    /build/python-visions/src/visions-0.6.1/build/...
Name: file_mixed_ext, dtype: object
type = Path, typeset = CompleteSet, difference = True

    @pytest.mark.parametrize(**get_inference_cases(series, inference_map, typeset))
    def test_inference(series, type, typeset, difference):
        """Test the generated combinations for "inference(series) == type"
    
        Args:
            series: the series to test
            type: the type to test against
        """
        result, message = infers(series, type, typeset, difference)
>       assert result, message
E       AssertionError: inference of file_mixed_ext expected Path to be False (typeset=CompleteSet)
E       assert False

tests/typesets/test_complete_set.py:317: AssertionError
_______________ test_inference[image_png x Image expected True] ________________

series = 0    /build/python-visions/src/visions-0.6.1/build/...
1    /build/python-visions/src/visions-0.6.1/build/...
2    /build/python-visions/src/visions-0.6.1/build/...
Name: image_png, dtype: object
type = Image, typeset = CompleteSet, difference = False

    @pytest.mark.parametrize(**get_inference_cases(series, inference_map, typeset))
    def test_inference(series, type, typeset, difference):
        """Test the generated combinations for "inference(series) == type"
    
        Args:
            series: the series to test
            type: the type to test against
        """
        result, message = infers(series, type, typeset, difference)
>       assert result, message
E       AssertionError: inference of image_png expected Image to be True (typeset=CompleteSet)
E       assert False

tests/typesets/test_complete_set.py:317: AssertionError
_______________ test_inference[image_png x Path expected False] ________________

series = 0    /build/python-visions/src/visions-0.6.1/build/...
1    /build/python-visions/src/visions-0.6.1/build/...
2    /build/python-visions/src/visions-0.6.1/build/...
Name: image_png, dtype: object
type = Path, typeset = CompleteSet, difference = True

    @pytest.mark.parametrize(**get_inference_cases(series, inference_map, typeset))
    def test_inference(series, type, typeset, difference):
        """Test the generated combinations for "inference(series) == type"
    
        Args:
            series: the series to test
            type: the type to test against
        """
        result, message = infers(series, type, typeset, difference)
>       assert result, message
E       AssertionError: inference of image_png expected Path to be False (typeset=CompleteSet)
E       assert False

tests/typesets/test_complete_set.py:317: AssertionError
___________ test_inference[image_png_missing x Image expected True] ____________

series = 0    /build/python-visions/src/visions-0.6.1/build/...
1    /build/python-visions/src/visions-0.6.1/build/...
2       ...c/visions-0.6.1/build/...
4                                                 None
Name: image_png_missing, dtype: object
type = Image, typeset = CompleteSet, difference = False

    @pytest.mark.parametrize(**get_inference_cases(series, inference_map, typeset))
    def test_inference(series, type, typeset, difference):
        """Test the generated combinations for "inference(series) == type"
    
        Args:
            series: the series to test
            type: the type to test against
        """
        result, message = infers(series, type, typeset, difference)
>       assert result, message
E       AssertionError: inference of image_png_missing expected Image to be True (typeset=CompleteSet)
E       assert False

tests/typesets/test_complete_set.py:317: AssertionError
___________ test_inference[image_png_missing x Path expected False] ____________

series = 0    /build/python-visions/src/visions-0.6.1/build/...
1    /build/python-visions/src/visions-0.6.1/build/...
2       ...c/visions-0.6.1/build/...
4                                                 None
Name: image_png_missing, dtype: object
type = Path, typeset = CompleteSet, difference = True

    @pytest.mark.parametrize(**get_inference_cases(series, inference_map, typeset))
    def test_inference(series, type, typeset, difference):
        """Test the generated combinations for "inference(series) == type"
    
        Args:
            series: the series to test
            type: the type to test against
        """
        result, message = infers(series, type, typeset, difference)
>       assert result, message
E       AssertionError: inference of image_png_missing expected Path to be False (typeset=CompleteSet)
E       assert False

tests/typesets/test_complete_set.py:317: AssertionError
=============================== warnings summary ===============================
tests/test_root.py::test_multiple_roots
  /build/python-visions/src/visions-0.6.1/build/lib/visions/typesets/typeset.py:88: UserWarning: {Generic} were isolates in the type relation map and consequently orphaned. Please add some mapping to the orphaned nodes.
    warnings.warn(message)

tests/test_summarization.py::test_complex_missing_summary
  /usr/lib/python3.8/site-packages/numpy/core/_methods.py:47: ComplexWarning: Casting complex values to real discards the imaginary part
    return umr_sum(a, axis, dtype, out, keepdims, initial, where)

-- Docs: https://docs.pytest.org/en/stable/warnings.html
=========================== short test summary info ============================
FAILED tests/typesets/test_complete_set.py::test_contains[file_mixed_ext x File]
FAILED tests/typesets/test_complete_set.py::test_contains[image_png x File]
FAILED tests/typesets/test_complete_set.py::test_contains[image_png x Image]
FAILED tests/typesets/test_complete_set.py::test_contains[image_png_missing x File]
FAILED tests/typesets/test_complete_set.py::test_contains[image_png_missing x Image]
FAILED tests/typesets/test_complete_set.py::test_inference[file_mixed_ext x File expected True]
FAILED tests/typesets/test_complete_set.py::test_inference[file_mixed_ext x Path expected False]
FAILED tests/typesets/test_complete_set.py::test_inference[image_png x Image expected True]
FAILED tests/typesets/test_complete_set.py::test_inference[image_png x Path expected False]
FAILED tests/typesets/test_complete_set.py::test_inference[image_png_missing x Image expected True]
FAILED tests/typesets/test_complete_set.py::test_inference[image_png_missing x Path expected False]
================= 11 failed, 8954 passed, 2 warnings in 15.94s =================

see also the complete build log here

(This is part of the review in openjournals/joss-reviews#2145)

When I try to execute the example given here, I get an error stating No module named 'shapely'. I see that this is a dependency of visions, but is only listed in the requirements_test.txt. You probably have to add this requirement to the requirements.txt as well.

PS: Currently, the requirements of the package are both listed in their own separate file and in the setup.py file. To avoid confusion for yourself, it is probably better to only use either. You can read in a requirements file and use it in the setup.py file by using:

# Get the requirements list
with open('requirements.txt', 'r') as f:
    requirements = f.read().splitlines()

Keep in mind that it is possible to link different requirements files together. For example, you can link requirements.txt and requirements_dev.txt together by adding the line -r requirements.txt to the top of requirements_dev.txt. This means that installing the requirements of requirements_dev.txt will use both files. This however won't work if you parse the file in a setup.py file. In that case, you can simply read both files and append them together if necessary.

Bumps pyarrow from 1.0.1 to 5.0.0.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

i don't think this should be expected behavior.

In [41]: infer_type(pd.DataFrame({'x':['', '']}), StandardSet())
Out[41]: {'x': DateTime}

In [39]: infer_type(pd.DataFrame({'x':[None, None]}), StandardSet())
Out[39]: {'x': Boolean}

New functionality

• comma separator handling for string digits
• new utility functionality for working with missing values

Major Proposed Changes

• Integer should be a strict subset of Float

Hi dylan-profiler/visions!

This is a one-off automatically generated pull request from LGTM.com :robot:. You might have heard that we’ve integrated LGTM’s underlying CodeQL analysis engine natively into GitHub. The result is GitHub code scanning!

With LGTM fully integrated into code scanning, we are focused on improving CodeQL within the native GitHub code scanning experience. In order to take advantage of current and future improvements to our analysis capabilities, we suggest you enable code scanning on your repository. Please take a look at our blog post for more information.

This pull request enables code scanning by adding an auto-generated codeql.yml workflow file for GitHub Actions to your repository — take a look! We tested it before opening this pull request, so all should be working :heavy_check_mark:. In fact, you might already have seen some alerts appear on this pull request!

Where needed and if possible, we’ve adjusted the configuration to the needs of your particular repository. But of course, you should feel free to tweak it further! Check this page for detailed documentation.

Questions? Check out the FAQ below!

FAQ

I've recently been made aware of this excellent and imo much needed library by @lmmentel.

The reason is its similarity to the datatypes module of sktime, which introduces semantic typing for time series related data types - we distinguish "mtypes" (machine representations) and "scitypes" (scientific types, what visions calls semantic type). More details here as reference.

Few questions for visions devs:

• time series are known to be a notoriously splintered field in terms of data representation, and even more when it comes to learning tasks (as in your ML example). Do you see visions moving in the direction of typing for ML?
• would you have time to look into the sktime datatypes module and assess how similar this is to visions? If similar, we might be tempted to take a dependency on visions and contribute. Key features are mtype conversions, scitype inference, checks that also return metadata (e.g., number of time stamps in a series, which can be represented 4 different ways)

src/visions/types/url.py does not correctly validate URLs.

First, the example code (lines 14--19) from the docs do not return True:

Python 3.9.4 (default, Apr  9 2021, 09:32:38)
[Clang 10.0.0 ] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import visions
>>> from urllib.parse import urlparse
>>> urls = ['http://www.cwi.nl:80/%7Eguido/Python.html', 'https://github.com/pandas-profiling/pandas-profiling']
>>> x = [urlparse(url) for url in urls]
>>> x in visions.URL
False
>>> x
[ParseResult(scheme='http', netloc='www.cwi.nl:80', path='/%7Eguido/Python.html', params='', query='', fragment=''), ParseResult(scheme='https', netloc='github.com', path='/pandas-profiling/pandas-profiling', params='', query='', fragment='')]
>>> import pkg_resources
>>> pkg_resources.get_distribution("visions").version
'0.7.4'

Second, non URLs are passing:

>>> urlparse('junk') in visions.URL
True
>>>

The code should probably check something like the following for each element of x:

    try:
        result = urlparse(x)
        return all([result.scheme, result.netloc])
    except:
        return False

Finally, and this is a suggested enhancement, I think the behavior would be more useful if it handled raw strings and did the parsing internally without the caller having to supply a parser:

urls = ['http://www.cwi.nl:80/%7Eguido/Python.html', 'https://github.com/pandas-profiling/pandas-profiling']
>>> urls in visions.URL
True

Follow the example of "Problem type inference".

From one dataframe, I already make a list of type for each column. Here is the type_list:

[Discrete,
 Nominal,
 Discrete,
 Nominal,
 Nominal,
 Nominal,
 Nominal,
 Nominal,
 Nominal,
 Binary,
 Discrete,
 Discrete,
 Discrete,
 Nominal,
 Binary]

type(type_list[0]) give visions.types.type.VisionsBaseTypeMeta

Now, I want to check if each type either have parent type of Categorical or Numeric.

for column, t in zip(column, type_list):
     if is_type_parent_of_categorical(t): 
            category_job(dataframe[column]) 

# binary is child if Categorical
is_type_parent_of_categorical(type_list[14]) -> True 

# Discrete is child of Numeric 
is_type_parent_of_categorical(type_list[0]) -> False 

How should I implement is_type_parent_of_categorical ?

My workaround seem to work because string comparision:

def is_type_parent_of_categorical(visions_type):
        type_str = str(visions_type)
            if type_str in ["Categorical", "Ordinal", "Nominal", "Binary"]:
                return True
            return False

Sometimes going through a whole array is not needed. You have the types of the subsets of the array and you just want to get a compatible data type for all subsets.

A common scenario when assembling horrible csvs is that the same column might be inferred as different types in different csvs. For example, (float <-- int). Worst case is to 'fall back' to string.