• Adds support for numpy types specified in dataclasses, converting them to and from standard python types as needed.
• ~Note that this commit does NOT handle the case where a numpy type is misrepresented as a standard python type. In most cases, these values will fail to serialize, or may be serialized incorrectly.~

The second commit now supports serialization of untyped or incorrectly typed numpy values (i.e., where the type of a variable is a regular Python type, but the contents are a numpy scalar or array) for json and toml only. Deserialization is unaffected in these cases--i.e., the value will be deserialized to the declared (non-numpy) type. (Deserialization to declared numpy types is handled in the first commit).

It should be possible to add support for yaml and toml, but the mechanism for doing so is different and not as obvious.

Closes #145

This pull request adds support for more types and also utilizes the code generation capabilities for them.

The following types where added:

• uuid.UUID
• PosixPath, WindowsPath, PurePath, PurePosixPath, PureWindowsPath from pathlib
• IPv4Address, IPv6Address, IPv4Network, IPv6Network, IPv4Interface, IPv6Interface from ipaddress
• date & datetime from datetime

I needed these types because I wanted to use pyserde in combination with asyncpg. asyncpg returns dict like objects which can be converted to dataclasses by pyserde's from_dict function.

Since asyncpg also supports the above mentioned types, the dicts already contain instances of them. That is the reason why I also added a new argument reuse_instances for from_dict & from_tuple. When this is set to True these functions check if the field already contains an instances of the target type and reuse it when possible.

This is faster for Path & IPAddress then calling the constructor again. For UUID it is also the only way to handle existing instances because uuid.UUID() does not accept them as an argument. It is also possible to change the default value of reuse_instances via the serialize & deserialize decorators.

To not cause slowdowns when serializing or deserializing to json/msgpack/toml/yaml reuse_instances is always set to False there, because we will never see existing instances there.

That is the part where this pull request drifted into refactoring. Since I had to add the reuse_instances argument at all these places, I also used the opportunity to remove some unused arguments (named & strict) and renamed asdict to to_dict and astuple to to_tuple. These are breaking changes, but I felt it was worth the gained similarity between serialization and deserialization code. In my option the public exposed functions are now also named more uniformly.

For msgpack external types I also changed the behaviour slightly. It is not required anymore that the dataclasses have a special meaning _type attribute. Instead to_msgpack & from_msgpack search the ext_dict for the correct type or type code and also throw exceptions if they can not find them.

I am sorry that this became so intertwined, I understand if you don't want to merge the breaking changes. I tried to make the commits cherry-pickable so maybe you only want to use some commits.

During development, I also noticed that Unions do not work properly for these types. I will make a separate pull request to fix that.

Finally, I have a question: What is the purpose of setting SE_NAME and why is it used in is_serializable? Could it be removed? is_serializable could use TO_ITER and TO_DICT for the check like is_deserializable.

Hi!

I've tried to rewrite your Travis CI workflow with Github Actions but your feedback needed.

Current status:

• there are two workflows:
  • test.yml workflow
    • fires on every push to non-master branch and every pull request
    • runs tests on multiple python versions in parallel
    • runs benchmark if all tests are passed
    • checks coverage if all tests are passed
  • build.yml
    • fires only if master branch was updated
    • do everything test.yml does + build docs and updates gh-pages branch (only if all tests are passed also)
• tests run smoothly on all defined Python versions (3.6 - 3.9, pypy3)
• coverage check and benchmark run are decoupled into two separate steps for better maintainability
• gh-pages update is now slightly more elegant thanks to "GitHub Pages Deploy Action" (https://github.com/JamesIves/github-pages-deploy-action)

Drawbacks:

• workflows are not optimised yet in terms of speed (no cache etc.)
• same codeblocks are used in test.yml and build.yml. This is avoidable, though. If you will never push directly to master branch you can safely remove 'test', 'check_coverage' and 'run_benchmark' steps from build.yml
• gh-pages commit message is always "docs: Update docs". I beleive this is also fixable
• .travis.yml should be removed because it will push changes in gh-pages branch instead

Questions:

1. Is decoupling coverage check from brnchmark OK for you?
2. make coverage also run tests. This is usable during local developement, but slow things up in CI. Maybe you can add a separate Makefile command just for checking coverage in CI?
3. Do you absolutely need gh-pages commit messages to be verbose (with a reference to commit or pull request)?

Please review the PR and let me know if anything else needed.

Hi @yukinarit, I added basic support for from __future__ import annotations.

Do you have any ideas how we could tell pytest to skip test_lazy_type_evaluation.py when using python 3.6 since the annotations future does not exist there?

Closes #112

Trying to figure out how to resolve this problem:

>>> from serde.msgpack import from_msgpack, to_msgpack
>>> from simpleRaft.messages.append_entries import AppendEntriesMessage, LogEntry
>>> a = LogEntry(term=0)

# Can serialize LogEntry below

>>> to_msgpack(a)
b'\x85\xa4term\x00\xa5index\x00\xa7command\x00\xa3key\xc0\xa5value\xc0'

# Just not when it's nested in another serializable object

>>> b = AppendEntriesMessage(sender=1, receiver=2, term=3, entries=[a])
>>> to_msgpack(b)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/arun/.local/lib/python3.8/site-packages/pyserde-0.2.1-py3.8.egg/serde/msgpack.py", line 34, in to_msgpack
    return se.serialize(obj, **opts)
  File "/home/arun/.local/lib/python3.8/site-packages/pyserde-0.2.1-py3.8.egg/serde/msgpack.py", line 23, in serialize
    return msgpack.packb(maybe_ext, use_bin_type=use_bin_type, **opts)
  File "/home/arun/.local/lib/python3.8/site-packages/msgpack/__init__.py", line 35, in packb
    return Packer(**kwargs).pack(o)
  File "msgpack/_packer.pyx", line 286, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 292, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 289, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 225, in msgpack._cmsgpack.Packer._pack
  File "msgpack/_packer.pyx", line 258, in msgpack._cmsgpack.Packer._pack
  File "msgpack/_packer.pyx", line 283, in msgpack._cmsgpack.Packer._pack
TypeError: can not serialize 'LogEntry' object

Hi @jennydaman!

It's tricky but possible to support Generics. Here is POC implementation.

Since I am not a heavy user of typing.Generics, I would like to ask you to confirm few things. It would be great if you could know.

1. What is the best way to check the type inheriting typing.Generics.

I use this code in the POC.

def is_generic(typ) -> bool:
    return typing.get_origin(typ) is not None

2. When a generic type is not specified like MyClass instead of MyClass[Nested], what do you expect the type of thing? Is it any? 🤔

This tries to add support for Unions with complex types like UUID or IPv4Address.

The basic idea is to use the code generation to add function for each Union the dataclass uses. This union function then tries to serialize or deserialize the field according to the order of types defined in the Union. The first one that works is used.

Using the code generation for this has the advantage that the user can influence the order of checks and that only types are checked that are defined in the Union. Which should be faster, then always checking against all types that pyserde supports.

Currently, not all tests work. I have not found a good solution for nested Unions or container types within Unions, e.g. Union[List[int],List[str]]. We probably need some form of recursion.

Some notable other changes:

• Implemented is_set in compat.py
• The rendered dict & iter functions now define a variable serde_scope which I needed to pass it to the union functions.
• The render functions now render NoneType as None. I think this broke the test_union_optional test, but I still have to look into it.

@yukinarit If you have time, I would appreciate a review and your opinion about the general idea? Also I noticed you increased the version to 0.2.3 so only at patch level. Do you use semantic versioning? Then it should be 0.3.0 in my opinion because the new type support is a feature :smile:.

Closes #60

The API as it stands now needs the caller to explicitly specify the type to be decoded. Sometimes its desirable to infer that from the serialized bytes.

from simpleRaft.messages.append_entries import AppendEntriesMessage
from simpleRaft.messages.request_vote import RequestVoteMessage
from simpleRaft.messages.base import BaseMessage
from serde.msgpack import from_msgpack, to_msgpack

ext_dict = {
  BaseMessage.MessageType.AppendEntries: AppendEntriesMessage,
  BaseMessage.MessageType.RequestVote: RequestVoteMessage,
}

a = AppendEntriesMessage("alice", "Bob", 123, 0, 0)
x = to_msgpack(a)
print(x)
b = from_msgpack(BaseMessage, x)
print(b, type(b))  # prints BaseMessage

a = RequestVoteMessage("alice", "Bob", 123, 0, 0)
x = to_msgpack(a, ext_dict=ext_dict)
print(x)
b = from_msgpack(BaseMessage, x, ext_dict=ext_dict)
print(b, type(b))  # prints RequestVoteMessage
a == b  # prints True

This pull request builds on #209, adding support for orjson and modifying numpy support to pass values through to orjson when that module is chosen as the (de)serializer and the appropriate flags are passed.

Still needs tests and probably discussion around how much of orjson to pull in.

Hi,

This is very cool!

Unfortunately, when I use mypy, I get Unexpected keyword argument when I try to return a @serdeed class from a function. I'm not sure if serde would need to have a mypy plugin or if I'm making a mistake.

I can make a minimal test case if needed.

Thank you!

The code generation doesn't put Optional or Union in scope when calling a custom class deserializer. This results in a NameError during deserialization.

Example

from dataclasses import dataclass
from typing import Optional

from serde import deserialize, from_dict


def deserialize_maybe_str(_typ, value):
    return value


@deserialize(deserializer=deserialize_maybe_str)
@dataclass
class CustomDe:
    maybe_str: Optional[str]


if __name__ == "__main__":
    data = {"maybe_str": "bye"}
    print(f"from_dict({data}) = {from_dict(CustomDe, data)}")

Output

Traceback (most recent call last):
  File "C:\Users\mauve\dev\modlunky2\src\union_demo.py", line 19, in <module>
    print(f"from_dict({data}) = {from_dict(CustomDe, data)}")
  File "C:\Users\mauve\dev\modlunky2\.venv\lib\site-packages\serde\de.py", line 319, in from_dict
    return from_obj(cls, o, named=True, reuse_instances=reuse_instances)
  File "C:\Users\mauve\dev\modlunky2\.venv\lib\site-packages\serde\de.py", line 253, in from_obj
    return serde_scope.funcs[FROM_DICT](o, reuse_instances=reuse_instances)
  File "<string>", line 10, in from_dict
NameError: name 'Optional' is not defined

Example:

from serde.se import serialize, to_dict

@serialize
class Foo:
    value: ClassVar[str] = "abc"

print(to_dict(Foo()))  # Prints {}

A rather complex bug which I struggle to describe. Here's an example:

from typing import Union, Optional, NewType
from serde import deserialize, Untagged, from_dict
from dataclasses import dataclass

OtherNewtypeThing = NewType('OtherNewtypeThing', str)


@deserialize
class SupposedlyUnrelated:
    name: OtherNewtypeThing  # works if you change type to str

@deserialize
class Innerclass:
    inner_value: str = 'SHOULD_NOT_APPEAR'


@deserialize(tagging=Untagged)
class MyDataclass:
    unrelateds: list[SupposedlyUnrelated]
    buggy_field: list[Union[str, Innerclass]]


data = {'unrelateds': [], 'buggy_field':[{'inner_value': 'value'}, 'something']}
actual = from_dict(MyDataclass, data)

print(actual)

Expected

MyDataclass(unrelateds=[], buggy_field=[Innerclass(inner_value='value'), 'something'])

Actual

MyDataclass(unrelateds=[], buggy_field=[Innerclass(inner_value='value'), Innerclass(optional_inner='SHOULD_NOT_APPEAR')])

Instead of deserializing "something" to str which matches Union[str, Innerclass], instead the value is mistakenly interpreted as Innerclass. This happens despite that "something" is not deserializable to Innerclass

Affected Version

pyserde 0.9.6 is affected.

This is a regression from pyserde 0.8.3

Workaround

Downgrade to pyserde 0.8.3

It would be convenient if pyserde could do like

@serde(prefix="foo", suffix="key")
class Foo:
    a: str

Then Foo can be deserialized from {"foo_a_key": "..."}.

If we set rename_all="camelcase", it will be {"fooAKey": "..."}.

Hey @yukinarit,

I'm running into an issue with a custom class level serializer. I have a dataclass with a field that is a dictionary and I want all datetimes in the dict to be serialized using .timestamp(). This seems to work for top level values, but not for anything nested ? I'm not sure exactly what the expected behavior is here? Is there a different way to accomplish this or is it a bug?

Example

from dataclasses import dataclass
from datetime import datetime
from typing import Any

from serde import SerdeSkip, serde
from serde.json import to_json


def serializer(cls, o):
    if cls is datetime or isinstance(o, datetime):
        return o.timestamp()
    else:
        raise SerdeSkip()


@serde(serializer=serializer)
@dataclass
class Foo:
    bar: dict[str, Any]


def main():
    d = datetime(2021, 1, 1, 0, 0, 0)
    foo = Foo(bar={"d": d})
    print(to_json(foo))  # prints {"bar":{"d":1609477200.0}}

    # The nested datetime here isn't serialized as I would expect it to be?
    d = datetime(2021, 1, 1, 0, 0, 0)
    foo = Foo(bar={"a": {"d": d}})
    print(to_json(foo))  # prints {"bar":{"a":{"d":"2021-01-01T00:00:00"}}}



if __name__ == '__main__':
    main()

toml and yaml support native datetime types (see e.g., https://toml.io/en/v1.0.0#offset-date-time). Currently pyserde converts all datetime types to string, which prevents tomli and pyyaml from handling these types directly.

I think that skip_if_default, skip_if_false, and skip_if don't work if there is a custom serializer?

Example:

from serde import serde, field
from dataclasses import dataclass
from serde.json import to_json

@serde
@dataclass
class Foo:
    bar: str = field(
        default="bar",
        serializer=lambda x: "...",
        skip_if_default=True,
    )

print(to_json(Foo()))  # prints '{"bar":"..."}'