This will become the version 2.0.0 release.

• [x] Update embedded simdjson to 0.3.0 (#31)
• [x] Update embedded simdjson to 0.4.0 (#31)
• [x] Move from cython to pybind11
• [ ] Rewrite documentation
• [ ] Better CI-generated benchmarks against json, ujson, rapidjson, and orjson.
• [x] Try to match the json.load, json.loads, json.dump and json.dumps interfaces. Will impact performance over the native simdjson API but users want plug-and-play.
• [x] Move from appveyor and circleci to github actions for CI tasks.
• [x] simdjson no longer requires C++17. We can greatly expand the versions of Python on Windows we can provide binary wheels for. This comes from older versions of CPython requiring C extensions to be built with the same compiler they were.

From simdjson/scripts/javascript, I generated a file called large.json. In C++, parsing this file takes about 0.25 s.

$parse large.json
Min:  0.252188 bytes read: 203130424 Gigabytes/second: 0.805471

I wrote the following Python script...

from timeit import default_timer as timer
with open('large.json', 'rb') as fin:
   x = fin.read()

for i in range(10):
   start = timer()
   doc = simdjson.loads(x)
   end = timer()
   print(end - start)

I get...

$ time python3 test.py
3.471898762974888
3.9210079659242183
3.3614078611135483
3.72252986789681
3.7506914171390235
3.756883286871016
3.752689895918593
3.751842977013439
3.7484844669234008
(...)

If my analysis is correct (and it could be wrong), pysimdjson takes 3.7 s to parse the file, and of that, 0.25 s are due to simdjson, leaving about 95% of the processing time to overhead.

I know that this is known, but I wanted to provide a data point.

[email protected]:~$ time python convert-to-pickle.py Traceback (most recent call last): File "convert-to-pickle.py", line 10, in data = simdjson.loads(ch.read()) File "/usr/local/lib/python3.8/dist-packages/simdjson/init.py", line 52, in loads return parser.parse(s, True) File "simdjson/csimdjson.pyx", line 468, in csimdjson.Parser.parse ValueError: This parser can't support a document that big

I am copying over issue https://github.com/simdjson/simdjson/issues/921 from simdjson. We do not see a crash in simdjson itself, but there is a crash in pysimdjson:

import simdjson
a = open("test.txt").read()
b = simdjson.loads(a.encode())

Using the file https://github.com/simdjson/simdjson/files/4749603/test.txt

Hello all!

When I try to serialize simdjson Object into Pickle, I get the following error:

TypeError: self.c_element,self.c_parser cannot be converted to a Python object for pickling

Would it be possible to add support for serializing/pickling simdjson instances directly, without converting them to dict? If not pickling, then at least an ability to serialize into .json would be lovely as well.

We are parsing a very high number of ~2KB JSON files in our Python-based application.

• The native (C++) SIMDJSON library delivers ~700k parser cycles per second.
• The pysimdjson delivers ~350k parser cycles per second.
• The Cython-based PoC implementation (in-house, so far) delivers ~700k parser cycles per second (very close to C++ implementation).

I also conducted a rather artificial test of "how many parser cycles" can I get with basically empty JSON ({}). The issue here is quite visible, the overhead of the Python<->pysymdjson boundary crossing is high relatively to other possible implementations.

A "parser cycle" is defined as a one call to parser.parse(json) on the existing parser instance.

I'm not 100% sure if this is a priority of this library, so feel free to close this one as irrelevant.

Here is a Python session that segfaults:

>>> import simdjson
>>> pa=simdjson.Parser().parse('{"a": 9999}')
>>> pa["a"]
zsh: segmentation fault (core dumped)  python

And here is one that works:

>>> import simdjson
>>> p = simdjson.Parser()
>>> pa = p.parse('{"a": 9999}')
>>> pa["a"]
9999

It's unclear to me why the first one segfaults, and it looks like a bug?

I imagine the parser is garbage collected by Python in the first example, but it's still clearly in use by the "pa" variable?

Version 0.4 of simdjson is now available

Highlights

• Test coverage has been greatly improved and we have resolved many static-analysis warnings on different systems.

New features:

• We added a fast (8GB/s) minifier that works directly on JSON strings.
• We added fast (10GB/s) UTF-8 validator that works directly on strings (any strings, including non-JSON).
• The array and object elements have a constant-time size() method.

Performance:

• Performance improvements to the API (type(), get<>()).
• The parse_many function (ndjson) has been entirely reworked. It now uses a single secondary thread instead of several new threads.
• We have introduced a faster UTF-8 validation algorithm (lookup3) for all kernels (ARM, x64 SSE, x64 AVX).

System support:

• C++11 support for older compilers and systems.
• FreeBSD support (and tests).
• We support the clang front-end compiler (clangcl) under Visual Studio.
• It is now possible to target ARM platforms under Visual Studio.
• The simdjson library will never abort or print to standard output/error.

Version 0.3 of simdjson is now available

Highlights

• Multi-Document Parsing: Read a bundle of JSON documents (ndjson) 2-4x faster than doing it individually. API docs / Design Details
• Simplified API: The API has been completely revamped for ease of use, including a new JSON navigation API and fluent support for error code and exception styles of error handling with a single API. Docs
• Exact Float Parsing: Now simdjson parses floats flawlessly without any performance loss (https://github.com/simdjson/simdjson/pull/558). Blog Post
• Even Faster: The fastest parser got faster! With a shiny new UTF-8 validator and meticulously refactored SIMD core, simdjson 0.3 is 15% faster than before, running at 2.5 GB/s (where 0.2 ran at 2.2 GB/s).

Minor Highlights

• Fallback implementation: simdjson now has a non-SIMD fallback implementation, and can run even on very old 64-bit machines.
• Automatic allocation: as part of API simplification, the parser no longer has to be preallocated-it will adjust automatically when it encounters larger files.
• Runtime selection API: We've exposed simdjson's runtime CPU detection and implementation selection as an API, so you can tell what implementation we detected and test with other implementations.
• Error handling your way: Whether you use exceptions or check error codes, simdjson lets you handle errors in your style. APIs that can fail return simdjson_result, letting you check the error code before using the result. But if you are more comfortable with exceptions, skip the error code and cast straight to T, and exceptions will be thrown automatically if an error happens. Use the same API either way!
• Error chaining: We also worked to keep non-exception error-handling short and sweet. Instead of having to check the error code after every single operation, now you can chain JSON navigation calls like looking up an object field or array element, or casting to a string, so that you only have to check the error code once at the very end.

Hi! Thanks again for this fantastic project. ^_^

I ran into some CI errors where my Windows 64-bit builds were dying due to compile errors with CPython 3.6. I noticed there's no wheel on PyPI for it.

Would it be possible to fix?

Thanks!

  Using cached https://files.pythonhosted.org/packages/9b/f6/c63260f8788574de8fdd0bbe70f803328cb058141c0903ba29637d89f863/pysimdjson-2.5.0.tar.gz
Installing collected packages: pysimdjson
  Running setup.py install for pysimdjson ... error
    Complete output from command /home/ubuntu/ctix-2/venv/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-wzmco2i3/pysimdjson/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-jelehspx/install-record.txt --single-version-externally-managed --compile --install-headers /home/ubuntu/ctix-2/venv/include/site/python3.6/pysimdjson:
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.6
    creating build/lib.linux-x86_64-3.6/simdjson
    copying simdjson/__init__.py -> build/lib.linux-x86_64-3.6/simdjson
    running build_ext
    building 'csimdjson' extension
    creating build/temp.linux-x86_64-3.6
    creating build/temp.linux-x86_64-3.6/simdjson
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/ubuntu/ctix-2/venv/lib/python3.6/site-packages/pybind11/include -I/home/ubuntu/ctix-2/venv/include -I/usr/include/python3.6m -c simdjson/binding.cpp -o build/temp.linux-x86_64-3.6/simdjson/binding.o -std=c++11
    In file included from /home/ubuntu/ctix-2/venv/lib/python3.6/site-packages/pybind11/include/pybind11/pytypes.h:12:0,
                     from /home/ubuntu/ctix-2/venv/lib/python3.6/site-packages/pybind11/include/pybind11/cast.h:13,
                     from /home/ubuntu/ctix-2/venv/lib/python3.6/site-packages/pybind11/include/pybind11/attr.h:13,
                     from /home/ubuntu/ctix-2/venv/lib/python3.6/site-packages/pybind11/include/pybind11/pybind11.h:44,
                     from simdjson/binding.cpp:5:
    /home/ubuntu/ctix-2/venv/lib/python3.6/site-packages/pybind11/include/pybind11/detail/common.h:112:20: fatal error: Python.h: No such file or directory
    compilation terminated.
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

    ----------------------------------------
Command "/home/ubuntu/ctix-2/venv/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-wzmco2i3/pysimdjson/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-jelehspx/install-record.txt --single-version-externally-managed --compile --install-headers /home/ubuntu/ctix-2/venv/include/site/python3.6/pysimdjson" failed with error code 1 in /tmp/pip-install-wzmco2i3/pysimdjson/
You are using pip version 19.0, however version 20.2.4 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.```

Support for clang-cl is coming. As part of the PR that allows CPython to build against clang-cl, distutils is updated to build with clang-cl (https://github.com/python/cpython/pull/18371). Once this PR is merged and a new CPython release includes it we can start using it for our binary releases.

Clang has reached a point where it's safe enough for us to use with CPython's built with MSVC2015 or newer. https://clang.llvm.org/docs/MSVCCompatibility.html

This would alleviate poor windows performance caused by MSVC issues (https://github.com/simdjson/simdjson/issues/847, but not entirely, https://github.com/simdjson/simdjson/issues/848).

We only need to do this if upstream simdjson doesn't figure out what's up with MSVC. @lemire

simdjson minify drops the trailing '.0' from floats, which is fine by JSON spec, but matters in practice. For example, Elasticsearch dynamic field type detection is affected. In general, Python distinguishes between int and float, so various type guarantees may fail. The dump/load cycle should not convert types for a few byte gain. Let users explicitly convert types, if they need to.

This modifies minify, so it does not drop the '.0'.

Note: simdjson started dropping '.0' with d0821adf0e7934f27a8eb5c2fe9b8254e4.

I'm getting increased latency in my application from simdjson but I can't figure out why.

This is a snippet from profiling the function that gets items from the simdjson object. The time is in seconds.

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
       26    0.017    0.001    0.017    0.001 {method 'get' of 'csimdjson.Object' objects}

When I time getting individual items from the same object I get timings of about 15 microseconds which seems comparable with getting items from a normal python dictionary. However when I test the whole function the performance is much worse.

We've added a check in v4 (https://github.com/TkTech/pysimdjson/blob/master/simdjson/csimdjson.pyx#L437) that prevents parsing new documents while references continue to exist to the old one. This is correct, in that it ensures no errors. I wasn't terribly happy with this, but it's better then segfaulting.

It has downsides:

• It sucks as a user (https://github.com/TkTech/pysimdjson/issues/53#issuecomment-850494991), where you might have to del the old objects, even if you didn't intend to use them again. Very un-pythonic.
• Doesn't work on PyPy, where del is unreliable. The objects may not be garbage collected until much later.

Brainstorming welcome. Alternatives:

• Probably the easiest approach would be for a Parser to keep a list of Object and Array proxies that hold a reference to it, and set a dirty bit on them when parse() is called with a different document. The performance of this would probably be unacceptable - I might be wrong.
• Use the new parse_into_document() and create a new document for every parse. This is potentially both slow and very wasteful with memory, but would let us keep a document around and valid for as long as Object or Array reference it.

Bundling a library is a serious sin in our book, so provide the ability to link to the system library. I've also done some refactoring to avoid exponential growth of Extension calls. The default behavior remains the same, so it shouldn't affect existing users.

That said, the patch isn't perfect. It still uses the bundled headers instead of system headers but it should be good enough for us.

The pysimdjson library could support our document_stream interface (parse_many function). It is well tested as of release 0.7 (with fuzz testing) and works well today. It supports streams of indefinite size.

See https://github.com/simdjson/simdjson/blob/master/doc/parse_many.md

Related to https://github.com/TkTech/pysimdjson/issues/70