Next step to hopefully get this on pypi (and letting me declare it as a dependency in my projects).

This adds two new actions to deploy to the test pypi server and to deploy to the live pypi server. There is a third new step that also builds source distributions, so any platform with a compiler can pip install this even if the wheels don't directly support it (which probably isn't too many people).

I haven't used the gh-action-pypi-publish action yet, so this may not work (I have custom publishing logic that also signs my wheels with a CI gpg key), but it should be too hard to debug.

@erezsh To support this it would be helpful if you could generate a token for test.pypi.org (you may need to create an account if you don't have one, it's distinct from pypi.org) and then populate the TEST_PYPI_API_TOKEN secret with that token. When that works, you can do the same for regular pypi.

The logic is setup such authorized PRs will try to publish to test.pypi. When you want to make a release, you would generate a new git tag (which can be done via the github release process), and the live pypi action will run when it sees a new tag.

I found out that only the source is published to pypi and not the wheels. It shouldn't be too difficult to get cibuildwheel working on this repo and then publishing those to pypi. It will also close #2

This is my parser:

    lark_kwargs = {
        "parser": "lalr",
        "lexer": "contextual",
        "propagate_positions": True,
        "_plugins": lark_cython.plugins,
    }
    lang_parser = Lark(lang_def, start="start", **lark_kwargs)

I'd like to propagate the line positions, without cython-lark it works fine, but once i enable the plugins the meta value is always empty.

Do you have any suggestion on what's wrong ?

I don't know if this is the correct solution but it fixes the case described below

👇🏽

with a grammar like:

display_decimals: "DECIMALS"i WS INT
start: display_decimals
%import common (INT, WS)

and a script like

script = """
      bla
"""

It raises

*** AttributeError: 'lark_cython.lark_cython.LexerThread' object has no attribute '_Token'

This PR handle the right exception when the token is wrong rather than raising an Attribute error.

Suggestion

I mentioned the differences from Lark, is there any way to resolve? Describe alternatives you've considered Deploy from lark without any changes

Token values in lark_cython are typed as str.

In my TransformerI'm changing the token values to their correct types, for example int:

    def int(self, t):
        v = t[0]
        v.value = int(v.value)
        # return full Token rather than an int as Token properties are required later
        return v

This throws the following error in lark_cython:

lark_cython\lark_cython.pyx:20: in lark_cython.lark_cython.Token.value.__set__  TypeError: Expected unicode, got int

I return the full token in the transformer, rather than simply an int value, as later logic(for error handling etc.) takes advantage of the token properties:

line, column = key_token.line, key_token.column

Do token values have to be strings in lark_cython? If so, is there any approach which would allow token values to be converted to support int, float etc.?

Describe the bug

install lark_cython on windows

  lark_cython.obj : error LNK2001: unresolved external symbol __imp__PyClassMethod_New
  lark_cython.obj : error LNK2001: unresolved external symbol __imp__PyList_Type
  lark_cython.obj : error LNK2001: unresolved external symbol __imp__PyDict_GetItemString
...

To Reproduce

pip install lark-cython

What do i need to install if i want to install on Windows

Bumps actions/setup-python from 4.3.0 to 4.3.1.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Bumps pypa/cibuildwheel from 2.11.2 to 2.11.3.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Bumps pypa/cibuildwheel from 2.11.3 to 2.11.4.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Bumps actions/setup-python from 4.3.1 to 4.4.0.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

@erezsh I appreciate you pointing me to this repo earlier.

I was able to demonstrate about a 2x speedup for my specific DSL, but I was curious what that speedup was more generally, and I as also looking for a nice way to quantify if any chances to the Cython code would provide further speedups.

I took a peek at the Cython code and I think it is ripe for optimization. Python types are being used everywhere, and if we were able to refactor those with pure C types, then I think there could be a large speed gain. Unfortunately, I'm not the greatest Cython coder, so I wasn't able to make any simple changes that did anything.

However, I did write a reproducible benchmark to quantify the speedup between CPython-lark, and Cython-Lark over a range of input sizes.

The timing module I use is timerit (which I wrote). It works similarly to timeit, but can work inline in existing code. I also use pandas, matplotlib, and seaborn to generate a nice figure showing the speedup over different input sizes.

The stdout of the script is:

Timed best=1.144 ms, mean=1.704 ± 0.4 ms for method=parse_python,size=16
Timed best=5.974 ms, mean=9.279 ± 1.5 ms for method=parse_python,size=87
Timed best=10.813 ms, mean=17.676 ± 2.1 ms for method=parse_python,size=158
Timed best=16.715 ms, mean=25.435 ± 3.1 ms for method=parse_python,size=229
Timed best=20.364 ms, mean=31.777 ± 4.4 ms for method=parse_python,size=299
Timed best=19.721 ms, mean=37.681 ± 8.5 ms for method=parse_python,size=370
Timed best=23.847 ms, mean=47.875 ± 7.2 ms for method=parse_python,size=441
Timed best=29.769 ms, mean=47.659 ± 11.1 ms for method=parse_python,size=512
Timed best=0.751 ms, mean=1.226 ± 0.2 ms for method=parse_cython,size=16
Timed best=4.176 ms, mean=6.391 ± 0.7 ms for method=parse_cython,size=87
Timed best=7.068 ms, mean=11.063 ± 1.5 ms for method=parse_cython,size=158
Timed best=8.398 ms, mean=13.830 ± 3.0 ms for method=parse_cython,size=229
Timed best=11.037 ms, mean=19.378 ± 4.7 ms for method=parse_cython,size=299
Timed best=13.332 ms, mean=25.704 ± 4.5 ms for method=parse_cython,size=370
Timed best=14.691 ms, mean=25.771 ± 6.9 ms for method=parse_cython,size=441
Timed best=20.890 ms, mean=31.757 ± 6.4 ms for method=parse_cython,size=512

Statistics:
                                   min style_key size_key              hue_key        method  size      mean
key                                                                                                         
method=parse_cython,size=16   0.000751        {}       {}  method=parse_cython  parse_cython    16  0.001226
method=parse_python,size=16   0.001144        {}       {}  method=parse_python  parse_python    16  0.001704
method=parse_cython,size=87   0.004176        {}       {}  method=parse_cython  parse_cython    87  0.006391
method=parse_python,size=87   0.005974        {}       {}  method=parse_python  parse_python    87  0.009279
method=parse_cython,size=158  0.007068        {}       {}  method=parse_cython  parse_cython   158  0.011063
method=parse_cython,size=229  0.008398        {}       {}  method=parse_cython  parse_cython   229  0.013830
method=parse_python,size=158  0.010813        {}       {}  method=parse_python  parse_python   158  0.017676
method=parse_cython,size=299  0.011037        {}       {}  method=parse_cython  parse_cython   299  0.019378
method=parse_cython,size=370  0.013332        {}       {}  method=parse_cython  parse_cython   370  0.025704
method=parse_cython,size=441  0.014691        {}       {}  method=parse_cython  parse_cython   441  0.025771
method=parse_python,size=229  0.016715        {}       {}  method=parse_python  parse_python   229  0.025435
method=parse_python,size=370  0.019721        {}       {}  method=parse_python  parse_python   370  0.037681
method=parse_python,size=299  0.020364        {}       {}  method=parse_python  parse_python   299  0.031777
method=parse_cython,size=512  0.020890        {}       {}  method=parse_cython  parse_cython   512  0.031757
method=parse_python,size=441  0.023847        {}       {}  method=parse_python  parse_python   441  0.047875
method=parse_python,size=512  0.029769        {}       {}  method=parse_python  parse_python   512  0.047659
Speedup:
           min style_key size_key              hue_key        method      mean  speedup_mean  speedup_min
size                                                                                                     
16    0.000751        {}       {}  method=parse_cython  parse_cython  0.001226      1.390526     1.523632
87    0.004176        {}       {}  method=parse_cython  parse_cython  0.006391      1.451871     1.430450
158   0.007068        {}       {}  method=parse_cython  parse_cython  0.011063      1.597770     1.529840
229   0.008398        {}       {}  method=parse_cython  parse_cython  0.013830      1.839145     1.990309
299   0.011037        {}       {}  method=parse_cython  parse_cython  0.019378      1.639875     1.845177
370   0.013332        {}       {}  method=parse_cython  parse_cython  0.025704      1.465973     1.479169
441   0.014691        {}       {}  method=parse_cython  parse_cython  0.025771      1.857683     1.623239
512   0.020890        {}       {}  method=parse_cython  parse_cython  0.031757      1.500740     1.425010
Average speedup
                  mean       std       min       25%       50%       75%       max
speedup_mean  1.592948  0.176645  1.390526  1.462448  1.549255  1.689692  1.857683
speedup_min   1.605853  0.206135  1.425010  1.466989  1.526736  1.678723  1.990309

This is also benchmarked against the lark.lark grammar that ships with lark. I have a helper function to generate a "random" (not really random, but it is simple) lark file to pass to the parser. One question I had was: Is there a way to use lark to generate a string from a grammar? If so that would make bench-marking more complex grammars much easier.

Anyways, I hope this is helpful. Thanks again for this library!