Python linting made easy. Also a casual yet honorific way to address individuals who have entered an organization prior to you.

Overview

pysen

What is pysen?

pysen aims to provide a unified platform to configure and run day-to-day development tools. We envision the following scenarios in the future:

  • You open any project and pysen run lint, pysen run format will check and format the entire codebase
  • Standardized coding styles are setup with a few lines in a single pyproject.toml file

pysen centralizes the code and knowledge related to development tools that teams have accumulated, most notably for python linters. You can make tasks that can be executed from both setup.py and our command-line tool. We currently provide tasks that manage setting files for the following tools:

  • linters
    • flake8
    • isort
    • mypy
    • black
  • utilities
    • (planned) protoc

What isn't pysen?

  • pysen is not a linting tool per se. Rather, pysen run lint orchestrates multiple python linting tools by automatically setting up their configurations from a more abstract setting for pysen.
  • pysen does not manage your depedencies and packages. We recommend using package managers such as pipenv or poetry to lock your dependecy versions, including the versions for the linting tools that pysen coordinates (i.e., isort, mypy, flake8, black). The supported versions for these tools can be found in the extra_requires/lint section in pysen's setup.py. You should not rely on pip install pysen[lint] to control the versions of your linting tools.
  • pysen is not limited to linting purposes or python. See the plugin section for details.

Install

PyPI

pip install "pysen[lint]"

Other installation examples

# pipenv
pipenv install --dev "pysen[lint]==0.9.0"
# poetry
poetry add -D pysen==0.9.0 -E lint

Quickstart: Set up linters using pysen

Put the following pysen configuration to pyproject.toml of your python package:

[tool.pysen]
version = "0.9"

[tool.pysen.lint]
enable_black = true
enable_flake8 = true
enable_isort = true
enable_mypy = true
mypy_preset = "strict"
line_length = 88
py_version = "py37"
[[tool.pysen.lint.mypy_targets]]
  paths = ["."]

then, execute the following command:

$ pysen run lint
$ pysen run format  # corrects errors with compatible commands (black, isort)

That's it! pysen, or more accurately pysen tasks that support the specified linters, generate setting files for black, isort, mypy, and flake8 and run them with the appropriate configuration. For more details about the configuration items that you can write in pyproject.toml, please refer to pysen/pyproject_model.py.

You can also add custom setup commands to your Python package by adding the following lines to its setup.py:

import pysen
setup = pysen.setup_from_pyproject(__file__)
$ python setup.py lint

We also provide a Python interface for customizing our configuration and extending pysen. For more details, please refer to the following two examples:

  • Example configuration from Python: examples/advanced_example/config.py
  • Example plugin for pysen: examples/plugin_example/plugin.py

How it works: Settings file directory

Under the hood, whenever you run pysen, it generates the setting files as ephemeral temporary files to be used by linters. You may want to keep those setting files on your disk, e.g. when you want to use them for your editor. If that is the case, run the following command to generate the setting files to your directory of choice:

$ pysen generate [out_dir]

You can specify the settings directory that pysen uses when you pysen run. To do so add the following section to your pyproject.toml:

[tool.pysen-cli]
settings_dir = "path/to/generate/settings"

When you specify a directory that already contains some configurations, pysen merges the contents. The resulting behavior may differ from when you don't specify settings_dir.

Also keep in mind that this option is honored only when you use pysen through its CLI. When using pre-commit or setuptools you need to specify settings_dir as arguments.

Tips: IDE / Text editor integration

vim

You can add errors that pysen reports to your quickfix window by:

:cex system("pysen run_files lint --error-format gnu ".expand('%:p'))

Another way is to set pysen to makeprg:

set makeprg=pysen\ run_files\ --error-format\ gnu\ lint\ %

Then running :make will populate your quickfix window with errors. This also works with vim-dispatch as long as you invoke :Make instead of :Dispatch (for this reason)

The result will look like the following:

pysen-vim

Emacs

Refer to the Compilation mode. The following is an example hook for python.

(add-hook 'python-mode-hook
    (lambda ()
        (set (make-local-variable 'compile-command)
            (concat "pysen run_files lint --error-format gnu  " buffer-file-name))))

VSCode

Refer to the example task setting. Running the task will populate your "PROBLEMS" window like so:

pysen-vscode

Note that this may report duplicate errors if you have configured linters like flake8 directly through your VSCode python extension. We do not currently recommend watching for file changes to trigger the task in large projects since pysen will check for all files and may consume a considerable amount of time.

Configure pysen

We provide two methods to write configuration for pysen.

One is the [tool.pysen.lint] section in pyproject.toml. It is the most simple way to configure pysen, but the settings we provide are limited.

The other method is to write a python script that configures pysen directly. If you want to customize configuration files that pysen generates, command-line arguments that pysen takes, or whatever actions pysen performs, we recommend you use this method. For more examples, please refer to pysen/examples.

pyproject.toml configuration model

Please refer to pysen/pyproject_model.py for the latest model.

Here is an example of a basic configuration:

[tool.pysen]
version = "0.9"

[tool.pysen.lint]
enable_black = true
enable_flake8 = true
enable_isort = true
enable_mypy = true
mypy_preset = "strict"
line_length = 88
py_version = "py37"
isort_known_third_party = ["numpy"]
isort_known_first_party = ["pysen"]
mypy_ignore_packages = ["pysen.generated.*"]
mypy_path = ["stubs"]
[[tool.pysen.lint.mypy_targets]]
  paths = [".", "tests/"]

[tool.pysen.lint.source]
  includes = ["."]
  include_globs = ["**/*.template"]
  excludes = ["third_party/"]
  exclude_globs = ["**/*_grpc.py"]

[tool.pysen.lint.mypy_modules."pysen.scripts"]
  preset = "entry"

[tool.pysen.lint.mypy_modules."numpy"]
  ignore_errors = true

Create a plugin to customize pysen

We provide a plugin interface for customizing our tool support, setting files management, setup commands and so on. For more details, please refer to pysen/examples/plugin_example.

Development

pipenv is required for managing our development environment.

# setup your environment
$ pipenv sync
# activate the environment
$ pipenv shell
  • Update depedencies in Pipfile.lock
$ pipenv lock --pre
  • Run all tests
$ pipenv run tox

Contributing

This repository serves only as a mirror of our main repository on our private repository. Therefore we do not plan to accept any pull requests. We encourage aspiring developers to make patches on their forked repositories.

Also, our resource limitations force us to prioritize development to fulfill our corporate-specific demands. As such we will keep Issues closed for the foreseeable future. With a heavy heart we direct all questions, troubleshooting, feature requests and bug reports to /dev/null.

Comments
  • Unfriendly error in the absence of `tool.pysen.lint` section

    Unfriendly error in the absence of `tool.pysen.lint` section

    When pyproject.toml does not contain a tool.pysen.lint section, running pysen run lint results in the following error:

    pysen run lint
    usage: pysen run [-h] [--error-format {gnu}] [--no-parallel] {} [{} ...]
    pysen run: error: argument targets: invalid choice: 'lint' (choose from )
    

    In our case I believe the best configuration is no configuration at all, so I suggest enabling black, isort, flake and mypy without explicit configuration by default. By doing so we will not need to directly address the superficial issue of the unfriendly error.

    opened by sergeant-wizard 8
  • `tool.pysen.lint.source` may not work

    `tool.pysen.lint.source` may not work

    Problem

    I could not exclude a specific path when I run mypy. I think I can configure this by setting tool.pysen.lint.source properly, but it didn't work. I checked the generated pyproject.toml which linters use, and found that the information I specified in tool.pysen.lint.source didn't exist there.

    How to reproduce

    1. Create a pyproject.toml following the example on https://github.com/pfnet/pysen#configuration-model
    [tool.pysen]
    version = "0.10"
    
    [tool.pysen.lint]
    enable_black = true
    enable_flake8 = true
    enable_isort = true
    enable_mypy = true
    mypy_preset = "strict"
    line_length = 88
    py_version = "py37"
    isort_known_third_party = ["numpy"]
    isort_known_first_party = ["pysen"]
    mypy_ignore_packages = ["pysen.generated.*"]
    mypy_path = ["stubs"]
    [[tool.pysen.lint.mypy_targets]]
      paths = [".", "tests/"]
    
    [tool.pysen.lint.source]
      includes = ["."]
      include_globs = ["**/*.template"]
      excludes = ["third_party/"]
      exclude_globs = ["**/*_grpc.py"]
    
    [tool.pysen.lint.mypy_modules."pysen.scripts"]
      preset = "entry"
    
    [tool.pysen.lint.mypy_modules."numpy"]
      ignore_errors = true
    
    1. Run pysen generate hoge

    Then, hoge/pyproject.toml looks like

    [tool.black] # automatically generated by pysen
    line-length = 88
    target-version = ["py37"]
    
    [tool.isort] # automatically generated by pysen
    default_section = "THIRDPARTY"
    ensure_newline_before_comments = true
    force_grid_wrap = 0
    force_single_line = false
    include_trailing_comma = true
    known_first_party = ["pysen"]
    known_third_party = ["numpy"]
    line_length = 88
    multi_line_output = 3
    use_parentheses = true
    

    and hoge/setup.cfg looks like

    [flake8]
    # automatically generated by pysen
    # e203: black treats : as a binary operator
    # e231: black doesn't put a space after ,
    # e501: black may exceed the line-length to follow other style rules
    # w503 or w504: either one needs to be disabled to select w error codes
    ignore = E203,E231,E501,W503
    max-line-length = 88
    select = B,B950,C,E,F,W
    
    [mypy]
    # automatically generated by pysen
    check_untyped_defs = True
    disallow_any_decorated = False
    disallow_any_generics = False
    disallow_any_unimported = False
    disallow_incomplete_defs = True
    disallow_subclassing_any = True
    disallow_untyped_calls = True
    disallow_untyped_decorators = False
    disallow_untyped_defs = True
    ignore_errors = False
    ignore_missing_imports = True
    mypy_path = stubs
    no_implicit_optional = True
    python_version = 3.7
    show_error_codes = True
    strict_equality = True
    strict_optional = True
    warn_redundant_casts = True
    warn_return_any = True
    warn_unreachable = True
    warn_unused_configs = True
    warn_unused_ignores = False
    
    [mypy-numpy]
    # automatically generated by pysen
    ignore_errors = True
    
    [mypy-pysen.generated.*]
    # automatically generated by pysen
    follow_imports = skip
    ignore_errors = True
    
    [mypy-pysen.scripts]
    # automatically generated by pysen
    disallow_untyped_calls = False
    disallow_untyped_defs = False
    warn_return_any = False
    

    In the above example, [tool.pysen.lint.source] information is lost.

    Environment

    • macos 12.0.1
    • pysen version 0.10.1
    $ python
    Python 3.8.5 (default, Sep  8 2020, 17:55:39) 
    [Clang 11.0.3 (clang-1103.0.32.59)] on darwin
    Type "help", "copyright", "credits" or "license" for more information.
    
    opened by xuzijian629 5
  • Force lint extras to use older click

    Force lint extras to use older click

    https://github.com/psf/black/issues/2964 にあるように最新のclickでは古いblackが動作しない問題への対処です。 問題になるのは pysen[lint] を使ってinstallしたときで、blackは古いversionが入る一方でclickは最新版が入ります (black側でclickの上限を制限していないため)。

    pysen[lint] でinstallするときにclickのversionを制限することで不整合が起きないようにしました。 別の解決策としてblackの制限を緩めるというのもありますが、コードの変更や動作確認の手間が最小になるこちらの方法をとりました。 blackあるいはclickの最新版を使いたいユーザーは extras_require を使わないという方法があるので大きな問題にはならないという認識です。

    chore 
    opened by Hakuyume 3
  • Update lint extras for py39

    Update lint extras for py39

    Since PFN uses py39 widely, pysen[lint] should work with py39. Current lint extras tries to install older black and it does not work with py39.

    (optional) Can we drop py36 and py37 support?

    opened by Hakuyume 2
  • Use 'release' event to trigger release workflow

    Use 'release' event to trigger release workflow

    There is a limitation that a workflow cannot trigger a new workflow. https://docs.github.com/en/actions/using-workflows/triggering-a-workflow#triggering-a-workflow-from-a-workflow

    Currently, a release is created by release-drafter, and that's why the release workflow wasn't triggered for the 0.10.2 tag.

    As such, I configured the release workflow to use the workflow_dispatch to publish a new release. I also removed push.tags trigger as we can use the workflow_dispatch trigger for the manual release as well, but I don't have a strong opinion about that.

    chore 
    opened by bonprosoft 2
  • Issue of required black version

    Issue of required black version

    In setup.py, required version of black is,

    "black>=19.10b0,<=20.8",

    But it's written to install black==21.10b0 on README.md. And it cause error during executing black.

    opened by kazyam53 2
  • State only that linter versions for [lint] are

    State only that linter versions for [lint] are "confirmed" instead of "supported" in README

    The current wording is not accurate because the versions are those that are confirmed to work by the developers. Higher versions are not necessarily unsupported. I think the word "confirmed" is often used to imply such scenarios.

    documentation 
    opened by sergeant-wizard 0
  • Add pysen-test action

    Add pysen-test action

    I created pysen-test action. The action builds the Dockerfile on each CI job and runs tox on the built image.

    Initially I used the docker image hosted on quay.io. But actually it would be problematic for some PRs like #18 to update the docker image.

    chore 
    opened by bonprosoft 0
Releases(0.10.2)
  • 0.10.2(Apr 23, 2022)

    📜 Documentation

    • Add missing assets/src directory for rendering images (#6) @bonprosoft

    🧰 Maintenance

    • Bump version to 0.10.2 (#14) @bonprosoft
    • supports python 3.10 as a target version (#15) @hiro-o918
    • Force lint extras to use older click (#13) @Hakuyume
    • Sort files to make output more readable (#11) @grafi-tt
    • Update comment in settings (#10) @Hakuyume
    • Support tomlkit 0.10.0 (#9) @bonprosoft
    • Add workflow for releasing PyPI packages (#5) @bonprosoft
    • Add release drafter (#3) @sergeant-wizard
    • Modify pull_request trigger to the proper indent level (#4) @bonprosoft
    • Fix typo (#1) @sergeant-wizard
    • Add pysen-test CI (#2) @bonprosoft
    Source code(tar.gz)
    Source code(zip)
  • 0.10.1(Nov 25, 2021)

    We have released pysen 0.10.1!

    pysen 0.10.1 includes...

    • Latest linter version support
      • We have added support for the following versions since v0.9.1
        • mypy>=0.800
        • black>=21.4b
        • flake8>=4.0
      • Note that lint extra is still restricted to older versions. Please refer to the Install section in README.
    • mypy_preset renewal
      • entry: Recommended for newbies
      • strict: Recommended for projects with libraries which do not provide complete type annotations, such as opencv, pytorch, etc.
      • very_strict: For strict type checking. This preset may not be practical when used with libraries without complete type annotation. This corresponds to the strict preset before v0.9.1.
    • Option to use pysen.toml as configuration file name
    • mypy namespace packages support
    • Many bug fixes

    We have also added a “Frequently Asked Questions” section in README. Please look for solutions to your problem in this section.

    Source code(tar.gz)
    Source code(zip)
  • 0.9.1(Mar 29, 2021)

Owner
Preferred Networks, Inc.
Preferred Networks, Inc.
Spacecrypto-bombcrypto-bot - SpaceCrypto And Bombcrypto Bot - MultiScreen

SpaceCrypto And Bombcrypto Bot - MultiScreen This is a open source project inspi

Paulo Bramante 5 Nov 03, 2022
Cryptocurrency Prices Telegram Bot For Python

Cryptocurrency Prices Telegram Bot How to Run Set your telegram bot token as environment variable TELEGRAM_BOT_TOKEN: export TELEGRAM_BOT_TOKEN=your_

Sina Nazem 3 Oct 31, 2022
Morpy Bot Linux - Morpy Bot Linux With Python

Morpy_Bot_Linux Guide to using the robot : 🔸 Lsmod = to identify admins and st

2 Jan 20, 2022
This is a script to forward forward large number of documents to another telegram channel.

ChannelForward 😇 This is a Script to Forward Large Number of Documents to Another Telegram Channel. If You Try to Forward Very Large Number of Files

Anjana Madushanka 10 Jun 08, 2021
Weather_besac is a French twitter bot that tweet the weather of the city of Besançon in Franche-Comté in France every day at 8am and 4pm.

Weather Bot Besac Weather_besac is a French twitter bot that tweet the weather of the city of Besançon in Franche-Comté in France every day at 8am and

Rgld_ 1 Nov 15, 2021
If you are in allot of groups or channel and you would like to leave them at once use this

Telegram-auto-leave-groups If you are in allot of groups or channel and you would like to leave them at once use this USER GUIDE 👣 Insert your telegr

Julius Njoroge 4 Jan 03, 2023
Pure Python 3 MTProto API Telegram client library, for bots too!

Telethon ⭐️ Thanks everyone who has starred the project, it means a lot! Telethon is an asyncio Python 3 MTProto library to interact with Telegram's A

LonamiWebs 7.3k Jan 01, 2023
A bot to get Statistics like the Playercount from your Minecraft-Server on your Discord-Server

Hey Thanks for reading me. Warning: My English is not the best I have programmed this bot to show me statistics about the player numbers and ping of m

spaffel 12 Sep 24, 2022
simple discord token grabber with webhook hiding feature.

Token Grabber A simple Discord token grabber with base64 webhook encoding, it uses pastebin as a database to get webhook, so next time u dont get your

0 Dec 01, 2021
Repositório para meu Discord Bot pessoal

BassetinhoBot Escrevi o código usando o Python 3.8.3 e até agora não tive problemas rodando nas versões mais recentes. Repositório para o Discord Bot

Vinícius Bassete 1 Jan 04, 2022
Бот Telegram для Школы в Капотне (ЦО № 1858)

co1858 Telegram Bot Активно разрабатывался в 2015-2016 году как учебный проект, с целью научиться создавать ботов для Telegram. Бот автоматически парс

Ilya Pavlov 4 Aug 30, 2022
OpenSource bot for control groups ...

⭕️ کمک به افراد برای اداره هرچه فان تره گروه 📟 همه گروه های بزرگ نیاز به یه بات خفن دارن تا از گروه مراقبت کنه این بات کارش همینه سعی کرده فیچر خیلی

Mehran Alam Beigi 2 Nov 26, 2021
A Python SDK for Tinybird 🐦

Verdin Verdin is a tiny bird, and also a Python SDK for Tinybird . Install pip install verdin Usage Query a Pipe # the tinybird module exposes all im

LocalStack 13 Dec 14, 2022
A discord bot consuming Notion API to add, retrieve data to Notion databases.

Notion-DiscordBot A discord bot consuming Notion API to add and retrieve data from Notion databases. Instructions to use the bot: Pre-Requisites: a)In

Servatom 57 Dec 29, 2022
Python implementation for PetitPotam

PetitPotam Coerce NTLM authentication from Windows hosts Installtion $ pip3 install impacket Usage usage: petitpotam.py [-h] [-debug] [-port [destinat

Oliver Lyak 137 Dec 28, 2022
Dodo - A graphical, hackable email client based on notmuch

Dodo Dodo is a graphical email client written in Python/PyQt5, based on the comm

Aleks Kissinger 44 Nov 12, 2022
StudyLion is a Discord bot that tracks members' study and work time while offering members to view their statistics and use productivity tools such as: To-do lists, Pomodoro timers, reminders, and much more.

StudyLion - Discord Productivity Bot StudyLion is a Discord bot that tracks members' study and work time while offering members the ability to view th

45 Dec 26, 2022
A modular Telegram Python bot running on python3 with a sqlalchemy database

Nao Tomori Robot Found Me On Telegram As Nao Tomori 🌼 A modular Telegram Python bot running on python3 with a sqlalchemy database. How to setup/deplo

Sena 84 Jan 04, 2023
Centralized whale instance using github actions, sourcing metadata from bigquery-public-data.

Whale Demo Instance: Bigquery Public Data This is a fully-functioning demo instance of the whale data catalog, actively scraping data from Bigquery's

Hyperquery 17 Dec 14, 2022
Paid Udemy Courses with Coupons

Freedemy Paid Udemy Courses with Coupons Steps to run pip3 install -r requirements.txt python3 free-courses.py Then you can click the Enroll Link and

GOKUL A.P 23 Dec 14, 2022