Problem:

The problem is the missing support for colors on windows machines and is described in #55.

Changes:

• I refactored the devtools.debug.Debug._env_bool classmethod as a function env_bool in devtools.prettier Reason: The method was not necessary to be a classmethod so I refactored it near env_true function which is used by the method.
• I added tests for new function env_bool.
• I introduced the highlight module which takes the roll to distinguish if highlighting is used. More info below
• I added tests for the new highlight`module.
• I changed Debug.__init__ and Debug.__call__ for the use of the new highlight.use_highlight function.

highlight module

On import the module will, when on a windows machine, try to activate windows ANSI support by setting the ENABLE_VIRTUAL_TERMINAL_PROCESSING flag via SetConsoleMode. See here for more info: https://docs.microsoft.com/en-us/windows/console/console-virtual-terminal-sequences. Depending on the outcome the color_active variable will be set.

The code which does the work was posted here: https://bugs.python.org/msg291732. Truth be told I only kinda get what the code does but I think the author knows what he does. I tested it on two Win10 machines and it works.

Next there is the use_highlight function. It took over the work to distinguish if DebugOutput.str function called by Debug.__call__ should use highlighting. The function will first check if the Debug class instance got created with highlight=True. If not set it will check the PY_DEVTOOLS_HIGHLIGHT environment variable. If also not set it decides by checking if isatty(file_) and color_active variable are True. The latter part is new and the variable gets set on import (see above). The part before is just the old behavior.

fixes #55

EDIT: Info: My added tests succeed on windows but 21 others do not.

Closes #114

I just tested it with executing==1.2.0 and asttokens==2.1.0 and everything seems to work, the tests pass.

This will allow users to update executing to a version greater than 1.0.0, for example in the case of global installation.

Please let me know if there is anything that would prevent this change from being possible. I checked the executing's API and there wasn't any breaking change.

This brings in two dependencies to do the heavy lifting for debug:

• https://github.com/alexmojaki/executing finds the AST node
• https://github.com/gristlabs/asttokens converts AST nodes to source code

Both are very well tested and reliable and are used in several other libraries with no issues, including https://github.com/gruns/icecream. You'll note that several test cases no longer give warnings. In particular this fixes #47 and #71.

executing can identify the AST node in almost all situations, which means:

• #69 will be easy to implement, not just in variable assignments but also in things like list comprehensions as a commenter mentions.
• Any function name is allowed, e.g. from devtools import debug as dbg will work. In fact the function can be any expression.

Hi,

I've been using the debug() function of devtools to print out some data structures for about 1-2 months now and I've been loving it!

After taking a break from a Python program for about 2 weeks I've gone back to start improving it again and I've noticed that now, when I use debug(), the colorized output looks very weird. Since I haven't seen any other GitHub issues on the subject I'm sure the problem is on my end, but I can't figure out what it is.

To simplify things, I've created a new venv, installed devtools and Pygments using the suggested pip install devtools[pygments]:

❯ pip list
Package    Version
---------- -------
devtools   0.6
pip        20.2.4
Pygments   2.7.1
setuptools 49.2.1

❯ python -V
Python 3.8.6

and created the simplest program I could think of:

#!/usr/bin/env python

from devtools import debug

a = 5

debug(a)

Still, the output is like this:

I normally use Bash on ArchLinux, but I've tried running this in Fish with the same result. I have installed a fresh Fedora 32 Workstation in a VM, created a new venv, and tried the same program, and get the same result.

I really don't understand why I am suddenly getting this results and what I am doing wrong.

Any tips/hints on where I should look to troubleshoot further? Thanks.

It's convenient to be able to use debug() as if it were a builtin function, but importing devtools directly in sitecustomize.py is a pretty significant performance hit for every invocation of python that doesn't use debug(). Specifically, this import takes 100-200 ms, which is a noticeable delay for interactive sessions and short-running scripts:

$ python -X importtime -c 'import devtools'

While it would probably be possible to make devtools faster to import, the best solution is simply to avoid importing devtools unless it will be used. It took me a while to figure out how to do this, but in the end the code is pretty clear and succinct, and I think it will be useful to others:

# sitecustomize.py
class lazy_debug:
    @property
    def __call__(self):
        from devtools import debug
        return debug

__builtins__['debug'] = lazy_debug()

This PR updates the README file to include the above snippet.

I'll describe fully how the __pretty__ unofficial dunder method is used by devtools on the remote off chance that @willmcgugan might adopt a similar thing in rich.

This approach seems to work but is completely unofficial and wasn't developed in collaboration with others, so might require changes. I'd be very happy to support a changes to this, particularly if it can be somewhat backwards compatible.

The Idea

Provide a way for objects to describe how they should be displayed in "rich" (no pun intended) contexts, without moving all the logic of pretty printing into the object.

The implementation

objects which want to be pretty printed include a __pretty__ method with takes as arguments:

• fmt a callable which takes one argument (usage described below)
• extra **kwargs in case we want to pass more things to the method later

It yields (or returns a list of) any of the following:

• string, in which case that string is printed on the current line and the current indentation
• int, in which case:
  • 1 means increase the indent by one level (4 spaces, 2 spaces, 1 tab, whatever you want it to mean)
  • -1 means decrease the indent by one level
  • 0 means new line, same indent
  • I think other numbers can be used to increase or decrease the indent more (e.g. 2 -> increase by two levels), but I can't remember if this is actually implemented, it's generally not needd
• the result of fmt(<whatever>) in which case the application calling __pretty__ can decide how to render the argument to fmt(). This is useful for recursive pretty printing - e.g. a custom type of list doesn't need to know how to render all it's values.

An example

Here' is the __pretty__ method from the reusable Representation method in pydantic with more comments:

    def __pretty__(self, fmt: Callable[[Any], Any], **kwargs: Any) -> Generator[Any, None, None]:
        # show the name of the class together with an opening bracket so it looks a bit like a instance definition:
        yield self.__repr_name__() + '('
        # new line and increase the indent
        yield 1
        for name, value in self.__repr_args__():
            # if this class wants to show name value pairs, show '<name>=`
            if name is not None:
                yield name + '='
            # render the value, devtools (or any tool calling this) can decide how to display the value
            yield fmt(value)
            # add a comma to the end of the line to keep it looking like an instance definition
            yield ','
            # new line without a change of indent
            yield 0
        # decrease indent, no extra new line
        yield -1
        # closing bracket to keep it looking a bit like python
        yield ')'

with self.__repr_name__() --> 'TheName' and self.__repr_args__() --> [('x', 1), ('y', 2), (None, 3)], it might display

TheName(
    x=1,
    y=2
    3,
)

Hi,

thanks for devtools I use it quite a lot. Now I stumbled over the following:

>>> from devtools import debug

>>> class W:
...     pass

>>> debug(W)
<stdin>:1 <module> (no code context for debug call, code inspection impossible)
    <class '__main__.W'> (type)

>>> class W(:
...     pass
>>> from pydantic import BaseModel

>>> debug(BaseModel)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/weissl/.pyenv/versions/3.8.2/envs/cauldron/lib/python3.8/site-packages/devtools/debug.py", line 122, in __call__
    s = d_out.str(highlight)
  File "/Users/weissl/.pyenv/versions/3.8.2/envs/cauldron/lib/python3.8/site-packages/devtools/debug.py", line 85, in str
    return prefix + '\n    ' + '\n    '.join(a.str(highlight) for a in self.arguments)
  File "/Users/weissl/.pyenv/versions/3.8.2/envs/cauldron/lib/python3.8/site-packages/devtools/debug.py", line 85, in <genexpr>
    return prefix + '\n    ' + '\n    '.join(a.str(highlight) for a in self.arguments)
  File "/Users/weissl/.pyenv/versions/3.8.2/envs/cauldron/lib/python3.8/site-packages/devtools/debug.py", line 45, in str
    s += pformat(self.value, indent=4, highlight=highlight)
  File "/Users/weissl/.pyenv/versions/3.8.2/envs/cauldron/lib/python3.8/site-packages/devtools/prettier.py", line 80, in __call__
    self._format(value, indent_current=indent, indent_first=indent_first)
  File "/Users/weissl/.pyenv/versions/3.8.2/envs/cauldron/lib/python3.8/site-packages/devtools/prettier.py", line 94, in _format
    gen = pretty_func(fmt=fmt, skip_exc=SkipPretty)
  File "pydantic/utils.py", line 263, in pydantic.utils.Representation.__pretty__
TypeError: __pretty__() takes exactly 2 positional arguments (0 given)

__pretty__() takes exactly 2 positional arguments (0 given)

>>> class W(BaseModel):
...     pass

>>> debug(W)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/weissl/.pyenv/versions/3.8.2/envs/cauldron/lib/python3.8/site-packages/devtools/debug.py", line 122, in __call__
    s = d_out.str(highlight)
  File "/Users/weissl/.pyenv/versions/3.8.2/envs/cauldron/lib/python3.8/site-packages/devtools/debug.py", line 85, in str
    return prefix + '\n    ' + '\n    '.join(a.str(highlight) for a in self.arguments)
  File "/Users/weissl/.pyenv/versions/3.8.2/envs/cauldron/lib/python3.8/site-packages/devtools/debug.py", line 85, in <genexpr>
    return prefix + '\n    ' + '\n    '.join(a.str(highlight) for a in self.arguments)
  File "/Users/weissl/.pyenv/versions/3.8.2/envs/cauldron/lib/python3.8/site-packages/devtools/debug.py", line 45, in str
    s += pformat(self.value, indent=4, highlight=highlight)
  File "/Users/weissl/.pyenv/versions/3.8.2/envs/cauldron/lib/python3.8/site-packages/devtools/prettier.py", line 80, in __call__
    self._format(value, indent_current=indent, indent_first=indent_first)
  File "/Users/weissl/.pyenv/versions/3.8.2/envs/cauldron/lib/python3.8/site-packages/devtools/prettier.py", line 94, in _format
    gen = pretty_func(fmt=fmt, skip_exc=SkipPretty)
  File "pydantic/utils.py", line 263, in pydantic.utils.Representation.__pretty__
TypeError: __pretty__() takes exactly 2 positional arguments (0 given)

__pretty__() takes exactly 2 positional arguments (0 given)

>>>

Could you please give me a hint what I am doing wrong here?

Due to documentation, it seems that using __builtins__ directly is not reliable. Instead they were using the __builtin__ module which was renamed to builtins.

Having __builtins__ and __builtin__ both is clearly a bad idea.

[Python-Dev] __builtin__ vs __builtins__

I hope this helps, as I am just starting with open source contributions.

• Add a PEP 561 marker file (py.typed).
• Run mypy during CI.
• Update black to avoid https://github.com/pallets/click/issues/2225
• Update mkdocs to avoid https://github.com/mkdocs/mkdocs/issues/2799
• Disable mkdocs strict mode to avoid https://github.com/mkdocs/mkdocs/issues/2252

https://python-devtools.helpmanual.io/usage/#usage-without-import

Suggests the following snippet:

    __builtins__['debug'] = debug

In python 3.10 (at least, only version tried), referencing builtins directly gives "TypeError: 'module' object is not subscriptable"

builtins.dict, however, is subscriptable.

That figuring out where sitecustomize.py (or usercustomize.py) actually can be put so it gets properly executed on startup on a multi-python and/or brew (linuxbrew) system is an absolute nightmare is a different issue and not one I'd expect python-devtools to address particularly.

Bug

When the from devtools import debug statement occurs within a method (as opposed to at toplevel of the module), an "error parsing code" message is printed.

Min repro:

The following works as expected:

# tmp.py
from devtools import debug
def foo():
    debug(123)
foo()

$ python tmp.py
tmp.py:4 foo
    123 (int)

But if you put the import statement inside the method, there is an error:

# tmp2.py
def foo():
    from devtools import debug
    debug(123)
foo()

$ python tmp2.py
tmp2.py:4 foo (error parsing code, unable to find "debug" function statement)
    123 (int)

we should move https://github.com/samuelcolvin/dirty-equals/pull/51 to this package.

Function should be debug.insert_assert, but we still need the library to become a pytest plugin so tests with insert_assert can fail.

(I thought this was already an issue, but I checked and it seems it only existed in my head)

I want a way to pretty-print pertinent attributes of an object, like vars(...) but more intelligent - in particular it should try to inspect __slots__ if __dict__ is not available.

This might be as simple as sugar around the following POC, but there are probably edge cases I haven't thought about.

I guess we should expose this via debug.inspect().

from __future__ import annotations
from devtools import debug

ignored_attributes = {
    '__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__',
    '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__',
    '__init__', '__init_subclass__', '__le__', '__lt__', '__module__',
    '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__',
    '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '__slots__',
}


class DebugInspect:
    __slots__ = ('obj',)

    def __init__(self, obj: Any):
        self.obj = obj

    def __pretty__(self, fmt: Callable[[Any], Any], **kwargs: Any) -> Generator[Any, None, None]:
        yield self.obj.__class__.__name__ + '('
        yield 1
        # use __dict__ if possible to maintain order, also should be slightly faster
        obj_dict = getattr(self.obj, '__dict__', None)
        if obj_dict is not None:
            for name, value in obj_dict.items():
                yield name + '='
                yield fmt(value)
                yield ','
                yield 0
        else:
            for name in dir(self.obj):
                if name not in ignored_attributes:
                    yield name + '='
                    yield fmt(getattr(self.obj, name))
                    yield ','
                    yield 0
        yield -1
        # closing bracket to keep it looking a bit like python
        yield ')'


class Foo:
    def __init__(self):
        self.x = 1
        self.y = 2
        self._private = 3
        self.__custom_dunder__ = 4


class Bar:
    __slots__ = 'x', 'y', '_private', '__custom_dunder__'

    def __init__(self):
        self.x = 1
        self.y = 2
        self._private = 3
        self.__custom_dunder__ = 4


f = Foo()
debug(DebugInspect(f))

b = Bar()
debug(DebugInspect(b))

foobar.py:61 <module>
    DebugInspect(f): Foo(
        x=1,
        y=2,
        _private=3,
        __custom_dunder__=4,
    ) (DebugInspect)
foobar.py:64 <module>
    DebugInspect(b): Bar(
        __custom_dunder__=4,
        _private=3,
        x=1,
        y=2,
    ) (DebugInspect)

Today I was trying to debug an issue where a function of mine was unexpectedly being called from two places. I was using debug() within the function, but it only reports the file/line number/function that it was directly invoked from, which in this case I knew. I wanted to know where this function was being called from, so I looked for a way to get a full stack trace by doing something like debug(trace=True). As far as I can tell, this option doesn't exist yet.

Is there interest in adding such an option? If so, I could probably try to make a PR in the near-ish future.

I get my "debugged" vars printed both by debug utils and as a return val, so in case of dataclasses or big dicts (for which it fits the most) it eats all screen space.

Am I missing something or it works as expected?

Python 3.10.2 (main, Jan 29 2022, 02:55:36) [GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from devtools import debug
>>> debug({'a': 1, 'b': 2})
<stdin>:1 <module> (no code context for debug call, code inspection impossible)
    {
        'a': 1,
        'b': 2,
    } (dict) len=2
{'a': 1, 'b': 2}

First of all: I just discover devtools and I think it is a fantastic tool. I definitely plan to use it instead of my simple minded debugging helper.

Currently, debug() is assumed to be called explicitly in a given module, with a hard-coded value for the frame in which it looks for information:

    def _process(self, args, kwargs) -> DebugOutput:
        """
        BEWARE: this must be called from a function exactly 2 levels below the top of the stack.
        """
        # HELP: any errors other than ValueError from _getframe? If so please submit an issue
        try:
            call_frame: 'FrameType' = sys._getframe(2)
        except ValueError:
            ...

I would find it useful if the constant 2 could be a configurable parameter. For example, I could then define custom functions like the following:

def handle_problem(*args, **kwargs):
    if dev_version:
        debug = Debug(additional_frame_depth=1)
        debug(*args, **kwargs)
    else:
        print("Internal problem: please report this case.")

Obviously, I could currently do this by subclassing Debug, but I think that this might be a useful feature to have. I understand that the idea is to use devtools.debug during development and one would normally remove all traces of it for releases. However, I find it useful in one of my projects (friendly-traceback) to leave in place such debugging functions such that end-users are never exposed to tracebacks generated by my own project while still being able to cope with the unexpected.