Hello everyone,

I got this message Scalene error: received signal SIGABRT what does it mean and what should i do ?

Thank you in advance for your answers

When trying scalene v1.1.1 (installed from source with pip install ., but I don't think that matters since libscalene.so is included in the source tarball), I'm getting this:

python3: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /.../lib/python3.6/site-packages/scalene/libscalene.so)

The issue here boils down to libscalene.so being compiled on an Linux OS that has a more recent glibc than is used in CentOS 7.8 (which is glibc 2.17).

Is there a specific reason why libscalene.so is packaged into the source tarball, as opposed to compiling it from source when scalene is being installed from source?

I'm happy to try and compile libscalene.so from source myself, but there doesn't seem to be any documentation available on that. I found heaplayers-make.mk which suggests that clang++ is needed (is that a strict requirement, or would a sufficiently recent g++ also work?), and that some source code from https://github.com/emeryberger/Heap-Layers is required as well.

Describe the bug When starting scalene (with poetry run python -m scalene my_script.py), I get the error Scalene: internal error: unable to find Python allocator functions. This happens both with poetry run python -m scalene my_script.py or if started within the venv (python -m scalene my_script.py or just scalene my_script.py).

Some seconds after this output, i get a long traceback (reported afterwards).

I don't really think this is something about my program, since it always worked without scalene, but I think it is something related to Trio.

To Reproduce Steps to reproduce the behavior:

Snippet that cause the error:

#asd.py

import time
import trio
time.sleep(3)

and then poetry run python -m scalene asd.py. This gives this traceback:

Error in program being profiled:
 string index out of range
Traceback (most recent call last):
  File "/home/didi/leaf/leaf-radio/.venv/lib/python3.7/site-packages/scalene/scalene_profiler.py", line 1326, in profile_code
    exec(code, the_globals, the_locals)
  File "asd.py", line 2, in <module>
    import trio
  File "/home/didi/leaf/leaf-radio/.venv/lib/python3.7/site-packages/trio/__init__.py", line 48, in <module>
    from ._sync import (
  File "/home/didi/leaf/leaf-radio/.venv/lib/python3.7/site-packages/trio/_sync.py", line 475, in <module>
    @attr.s(eq=False, hash=False, repr=False)
  File "/home/didi/leaf/leaf-radio/.venv/lib/python3.7/site-packages/attr/_make.py", line 1312, in wrap
    field_transformer,
  File "/home/didi/leaf/leaf-radio/.venv/lib/python3.7/site-packages/attr/_make.py", line 610, in __init__
    field_transformer,
  File "/home/didi/leaf/leaf-radio/.venv/lib/python3.7/site-packages/attr/_make.py", line 503, in _transform_attrs
    AttrsClass = _make_attr_tuple_class(cls.__name__, attr_names)
  File "/home/didi/leaf/leaf-radio/.venv/lib/python3.7/site-packages/attr/_make.py", line 299, in _make_attr_tuple_class
    eval(compile("\n".join(attr_class_template), "", "exec"), globs)
  File "", line 1, in <module>
  File "/home/didi/leaf/leaf-radio/.venv/lib/python3.7/site-packages/scalene/scalene_profiler.py", line 231, in invalidate_lines
    if f.f_code.co_filename[0] == "<" or "scalene" in f.f_code.co_filename:
IndexError: string index out of range

Commenting out the line import trio, makes everything works.

However, the line Scalene: internal error: unable to find Python allocator functions is always present.

Desktop (please complete the following information):

• OS: WSL2
• Python version: 3.7.10, installed with pyenv
• Scalene version: 1.3.15

When running a very simple script based on numpy functions, we can get the following results:

test2.py: % of CPU time = 100.00% out of   3.59s.
  	 |     CPU % |     CPU % | Avg memory  | Memory      |
  Line	 |  (Python) |  (native) | growth (MB) | usage (%)   | [test2.py]
--------------------------------------------------------------------------------
     1	 |           |           |             |             | import numpy as np
     2	 |           |           |             |             |
     3	 |     0.30% |    48.40% |         -80 |       1.03% | x = np.array(range(10**7))
     4	 |     0.59% |    50.72% |           0 |      98.97% | np.array(np.random.uniform(0, 100, size=10**8))
     5	 |           |           |             |             |

How can we get:

• A negative memory growth for the first line?
• A null memory growth on the second line?

System info

• Platform : Mac OS X
• Python: 3.7 (brew)
• Numpy: 1.16.4

Describe the bug I am trying to profile code where we import the pyproj library, and scalene fails with Scalene error: received signal SIGSEGV

To Reproduce

• Install the pyproj library
• create a file with the following line from pyproj import Proj
• run scalene my_file.py

Scalene will fail with the error above

Expected behaviour Scalene does not fail and I can profile like for any library !

I'm usually not interested in profiling an entire program, I'm more interested in profiling some hotspot of code, or just some new piece of code.

pprofile allows for just profiling a specific region of code (from pprofile's main page):

def someOtherHotSpotCallable():
    # Statistic profiler
    prof = pprofile.StatisticalProfile()
    with prof(
        period=0.001, # Sample every 1ms
        single=True, # Only sample current thread
    ):
        # Code to profile
    prof.print_stats()

It would be nice if scalene allowed for such granularity.

Because the use of sys.settrace created a large amount of overhead and was partially incorrect, we decided to reimplement the functionality in C. This is a relatively naive translation from Python to C. In addition, the __last_profiled field was set to a list so it could be cached in the C code, and some caching was added to should_trace

Hi! Thanks for making this profiler, I think it's awesome.

Describe the bug The profiler sometimes doesn't measure the memory usage.

To Reproduce

1. Define prof_test.py with the following:

import numpy as np

class A:

    def __init__(self, n):
        self.arr = np.random.rand(n)
        self.lst = [1] * n

if __name__ == '__main__':
    a = A(10_000_000)

2. Run scalene from the terminal

scalene prof_test.py --reduced-profile

Expected behavior Show the two allocations for the attributes arr and lst (approx 75 MB each).

Screenshots This is the expected behaviour:

Sometimes only the time is measured:

Other times only one of the allocations is measured:

Desktop (please complete the following information):

• OS: Ubuntu 20.10
• Version: 1.3.0

Getting Scalene error: received signal SIGSEGV when trying to profile a python program.

I believe one of the imports at the top of my code breaks the profiler.

test.py:

#!/usr/bin/env python
# coding: utf-8

import obspy

profiling

scalene test.py

output: Scalene error: received signal SIGSEGV

Desktop:

• OS: macOS-10.16-x86_64

PyEval_CallObject can be used to invoke python code from C. Seems scalene currently can't profile code run this way. Is this a fundamental limit, or can this functionality be added relatively easily? The use case I'm looking at is profiling tf.py_func calls in tensorflow.

Thanks for this great resource with amazing potential.

I have a script that runs in about 30 seconds (or up to 2h with different input files), but when I try to profile it, it just hangs on one step (for up to 8 hours now) , and I can't tell what the problem is.

I've tried the --profile-interval option, but it never outputs anything.

Is there something in the /tmp/scalene######/ folder or elsewhere that could give an idea?

Should I try setting the cpu-sampling-rate slower?

I am currently running it on Ubuntu 18 with python 3.8 and these options: --reduced-profile --cpu-only --outfile pppscalene.txt --cpu-percent-threshold 3

top reports that the associated python process is taking only 1% cpu

I have gotten it to successfully profile the benchmark.py program, although it seemed to also fail on that if I didn't specify --cpu-only

Does it have issues with multiprocessing? I have it set to one thread at the moment, but this script does use the multiprocessing library.

Suggestions on where to look?

Describe the bug I have a script main.py that imports another module, utils.py.

main.py

from utils import func1
...

I want to profile all of the functions (not just func1) in utils.py after calling main. But I don't see any output in the Memory column.

Expected behavior I expect the rows profiled in utils.py to show values in the Memory column. I tried --profile-only 'utils', but I still don't see values in that column.

Screenshots

Desktop (please complete the following information):

• macOS Monterey
• Firefox 108.0.1
• Scalene version 1.5.16 (2022.12.08)

Describe the bug When I attempt to use scalene to profile a program that uses python-isal via xopen, I get an error from an internal import within the python-isal package.

Specifically:

OSError: b'Traceback (most recent call last):
  File "/home/alex/.conda/envs/alex/envs/alyssa-profile/lib/python3.11/site-packages/scalene/scalene_profiler.py", line 1776, in profile_code
    exec(code, the_globals, the_locals)
  File "/home/alex/.conda/envs/alex/envs/alyssa-profile/lib/python3.11/site-packages/isal/igzip.py", line 41, in <module>
    from . import igzip_lib, isal_zlib
ImportError: cannot import name \'igzip_lib\' from \'scalene\' (/home/alex/.conda/envs/alex/envs/alyssa-profile/lib/python3.11/site-packages/scalene/__init__.py)
' (exit code 1)

Originating stack trace, just coming from xopen's error handling code:

Traceback (most recent call last):
  File "/home/alex/.conda/envs/alex/envs/alyssa-profile/lib/python3.11/site-packages/scalene/scalene_profiler.py", line 1776, in profile_code
    exec(code, the_globals, the_locals)
  File "minimal-reproduction.py", line 3, in <module>
    with xopen('some_file.txt.gz') as infile:
  File "/home/alex/.conda/envs/alex/envs/alyssa-profile/lib/python3.11/site-packages/xopen/__init__.py", line 157, in __exit__
    self.close()
  File "/home/alex/.conda/envs/alex/envs/alyssa-profile/lib/python3.11/site-packages/xopen/__init__.py", line 386, in close
    self._raise_if_error(check_allowed_code_and_message, stderr_message)
  File "/home/alex/.conda/envs/alex/envs/alyssa-profile/lib/python3.11/site-packages/xopen/__init__.py", line 452, in _raise_if_error
    raise OSError("{!r} (exit code {})".format(stderr_message, retcode))

The problem is coming from this relative import in python-isal: https://github.com/pycompression/python-isal/blob/develop/src/isal/igzip.py#L41

from . import igzip_lib, isal_zlib

This code works when I run the python program outside of scalene, as a script, or as a module. Only with scalene do I get this import failure.

To Reproduce

from xopen import xopen

with xopen('some_file.txt.gz') as infile:
  for line in infile:
    continue

Expected behavior I expect scalene to profile this code without the error

Desktop (please complete the following information):

• OS: Ubuntu 20.04.5 LTS
• Python 3.11.0

Additional context This does not seem to reproduce unless doing long reads. The first minimal reproduction case I tried was a small file and a single read instead of a loop, which somehow did not reproduce this problem for me.

Full pip freeze of my environment:

cloudpickle==2.2.0
commonmark==0.9.1
isal==1.1.0
Jinja2==3.1.2
MarkupSafe==2.1.1
numpy @ file:///home/conda/feedstock_root/build_artifacts/numpy_1668919096335/work
pandas==1.5.2
Pygments==2.13.0
pynvml==11.4.1
python-dateutil @ file:///home/conda/feedstock_root/build_artifacts/python-dateutil_1626286286081/work
pytz @ file:///home/conda/feedstock_root/build_artifacts/pytz_1667391478166/work
rich==12.6.0
scalene==1.5.16
six @ file:///home/conda/feedstock_root/build_artifacts/six_1620240208055/work
xopen==1.7.0

P.S. Thank you so much for Scalene! It's an awesome tool. This may well be a problem with python-isal, but I've used that package for a long time and never seen failures like this.

AWS and GCP provide a Serverless service where a server in a container starts on demand. This container has a default timeout that, usually, allows Scalene to profile the current job and not raise a job was too fast error. If there are no requests to the service the container stops.

Given that mostly all server frameworks (flask, sanic etc) have on_server_start and on_server_stop custom functions, would be nice to profile in production starting/stopping Scalene in these custom functions. (--profile-interval < container default timeout)

I've tested using scalene in GCP Cloud Run with CMD python -m scalene --profile-interval 2.0 --cli --html --outfile scalene.html app.py and failed to start the container even tho the CPU is always allocated (not allocated only during request processing).

#483 #432 #35

NOTE: This changeset should be merged with care because of the changes it makes to scalene's build workflows. At a minimum it should be tested on MacOS and Windows before merging!

This PR introduces a pyproject.toml to bring scalene into the world of PEP 517/518. setuptools still does most of the lifting, but most of the configuration is now declared in this more modern file. One big advantage of this change is that build-time dependencies are clearly separated from runtime dependencies, so end-users should not need to worry about installing packages like Cython or wheel in order to build the project. I think this means requirements.txt can be removed from the project, as it is no longer necessary in the GitHub workflow that depends on it, but I would appreciate maintainer insight on this.

Pre-merge checklist

This checklist may be incomplete, please add to it if anything is missing.

• [x] Test build/install on Linux
  • I can install (with pip install .) and build (with make bdist and make sdist) with a fresh 3.8.12 venv on Ubuntu 20.04.3
  • I also checked that DEV_BUILD is respected when performing a build, adding a .devN tag to the resulting package version
• [ ] Test build/install on MacOS
  • The MacOS smoketests passed, so pip install looks good. It would be a good idea to check the Makefile-driven workflow too (unfortunately, I don't have a Mac to test with)
• [ ] Test build/install on Windows
  • I was able to pip install the repository on Windows 10 in a fresh Python 3.8.10 environment, but the Makefile-driven workflow should be checked too

Hello,

this request comes from this discussion: https://github.com/plasma-umass/scalene/discussions/505

I was trying to run scalene inside Autodesk Maya 2023 from the script editor, but I got this warning:

I've seen other answers about this issue, but they rely on launching scalene like scalene foo.py, and I'm not quite sure how I can workaround this from inside the live script editor.

For context, I used to perform profiling using cProfile inside Maya using this:

I also tried to run scalene as a module as @emeryberger suggested (using runpy), but the warning keeps appearing:

import runpy
d = runpy.run_module('scalene')

def foo():
    from maya import cmds
    for _ in range(50):
        cmds.createNode('locator')

# Turn profiling on
d['scalene_profiler'].start()

foo()

# Turn profiling off
d['scalene_profiler'].stop()

Thanks!

Is your feature request related to a problem? Please describe. Not directly, but I am filing this issue after a short chat on the Boston Python Slack about how to update one of the project's dependencies.

Describe the solution you'd like To satisfy this request, scalene should support build systems that follow PEP 517, which mostly comes down to following PEP 518 by introducing a pyproject.toml file to the project's source tree.

The biggest advantage of this approach is that build-time dependencies (namely Cython and possibly crdp as well) can be declared as such in the modern style, so end-user installation should be slightly smoother.

Describe alternatives you've considered N/A

Additional context I sat down for an hour or so tonight and saw how much of the project's setup.py could be migrated to a pyproject.toml. The biggest snag I ran into was the DEV_BUILD functionality that appends a .devN suffix to builds. And of course all the dynamic stuff related to available compilers and extension modules still needs to be expressed in setup.py

The upshot here is that it's pretty easy to start small with a pyproject.toml, it is not a wholesale replacement for setup.py (or its declarative counterpart setup.cfg), as described in the setuptools documentation.