refcount
master:
testing:
This package is primarily for managing resources in native libraries, written for instance in C++, from Python. While it boils down to "simply" maintaining a set of counters, it is deceptively complicated to do so properly and not end up with memory leaks or crashes. This package offers structured options for reliably managing external native resources. Surprisingly I could not locate an existing package doing just what I needed. Other use cases requiring reference counting, aside from native library resources, may benefit from reusing and extending classes in refcount.
refcount from version 0.7 includes classes using cffi. Other low-level interoperability mechanisms may well be added in the future.
License
MIT (see License.txt)
Documentation
Hosted at pyrefcount.readthedocs.io
Source code
The code repository is on GitHub.
Installation
pip install refcount
From source:
pip install -r requirements.txt
python setup.py install
Sample use
The following example is based on one of the unit tests.
Say we have a C++ library with objects and a C API:
#define TEST_DOG_PTR testnative::dog*
#define TEST_OWNER_PTR testnative::owner*
#define TEST_COUNTED_PTR testnative::reference_counter*
testnative::dog* create_dog();
testnative::owner* create_owner(testnative::dog* d);
void say_walk(testnative::owner* owner);
void release(testnative::reference_counter* obj);
// etc.
From the outside of the library the API is exported with opaque pointers void* (C structs pointers and native C99 types could be handled too).
void* create_dog();
void* create_owner(void* d);
void say_walk(void* owner);
void release(void* obj);
// etc.
Starting with the end in mind, from Python we want an API hiding the low level details close to the C API, in particular avoiding managing native memory via release C API calls, piggybacking the python GC instead.
dog = Dog()
owner = DogOwner(dog)
owner.say_walk()
print(dog.position)
dog = None # the "native dog" is still alive though, as the owner incremented the ref count
owner = None
This is doable with refcount and the cffi package. One possible design is:
ut_ffi = cffi.FFI()
ut_ffi.cdef('extern void* create_dog();')
ut_ffi.cdef('extern void* create_owner( void* d);')
ut_ffi.cdef('extern void say_walk( void* owner);')
ut_ffi.cdef('extern void release( void* obj);')
# etc.
ut_dll = ut_ffi.dlopen('c:/path/to/test_native_library.dll', 1) # Lazy loading
class CustomCffiNativeHandle(CffiNativeHandle):
def __init__(self, pointer, prior_ref_count = 0):
super(CustomCffiNativeHandle, self).__init__(pointer, type_id='', prior_ref_count = prior_ref_count)
def _release_handle(self) -> bool:
ut_dll.release(self.get_handle())
return True
class Dog(CustomCffiNativeHandle):
def __init__(self, pointer = None):
if pointer is None:
pointer = ut_dll.create_dog()
super(Dog, self).__init__(pointer)
# etc.
class DogOwner(CustomCffiNativeHandle):
def __init__(self, dog):
super(DogOwner, self).__init__(None)
self._set_handle(ut_dll.create_owner(dog.get_handle()))
self.dog = dog
self.dog.add_ref() # Do note this important reference increment
def say_walk(self):
ut_dll.say_walk(self.get_handle())
def _release_handle(self) -> bool:
super(DogOwner, self)._release_handle()
# super(DogOwner, self)._release_handle()
self.dog.release()
return True
Related work
Ancestry, acknowledgements
This python package refcount actually spawned from prior work for interoperability between C++, R and .NET (R.NET)
refcount features using cffi were also significantly informed by Kevin Plastow's work while he was at the Australian Bureau of Meteorology; this contribution is gratefully acknowledged.
In you have native interop needs you may also want to look at:
- the nuget package dynamic-interop-dll for .NET/native interop.
- a set of mostly c++ software tools for interop with C/C++
- a C# library for generating interop glue code on top of C API glue code.
Other python packages
refcount was created in part because no existing prior (Python) work could quite fit the need. There are however packages that may better address your particular need:
- infi.pyutils contains a reference counting class.
1 Feb 9, 2022
1 Nov 20, 2021
10 Nov 19, 2022
55 Sep 4, 2022
44 Dec 20, 2022
18.6k Jan 2, 2023
19 Apr 24, 2022
10 Jun 5, 2022
40 Dec 22, 2022
2 Mar 20, 2022
1.7k Dec 30, 2022
65 Dec 25, 2022
4 Mar 18, 2022
27 Nov 29, 2022
3 Nov 30, 2021
1 Oct 20, 2021
1 Jan 26, 2022
46 Dec 14, 2022
23 Dec 27, 2022
147 Dec 01, 2022
55 Dec 26, 2022
3 Jun 03, 2022
43 Dec 26, 2022
61 Sep 21, 2022
9 Oct 19, 2022
14 Dec 22, 2022
5 Nov 21, 2021
23 Nov 26, 2021
15 Dec 08, 2022