Bringing Async Capabilities to django ORM

Overview

Disclaimer: Don't use this module in production it's still in active development.

Django Async Orm

Django module that brings async to django ORM.

Installing

python -m pip install django-async-orm

then add django_async_orm to your INSTALLED_APPS list:

INSTALLED_APPS = [
    ...,
    'django_async_orm'
]

Usage

Django Async Orm will patch all your existing models to add async_* prefixed methods. To be

example:

class MyModel(models.Model):
    name = models.CharField(max_length=250)

you can use it as follow:

async def get_model():
    return await  MyModel.objects.async_get(name="something")

you can also iterate over a query set with async for:

async def all_models():
    all_result_set = await MyModel.objects.async_all()
    async for obj in all_result_set:
        print(obj)

Some wrappers are also available for template rendering, form validation and login/logout

Async login

from django_async_orm.wrappers import async_login

async def my_async_view(request):
    await async_login(request)
    ...

Form validation

from django_async_orm.wrappers import async_form_is_valid
async def a_view(request):
    form = MyForm(request.POST)
    is_valid_form = await async_form_is_valid(form)
    if is_valid_form:
        ...
    

Django ORM support:

This is an on going projects, not all model methods are ported.

Manager:

methods supported comments
Model.objects.async_get
Model.objects.async_create
Model.objects.async_bulk_create
Model.objects.async_bulk_update
Model.objects.async_get_or_create
Model.objects.async_update_or_create
Model.objects.async_earliest
Model.objects.async_latest
Model.objects.async_first
Model.objects.async_last
Model.objects.async_in_bulk
Model.objects.async_delete
Model.objects.async_update
Model.objects.async_exists
Model.objects.async_explain
Model.objects.async_raw
Model.objects.async_all
Model.objects.async_filter
Model.objects.async_exclude
Model.objects.async_complex_filter
Model.objects.async_union
Model.objects.async_intersection
Model.objects.async_difference
Model.objects.async_select_for_update
Model.objects.async_prefetch_related
Model.objects.async_annotate
Model.objects.async_order_by
Model.objects.async_distinct
Model.objects.async_difference
Model.objects.async_extra
Model.objects.async_reverse
Model.objects.async_defer
Model.objects.async_only
Model.objects.async_using
Model.objects.async_resolve_expression
Model.objects.async_ordered
__aiter__
__repr__
__len__
__getitem__
Model.objects.async_iterator

Model:

methods supported comments
Model.async_save
Model.async_update
Model.async_delete
...

User Model / Manager

methods supported comments
UserModel.is_authenticated
UserModel.is_super_user
UserModel.objects.async_create_user
...

Foreign object lazy loading:

Not supported

Wrappers:

methods supported comments
wrappers.async_render
wrappers.async_login
wrappers.async_logout
Comments
  • add none query feature and test for the same

    add none query feature and test for the same

    Added none query feature:

    in Django ORM, calling none() will create a queryset that never returns any objects and no query will be executed when accessing the results. A qs.none() queryset is an instance of EmptyQuerySet.

    example:

    Entry.objects.none() <QuerySet []> from django.db.models.query import EmptyQuerySet isinstance(Entry.objects.none(), EmptyQuerySet) True

    opened by rkisdp 4
  • replace the default ThreadPoolExecutor with gevent.threadpool.ThreadPoolExecutor

    replace the default ThreadPoolExecutor with gevent.threadpool.ThreadPoolExecutor

    I have no idea if this will work, but maybe it will?

    Here's a bit of my research on what I was thinking: https://github.com/rednaks/django-async-orm/discussions/9

    web_1         | DEBUG:root:concurrent.futures.Future is expected, got <gevent.threadpool._FutureProxy object at 0x7fcabce2e9d0>
    web_1         | DEBUG:root:concurrent.futures.Future is expected, got <gevent.threadpool._FutureProxy object at 0x7fcabf109bb0>
    web_1         | DEBUG:root:concurrent.futures.Future is expected, got <gevent.threadpool._FutureProxy object at 0x7fcabf109250>
    web_1         | DEBUG:root:concurrent.futures.Future is expected, got <gevent.threadpool._FutureProxy object at 0x7fcabc5654f0>
    web_1         | DEBUG:root:concurrent.futures.Future is expected, got <gevent.threadpool._FutureProxy object at 0x7fcabd7846d0>
    ^CGracefully stopping... (press Ctrl+C again to force)
    Stopping benchmark-django-fastapi_web_1      ... done
    Stopping benchmark-django-fastapi_postgres_1 ... done
    
    

    looks kind of close ... although something is expecting concurrent.futures.Future instead of also allowing gevent.threadpool._FutureProxy

    update

    a recursion error of some sort logged from here : https://github.com/rednaks/django-async-orm/pull/7/files#diff-142ac986547c5b13f77ae025cb60609eed65082cc41ac6e81f4b778422d50017R116

    recursion.error.txt

    opened by allen-munsch 4
  • TypeError: 'NoneType' object is not iterable

    TypeError: 'NoneType' object is not iterable

    Thanks for sharing this library. Very interesting!

    Any suggestions on why NoneType would be returned from async_all?

    web              |   File "/venv/lib/python3.8/site-packages/fastapi/applications.py", line 199, in __call__
    web              |     await super().__call__(scope, receive, send)
    web              |   File "/venv/lib/python3.8/site-packages/starlette/applications.py", line 112, in __call__
    web              |     await self.middleware_stack(scope, receive, send)
    web              |   File "/venv/lib/python3.8/site-packages/starlette/middleware/errors.py", line 181, in __call__
    web              |     raise exc from None
    web              |   File "/venv/lib/python3.8/site-packages/starlette/middleware/errors.py", line 181, in __call__
    web              |     raise exc from None
    web              |   File "/venv/lib/python3.8/site-packages/starlette/exceptions.py", line 93, in __call__
    web              |     await response(scope, receive, sender)
    web              |   File "/venv/lib/python3.8/site-packages/starlette/exceptions.py", line 93, in __call__
    web              |     await response(scope, receive, sender)
    web              |   File "/venv/lib/python3.8/site-packages/starlette/routing.py", line 580, in __call__
    web              |     await route.handle(scope, receive, send)
    web              |   File "/venv/lib/python3.8/site-packages/starlette/routing.py", line 241, in handle
    web              |     await self.app(scope, receive, send)
    web              |   File "/venv/lib/python3.8/site-packages/starlette/routing.py", line 52, in app
    web              |     response = await func(request)
    web              |   File "/venv/lib/python3.8/site-packages/fastapi/routing.py", line 201, in app
    web              |     raw_response = await run_endpoint_function(
    web              |   File "/venv/lib/python3.8/site-packages/fastapi/routing.py", line 148, in run_endpoint_function
    web              |     return await dependant.call(**values)
    web              |   File "/app/api/generated_app/main.py", line 100, in get_buildings
    web              |     data = await get_data()
    web              |   File "/app/api/generated_app/main.py", line 88, in get_data
    web              |     async for building in data:
    web              |   File "/venv/lib/python3.8/site-packages/django_async_orm/query.py", line 71, in __aiter__
    web              |     return AsyncIter(self._result_cache)
    web              |   File "/venv/lib/python3.8/site-packages/django_async_orm/iter.py", line 6, in __init__
    web              |     self._iter = iter(iterable)
    web              | TypeError: 'NoneType' object is not iterable
    

    py3.8.10 django3.2

    async def get_data():
        data = await Building.objects.async_all()
        result = []
        async for building in data:
            print(data, dir(data))
            print(building.__dict__)
            result.append({"id": building.id, "number": building.number})
        return result
    
    @app.get("/buildings", response_model=List[BuildingSummary])
    async def get_buildings(limit: conint(le=500), skip: conint(le=500)) -> List[BuildingSummary]:
        """
        Building List
        """
        # this is a custom thing
        data = await get_data()
        return data
    
    opened by allen-munsch 3
  • PR for issue #2

    PR for issue #2

    async def get_data():
        return await Building.objects.async_all()
    
    @app.get("/buildings", response_model=List[BuildingSummary])
    async def get_buildings(
        limit: conint(le=500),
        skip: conint(le=500),
        x_pm_org: Optional[List[int]] = Header(None, alias="x-pm-org"),
        x_pm_org_id: Optional[List[int]] = Header(None, alias="x-pm-org-id"),
        x_pm_user_type: Optional[str] = Header(None, alias="x-pm-user-type"),
    ) -> List[BuildingSummary]:
        """
        Building List
        [{'ApiKeyAuth': []}, {'PmOrg': []}]
        """
    
        org.set(x_pm_org)
        pydant, item_all = paths["get_buildings"]
    
    
    
        # this works
        data = await get_data()
    
        # this also works
        data2 = await Buildings.objects.async_all()
    
    
    
        result = []
        async for item in data[skip:limit]:
            print(item.__dict__)
            result.append(pydant.from_orm(item).dict())
        return result
    

    Related to: #2

    I just tested it locally and it appears to work.

    Thanks for the quick response.

    Not sure where to ask this, possibly could enable the "discussions" section on the repo?

    But how does the ThreadPoolExecutor work with something like psycogreen/eventlet?

    opened by allen-munsch 1
  • comply with official django async orm syntax

    comply with official django async orm syntax

    when we patch models, we prefix the methods with async_ but the new official django api prefixes the methods with a ex:

    patched api : sync objects.get => async objects.async_get official api: sync objects.get => async objects.aget

    so to make it forward compatible we need to comply to django's new api.

    be aware to keep our api so we don't break the internet

    opened by rednaks 0
  • Problem with default update_or_create()

    Problem with default update_or_create()

    Hello! After installing django_async_orm facing such problem with default update_or_create() function

    image

    Do you have any ideas by a chance why is it going like this?

    python 3.8 django 3.2.9 django-async-orm latest

    opened by max4nik 0
  • Count feature, related test and import optimisation

    Count feature, related test and import optimisation

    Hi, @rednaks,

    I have added the count feature and test for the same, and with that, I have also changed some other tests accordingly and removed unused imports from the file. Please have a look and give your feedback.

    opened by rkisdp 8
  • Perhaps it is better to use channels?

    Perhaps it is better to use channels?

    Perhaps it's worth starting django-channels library for asynchronous db calls?

    There is a function there. And, as I understand it, it will be more effective than it is now. image

    opened by daveusa31 0
Releases(0.1.4)
Python helpers for using SQLAlchemy with Tornado.

tornado-sqlalchemy Python helpers for using SQLAlchemy with Tornado. Installation $ pip install tornado-sqlalchemy In case you prefer installing from

Siddhant Goel 122 Aug 23, 2022
Sqlalchemy seeder that supports nested relationships.

sqlalchemyseed Sqlalchemy seeder that supports nested relationships. Supported file types json yaml csv Installation Default installation pip install

Jedy Matt Tabasco 10 Aug 13, 2022
A pure Python Database Abstraction Layer

pyDAL pyDAL is a pure Python Database Abstraction Layer. It dynamically generates the SQL/noSQL in realtime using the specified dialect for the databa

440 Nov 13, 2022
Bringing Async Capabilities to django ORM

Bringing Async Capabilities to django ORM

Skander BM 119 Dec 01, 2022
MongoEngine flask extension with WTF model forms support

Flask-MongoEngine Info: MongoEngine for Flask web applications. Repository: https://github.com/MongoEngine/flask-mongoengine About Flask-MongoEngine i

MongoEngine 815 Jan 03, 2023
A single model for shaping, creating, accessing, storing data within a Database

'db' within pydantic - A single model for shaping, creating, accessing, storing data within a Database Key Features Integrated Redis Caching Support A

Joshua Jamison 178 Dec 16, 2022
SQLAlchemy support for aiohttp.

aiohttp-sqlalchemy SQLAlchemy 1.4 / 2.0 support for AIOHTTP. The library provides the next features: initializing asynchronous sessions through a midd

Ruslan Ilyasovich Gilfanov 5 Dec 11, 2022
A dataclasses-based ORM framework

dcorm A dataclasses-based ORM framework. [WIP] - Work in progress This framework is currently under development. A first release will be announced in

HOMEINFO - Digitale Informationssysteme GmbH 1 Dec 24, 2021
A Python Library for Simple Models and Containers Persisted in Redis

Redisco Python Containers and Simple Models for Redis Description Redisco allows you to store objects in Redis. It is inspired by the Ruby library Ohm

sebastien requiem 436 Nov 10, 2022
Easy-to-use data handling for SQL data stores with support for implicit table creation, bulk loading, and transactions.

dataset: databases for lazy people In short, dataset makes reading and writing data in databases as simple as reading and writing JSON files. Read the

Friedrich Lindenberg 4.2k Dec 26, 2022
A PostgreSQL or SQLite orm for Python

Prom An opinionated lightweight orm for PostgreSQL or SQLite. Prom has been used in both single threaded and multi-threaded environments, including en

Jay Marcyes 18 Dec 01, 2022
Adds SQLAlchemy support to Flask

Flask-SQLAlchemy Flask-SQLAlchemy is an extension for Flask that adds support for SQLAlchemy to your application. It aims to simplify using SQLAlchemy

The Pallets Projects 3.9k Jan 09, 2023
Prisma Client Python is an auto-generated and fully type-safe database client

Prisma Client Python is an unofficial implementation of Prisma which is a next-generation ORM that comes bundled with tools, such as Prisma Migrate, which make working with databases as easy as possi

Robert Craigie 930 Jan 08, 2023
A pythonic interface to Amazon's DynamoDB

PynamoDB A Pythonic interface for Amazon's DynamoDB. DynamoDB is a great NoSQL service provided by Amazon, but the API is verbose. PynamoDB presents y

2.1k Dec 30, 2022
The Orator ORM provides a simple yet beautiful ActiveRecord implementation.

Orator The Orator ORM provides a simple yet beautiful ActiveRecord implementation. It is inspired by the database part of the Laravel framework, but l

Sébastien Eustace 1.4k Jan 01, 2023
A database migrations tool for TortoiseORM, ready to production.

Aerich Introduction Aerich is a database migrations tool for Tortoise-ORM, which is like alembic for SQLAlchemy, or like Django ORM with it's own migr

Tortoise 596 Jan 06, 2023
A very simple CRUD class for SQLModel! ✨

Base SQLModel A very simple CRUD class for SQLModel! ✨ Inspired on: Full Stack FastAPI and PostgreSQL - Base Project Generator FastAPI Microservices I

Marcelo Trylesinski 40 Dec 14, 2022
a small, expressive orm -- supports postgresql, mysql and sqlite

peewee Peewee is a simple and small ORM. It has few (but expressive) concepts, making it easy to learn and intuitive to use. a small, expressive ORM p

Charles Leifer 9.7k Jan 08, 2023
A new ORM for Python specially for PostgreSQL

A new ORM for Python specially for PostgreSQL. Fully-typed for any query with Pydantic and auto-model generation, compatible with any sync or async driver

Yan Kurbatov 3 Apr 13, 2022
A Python Object-Document-Mapper for working with MongoDB

MongoEngine Info: MongoEngine is an ORM-like layer on top of PyMongo. Repository: https://github.com/MongoEngine/mongoengine Author: Harry Marr (http:

MongoEngine 3.9k Dec 30, 2022