A django integration for huey task queue that supports multi queue management

Related tags

Djangodjango-huey
Overview

Version

django-huey

This package is an extension of huey contrib djhuey package that allows users to manage multiple queues.

Installation

Using pip package manager run:

# pip install Django  if not installed
# pip install huey    if not installed
pip install django-huey

Note: use a virtualenv to isolate your dependencies. Note 2: django and huey must be installed.

Then, in your settings.py file add django_huey to the INSTALLED_APPS:

INSTALLED_APPS = [
	...
    'django_huey',
]

Configuration

In settings.py you must add the HUEYS setting:

HUEYS = {
    'first': {#this name will be used in decorators below
        'huey_class': 'huey.RedisHuey',  
        'name': 'first_tasks',  
        'consumer': {
            'workers': 2,
            'worker_type': 'thread',
        },
    },
    'emails': {#this name will be used in decorators below
        'huey_class': 'huey.RedisHuey',  
        'name': 'emails_tasks',  
        'consumer': {
            'workers': 5,
            'worker_type': 'thread',
        },
    }
}

Note: This setting is incompatible with HUEY setting.

Usage

Now you will be able to run multiple queues using:

python manage.py run_djangohuey --queue first
python manage.py run_djangohuey --queue emails

Each queue must be run in a different terminal.

Configuring tasks

You can use usual huey decorators to register tasks, but they must be imported from django_huey as shown below:

from django_huey import db_task, task

@db_task(queue='first')
	# perform some db task

@task(queue='emails')
	# send some emails

All the args and kwargs defined in huey decorators should work in the same way, if not, let us know.

You might also like...
Bootstrap 3 integration with Django.

django-bootstrap3 Bootstrap 3 integration for Django. Goal The goal of this project is to seamlessly blend Django and Bootstrap 3. Want to use Bootstr

Bootstrap 4 integration with Django.

django-bootstrap 4 Bootstrap 4 integration for Django. Goal The goal of this project is to seamlessly blend Django and Bootstrap 4. Requirements Pytho

Plug and play continuous integration with django and jenkins

django-jenkins Plug and play continuous integration with Django and Jenkins Installation From PyPI: $ pip install django-jenkins Or by downloading th

Django admin CKEditor integration.

Django CKEditor NOTICE: django-ckeditor 5 has backward incompatible code moves against 4.5.1. File upload support has been moved to ckeditor_uploader.

TinyMCE integration for Django

django-tinymce django-tinymce is a Django application that contains a widget to render a form field as a TinyMCE editor. Quickstart Install django-tin

Bootstrap 3 integration with Django.

django-bootstrap3 Bootstrap 3 integration for Django. Goal The goal of this project is to seamlessly blend Django and Bootstrap 3. Want to use Bootstr

Django + Next.js integration

Django Next.js Django + Next.js integration From a comment on StackOverflow: Run 2 ports on the same server. One for django (public facing) and one fo

This is a repository for collecting global custom management extensions for the Django Framework.

Django Extensions Django Extensions is a collection of custom extensions for the Django Framework. Getting Started The easiest way to figure out what

File and Image Management Application for django
File and Image Management Application for django

Django Filer django Filer is a file management application for django that makes handling of files and images a breeze. Contributing This is a an open

Comments
  • HueyException <task> not found in TaskRegistry

    HueyException not found in TaskRegistry

    When using django-huey, the consumer is able to correctly find the registered tasks, however when running the Django server and trying to enqueue a task, I get an "HueyException not found in TaskRegistry" error. From doing wayy too much debugging, I finally tracked this down to the fact that the huey package initializes itself with RegistryA, and then django_huey initializes itself with a separate RegistryB. So when trying to enqueue a task, it attempts to find the task in RegistryA which is empty, because all of the tasks are in RegistryB.

    I use import django_huey as huey everywhere, and then use @huey.task(queue="test"), but I'm still getting this error. I'm not sure what I'm doing wrong ๐Ÿ˜ž

    opened by wgordon17 7
  • django-huey-monitor not working with djangohuey

    django-huey-monitor not working with djangohuey

    I use djangohuey --queue [name of queue] to run consumer, this runs my task, but the django-huey-monitor does not track the tasks, or are not created in the huey_monitor database tables. Which configuration am I missing?

    opened by EvelynArinaitwe 2
  • Unable to use `django-huey` with `SqliteHuey`

    Unable to use `django-huey` with `SqliteHuey`

    Settings:

    DJANGO_HUEY = {
        'default': 'xxx',
        'queues': {
            'xxx': {
                'huey_class': 'huey.SqliteHuey',
                'name': 'xxx_tasks',
                'consumer': {
                    'workers': 4,
                    'worker_type': 'thread',
                },
            },
            'yyy': { 
                'huey_class': 'huey.SqliteHuey',
                'name': 'yyy_tasks',
                'consumer': {
                    'workers': 2,
                    'worker_type': 'thread',
                },
            }
        }
    }
    

    Exception raised:

    huey.exceptions.ConfigurationError: "redis" python module not found, cannot use Redis storage backend. Run "pip install redis" to install.
    

    redis should be optional.

    opened by fabiocaccamo 2
  • Cannot configure djangohuey with different workers.

    Cannot configure djangohuey with different workers.

    Is there a way to configure the djangohuey settings, so that a specific worker is assigned to a specific queue? At the moment, I implement different queues with different number of workers in each consumer, but anytime a worker is free, it is taken on by any of the queue tasks that are waiting. what I want is for example: worker1 listen to only queue_a and worker2&worker3 listen to queue_b, regardless of which worker is free. Is this possible? Here are my settings:

    DJANGO_HUEY = {
        'default': 'first', #this name must match with any of the queues defined below.
        'queues': {
            'queue_a': {
                'huey_class': 'huey.RedisHuey',
                'name': 'tasks_a',
                "immediate": False,
                'results': True,
                'blocking': True,
                'connection': {
                        'host': 'localhost',
                        'port': 6379,
                        'db': 0,
                        'connection_pool': None,
                        'read_timeout': 1,
                        'url': None,  
                    },
                'consumer': {
                    'workers': 1,
                    'worker_type': 'thread',
                },
            },
            'queue_b': {
                'huey_class': 'huey.RedisHuey',
                'name': 'tasks_b',
                "immediate": False,
                'blocking': True,
                
                'consumer': {
                    'workers': 2,
                    'worker_type': 'thread',
                },
            },
        }
    }
    
    opened by EvelynArinaitwe 1
Releases(v1.1.1)
  • v1.1.1(Feb 7, 2022)

  • v1.1.0(Jan 19, 2022)

  • v1.0.1(Jan 14, 2022)

    1.0.1 - 2022-01-14

    Added

    • Close db connections before task body. https://github.com/coleifer/huey/commit/e77acf307bfdade914ab7f91c65dbbc183af5d8f
    Source code(tar.gz)
    Source code(zip)
  • v1.0.0(May 19, 2021)

    1.0.0 - 2021-05-19

    Note: This release contains breaking changes, see them below with the migration instructions.

    Added

    • Allow definition of a default queue.

    Changed

    • HUEYS django setting renamed to DJANGO_HUEY.
    • Change command run_djangohuey to djangohuey.
    Source code(tar.gz)
    Source code(zip)
  • v0.2.0(Apr 18, 2021)

    0.2.0 - 2021-04-18

    Added

    Nothing added this release

    Changed

    Nothing changed this release

    Fixed

    • When a huey name was not provided, default django db name was used. Now it's defaulted to queue name.

    Removed

    • Removed incompatibility with HUEY setting used by huey project.
    Source code(tar.gz)
    Source code(zip)
  • v0.1.1(Apr 4, 2021)

Owner
GAIA Software
Cooperativa de Desarrollo de Software | Quilmes, Buenos Aires, Argentina
GAIA Software
a little task queue for python

a lightweight alternative. huey is: a task queue (2019-04-01: version 2.0 released) written in python (2.7+, 3.4+) clean and simple API redis, sqlite,

Charles Leifer 4.3k Dec 29, 2022
Store events and publish to Kafka

Create an event from Django ORM object model, store the event into the database and also publish it into Kafka cluster.

Diag 6 Nov 30, 2022
๐Ÿ“ Sticky Notes in Django admin

django-admin-sticky-notes Share notes between superusers. Installation Install via pip: pip install django_admin_sticky_notes Put django_admin_sticky_

Dariusz Choruลผy 7 Oct 06, 2021
Probably the best abstract model / admin for your tree based stuff.

django-treenode Probably the best abstract model / admin for your tree based stuff. Features Fast - get ancestors, children, descendants, parent, root

Fabio Caccamo 360 Jan 05, 2023
Run Django tests with testcontainers.

django-rdtwt (Run Django Tests With Testcontainers) This targets users who wish to forget setting up a database for tests. There's no manually startin

2 Jan 09, 2022
A handy tool for generating Django-based backend projects without coding. On the other hand, it is a code generator of the Django framework.

Django Sage Painless The django-sage-painless is a valuable package based on Django Web Framework & Django Rest Framework for high-level and rapid web

sageteam 51 Sep 15, 2022
Comprehensive Markdown plugin built for Django

Django MarkdownX Django MarkdownX is a comprehensive Markdown plugin built for Django, the renowned high-level Python web framework, with flexibility,

neutronX 738 Dec 21, 2022
django app that allows capture application metrics by each user individually

Django User Metrics django app that allows capture application metrics by each user individually, so after you can generate reports with aggregation o

Reiner Marquez 42 Apr 28, 2022
A debug/profiling overlay for Django

Django Debug Toolbar The Django Debug Toolbar is a configurable set of panels that display various debug information about the current request/respons

David Cramer 228 Oct 17, 2022
An example of Django project with basic user functionality and account activation.

Simple Django Login and Registration An example of Django project with basic user functionality. Screenshots Log In Create an account Authorized page

Hussein Sarea 3 Oct 19, 2022
Django admin CKEditor integration.

Django CKEditor NOTICE: django-ckeditor 5 has backward incompatible code moves against 4.5.1. File upload support has been moved to ckeditor_uploader.

2.2k Dec 31, 2022
Extensions for using Rich with Django.

django-rich Extensions for using Rich with Django. Requirements Python 3.6 to 3.10 supported. Django 2.2 to 4.0 supported. Are your tests slow? Check

Adam Johnson 88 Dec 26, 2022
A BitField extension for Django Models

django-bitfield Provides a BitField like class (using a BigIntegerField) for your Django models. (If you're upgrading from a version before 1.2 the AP

DISQUS 361 Dec 22, 2022
A helper for organizing Django project settings by relying on well established programming patterns.

django-configurations django-configurations eases Django project configuration by relying on the composability of Python classes. It extends the notio

Jazzband 953 Dec 29, 2022
Dockerizing Django with Postgres, Gunicorn, Nginx and Certbot. A fully Django starter project.

Dockerizing Django with Postgres, Gunicorn, Nginx and Certbot ๐Ÿš€ Features A Django stater project with fully basic requirements for a production-ready

8 Jun 27, 2022
Displaying objects on maps in the Django views and administration site.

DjangoAdminGeomap library The free, open-source DjangoAdminGeomap library is designed to display objects on the map in the Django views and admin site

Vitaly Bogomolov 31 Dec 28, 2022
A better and faster multiple selection widget with suggestions

django-searchable-select A better and faster multiple selection widget with suggestions for Django This project is looking for maintainers! Please ope

Andrew Dunai 105 Oct 22, 2022
A Django web application to receive, virus check and validate transfers of digital archival records, and allow archivists to appraise and accession those records.

Aurora Aurora is a Django web application that can receive, virus check and validate transfers of digital archival records, and allows archivists to a

Rockefeller Archive Center 20 Aug 30, 2022
A django model and form field for normalised phone numbers using python-phonenumbers

django-phonenumber-field A Django library which interfaces with python-phonenumbers to validate, pretty print and convert phone numbers. python-phonen

Stefan Foulis 1.3k Dec 31, 2022
GeoDjango provides geospatial extensions to the Django web dev framework

Django is a high-level Python Web framework that encourages rapid development and clean, pragmatic design. All documentation is in the "docs" directo

Paul Smith 20 Sep 20, 2022