A django integration for huey task queue that supports multi queue management

Related tags

Djangodjango-huey
Overview

Version

django-huey

This package is an extension of huey contrib djhuey package that allows users to manage multiple queues.

Installation

Using pip package manager run:

# pip install Django  if not installed
# pip install huey    if not installed
pip install django-huey

Note: use a virtualenv to isolate your dependencies. Note 2: django and huey must be installed.

Then, in your settings.py file add django_huey to the INSTALLED_APPS:

INSTALLED_APPS = [
	...
    'django_huey',
]

Configuration

In settings.py you must add the HUEYS setting:

HUEYS = {
    'first': {#this name will be used in decorators below
        'huey_class': 'huey.RedisHuey',  
        'name': 'first_tasks',  
        'consumer': {
            'workers': 2,
            'worker_type': 'thread',
        },
    },
    'emails': {#this name will be used in decorators below
        'huey_class': 'huey.RedisHuey',  
        'name': 'emails_tasks',  
        'consumer': {
            'workers': 5,
            'worker_type': 'thread',
        },
    }
}

Note: This setting is incompatible with HUEY setting.

Usage

Now you will be able to run multiple queues using:

python manage.py run_djangohuey --queue first
python manage.py run_djangohuey --queue emails

Each queue must be run in a different terminal.

Configuring tasks

You can use usual huey decorators to register tasks, but they must be imported from django_huey as shown below:

from django_huey import db_task, task

@db_task(queue='first')
	# perform some db task

@task(queue='emails')
	# send some emails

All the args and kwargs defined in huey decorators should work in the same way, if not, let us know.

You might also like...
Bootstrap 3 integration with Django.

django-bootstrap3 Bootstrap 3 integration for Django. Goal The goal of this project is to seamlessly blend Django and Bootstrap 3. Want to use Bootstr

Bootstrap 4 integration with Django.

django-bootstrap 4 Bootstrap 4 integration for Django. Goal The goal of this project is to seamlessly blend Django and Bootstrap 4. Requirements Pytho

Plug and play continuous integration with django and jenkins

django-jenkins Plug and play continuous integration with Django and Jenkins Installation From PyPI: $ pip install django-jenkins Or by downloading th

Django admin CKEditor integration.

Django CKEditor NOTICE: django-ckeditor 5 has backward incompatible code moves against 4.5.1. File upload support has been moved to ckeditor_uploader.

TinyMCE integration for Django

django-tinymce django-tinymce is a Django application that contains a widget to render a form field as a TinyMCE editor. Quickstart Install django-tin

Bootstrap 3 integration with Django.

django-bootstrap3 Bootstrap 3 integration for Django. Goal The goal of this project is to seamlessly blend Django and Bootstrap 3. Want to use Bootstr

Django + Next.js integration

Django Next.js Django + Next.js integration From a comment on StackOverflow: Run 2 ports on the same server. One for django (public facing) and one fo

This is a repository for collecting global custom management extensions for the Django Framework.

Django Extensions Django Extensions is a collection of custom extensions for the Django Framework. Getting Started The easiest way to figure out what

File and Image Management Application for django
File and Image Management Application for django

Django Filer django Filer is a file management application for django that makes handling of files and images a breeze. Contributing This is a an open

Comments
  • HueyException <task> not found in TaskRegistry

    HueyException not found in TaskRegistry

    When using django-huey, the consumer is able to correctly find the registered tasks, however when running the Django server and trying to enqueue a task, I get an "HueyException not found in TaskRegistry" error. From doing wayy too much debugging, I finally tracked this down to the fact that the huey package initializes itself with RegistryA, and then django_huey initializes itself with a separate RegistryB. So when trying to enqueue a task, it attempts to find the task in RegistryA which is empty, because all of the tasks are in RegistryB.

    I use import django_huey as huey everywhere, and then use @huey.task(queue="test"), but I'm still getting this error. I'm not sure what I'm doing wrong 😞

    opened by wgordon17 7
  • django-huey-monitor not working with djangohuey

    django-huey-monitor not working with djangohuey

    I use djangohuey --queue [name of queue] to run consumer, this runs my task, but the django-huey-monitor does not track the tasks, or are not created in the huey_monitor database tables. Which configuration am I missing?

    opened by EvelynArinaitwe 2
  • Unable to use `django-huey` with `SqliteHuey`

    Unable to use `django-huey` with `SqliteHuey`

    Settings:

    DJANGO_HUEY = {
        'default': 'xxx',
        'queues': {
            'xxx': {
                'huey_class': 'huey.SqliteHuey',
                'name': 'xxx_tasks',
                'consumer': {
                    'workers': 4,
                    'worker_type': 'thread',
                },
            },
            'yyy': { 
                'huey_class': 'huey.SqliteHuey',
                'name': 'yyy_tasks',
                'consumer': {
                    'workers': 2,
                    'worker_type': 'thread',
                },
            }
        }
    }
    

    Exception raised:

    huey.exceptions.ConfigurationError: "redis" python module not found, cannot use Redis storage backend. Run "pip install redis" to install.
    

    redis should be optional.

    opened by fabiocaccamo 2
  • Cannot configure djangohuey with different workers.

    Cannot configure djangohuey with different workers.

    Is there a way to configure the djangohuey settings, so that a specific worker is assigned to a specific queue? At the moment, I implement different queues with different number of workers in each consumer, but anytime a worker is free, it is taken on by any of the queue tasks that are waiting. what I want is for example: worker1 listen to only queue_a and worker2&worker3 listen to queue_b, regardless of which worker is free. Is this possible? Here are my settings:

    DJANGO_HUEY = {
        'default': 'first', #this name must match with any of the queues defined below.
        'queues': {
            'queue_a': {
                'huey_class': 'huey.RedisHuey',
                'name': 'tasks_a',
                "immediate": False,
                'results': True,
                'blocking': True,
                'connection': {
                        'host': 'localhost',
                        'port': 6379,
                        'db': 0,
                        'connection_pool': None,
                        'read_timeout': 1,
                        'url': None,  
                    },
                'consumer': {
                    'workers': 1,
                    'worker_type': 'thread',
                },
            },
            'queue_b': {
                'huey_class': 'huey.RedisHuey',
                'name': 'tasks_b',
                "immediate": False,
                'blocking': True,
                
                'consumer': {
                    'workers': 2,
                    'worker_type': 'thread',
                },
            },
        }
    }
    
    opened by EvelynArinaitwe 1
Releases(v1.1.1)
  • v1.1.1(Feb 7, 2022)

  • v1.1.0(Jan 19, 2022)

  • v1.0.1(Jan 14, 2022)

    1.0.1 - 2022-01-14

    Added

    • Close db connections before task body. https://github.com/coleifer/huey/commit/e77acf307bfdade914ab7f91c65dbbc183af5d8f
    Source code(tar.gz)
    Source code(zip)
  • v1.0.0(May 19, 2021)

    1.0.0 - 2021-05-19

    Note: This release contains breaking changes, see them below with the migration instructions.

    Added

    • Allow definition of a default queue.

    Changed

    • HUEYS django setting renamed to DJANGO_HUEY.
    • Change command run_djangohuey to djangohuey.
    Source code(tar.gz)
    Source code(zip)
  • v0.2.0(Apr 18, 2021)

    0.2.0 - 2021-04-18

    Added

    Nothing added this release

    Changed

    Nothing changed this release

    Fixed

    • When a huey name was not provided, default django db name was used. Now it's defaulted to queue name.

    Removed

    • Removed incompatibility with HUEY setting used by huey project.
    Source code(tar.gz)
    Source code(zip)
  • v0.1.1(Apr 4, 2021)

Owner
GAIA Software
Cooperativa de Desarrollo de Software | Quilmes, Buenos Aires, Argentina
GAIA Software
An extremely fast JavaScript and CSS bundler and minifier

Website | Getting started | Documentation | Plugins | FAQ Why? Our current build tools for the web are 10-100x slower than they could be: The main goa

Evan Wallace 34.2k Jan 04, 2023
MAC address Model Field & Form Field for Django apps

django-macaddress MAC Address model and form fields for Django We use netaddr to parse and validate the MAC address. The tests aren't complete yet. Pa

49 Sep 04, 2022
📝 Sticky Notes in Django admin

django-admin-sticky-notes Share notes between superusers. Installation Install via pip: pip install django_admin_sticky_notes Put django_admin_sticky_

Dariusz Choruży 7 Oct 06, 2021
A beginner django project and also my first Django project which involves shortening of a longer URL into a short one using a unique id.

Django-URL-Shortener A beginner django project and also my first Django project which involves shortening of a longer URL into a short one using a uni

Rohini Rao 3 Aug 08, 2021
Media-Management with Grappelli

Django FileBrowser Media-Management with Grappelli. The FileBrowser is an extension to the Django administration interface in order to: browse directo

Patrick Kranzlmueller 913 Dec 28, 2022
Bootstrap 3 integration with Django.

django-bootstrap3 Bootstrap 3 integration for Django. Goal The goal of this project is to seamlessly blend Django and Bootstrap 3. Want to use Bootstr

Zostera B.V. 2.3k Jan 03, 2023
Basic Form Web Development using Python, Django and CSS

thebookrain Basic Form Web Development using Python, Django and CSS This is a basic project that contains two forms - borrow and donate. The form data

Ananya Dhulipala 1 Nov 27, 2021
A fresh approach to autocomplete implementations, specially for Django. Status: v3 stable, 2.x.x stable, 1.x.x deprecated. Please DO regularely ping us with your link at #yourlabs IRC channel

Features Python 2.7, 3.4, Django 2.0+ support (Django 1.11 (LTS), is supported until django-autocomplete-light-3.2.10), Django (multiple) choice suppo

YourLabs 1.7k Jan 01, 2023
This is a simple Todo web application built Django (back-end) and React JS (front-end)

Django REST Todo app This is a simple Todo web application built with Django (back-end) and React JS (front-end). The project enables you to systemati

Maxim Mukhin 5 May 06, 2022
Coltrane - A simple content site framework that harnesses the power of Django without the hassle.

coltrane A simple content site framework that harnesses the power of Django without the hassle. Features Can be a standalone static site or added to I

Adam Hill 58 Jan 02, 2023
Realworld - Realworld using Django and HTMX

Realworld - Realworld using Django and HTMX

Dan Jacob 53 Jan 05, 2023
Projeto Crud Django and Mongo

Projeto-Crud_Django_and_Mongo Configuração para rodar o projeto Download Project

Samuel Fernandes Oliveira 2 Jan 24, 2022
🏭 An easy-to-use implementation of Creation Methods for Django, backed by Faker.

Django-fakery An easy-to-use implementation of Creation Methods (aka Object Factory) for Django, backed by Faker. django_fakery will try to guess the

Flavio Curella 93 Oct 12, 2022
The Django Leaflet Admin List package provides an admin list view featured by the map and bounding box filter for the geo-based data of the GeoDjango.

The Django Leaflet Admin List package provides an admin list view featured by the map and bounding box filter for the geo-based data of the GeoDjango. It requires a django-leaflet package.

Vsevolod Novikov 33 Nov 11, 2022
System checks for your project's environment.

django-version-checks System checks for your project's environment. Requirements Python 3.6 to 3.9 supported. Django 2.2 to 3.2 supported. Are your te

Adam Johnson 33 Dec 22, 2022
A Django backed for PostgreSQL using Psycopg 3

A Django backend for PostgreSQL using Psycopg 2 The backend passes the entire Django test suite, but it needs a few modifications to Django and to i

Daniele Varrazzo 42 Dec 16, 2022
A Student/ School management application built using Django and Python.

Student Management An awesome student management app built using Django.! Explore the docs » View Demo · Report Bug · Request Feature Table of Content

Nishant Sethi 1 Feb 10, 2022
A starter template for building a backend with Django and django-rest-framework using docker with PostgreSQL as the primary DB.

Django-Rest-Template! This is a basic starter template for a backend project with Django as the server and PostgreSQL as the database. About the templ

Akshat Sharma 11 Dec 06, 2022
Django CRUD REST API Generator

Django CRUD REST API Generator This is a simple tool that generates a Django REST API with the given models. Specs: Authentication, DRF generic views,

Mehmet Alp Sümer 57 Nov 24, 2022
Django Serverless Cron - Run cron jobs easily in a serverless environment

Django Serverless Cron - Run cron jobs easily in a serverless environment

Paul Onteri 41 Dec 16, 2022