A reusable Django model field for storing ad-hoc JSON data

Overview

jsonfield

https://circleci.com/gh/rpkilby/jsonfield.svg?style=shield

jsonfield is a reusable model field that allows you to store validated JSON, automatically handling serialization to and from the database. To use, add jsonfield.JSONField to one of your models.

Note: django.contrib.postgres now supports PostgreSQL's jsonb type, which includes extended querying capabilities. If you're an end user of PostgreSQL and want full-featured JSON support, then it is recommended that you use the built-in JSONField. However, jsonfield is still useful when your app needs to be database-agnostic, or when the built-in JSONField's extended querying is not being leveraged. e.g., a configuration field.

Requirements

jsonfield aims to support all current versions of Django, however the explicity tested versions are:

  • Python: 3.6, 3.7, 3.8
  • Django: 2.2, 3.0

Installation

pip install jsonfield

Usage

from django.db import models
from jsonfield import JSONField

class MyModel(models.Model):
    json = JSONField()

Querying

As stated above, JSONField is not intended to provide extended querying capabilities. That said, you may perform the same basic lookups provided by regular text fields (e.g., exact or regex lookups). Since values are stored as serialized JSON, it is highly recommended that you test your queries to ensure the expected results are returned.

Handling null values

A model field's null argument typically controls whether null values may be stored in its column by setting a not-null constraint. However, because JSONField serializes its values (including nulls), this option instead controls how null values are persisted. If null=True, then nulls are not serialized and are stored as a null value in the database. If null=False, then the null is instead stored in its serialized form.

This in turn affects how null values may be queried. Both fields support exact matching:

MyModel.objects.filter(json=None)

However, if you want to use the isnull lookup, you must set null=True.

class MyModel(models.Model):
    json = JSONField(null=True)

MyModel.objects.filter(json__isnull=True)

Note that as JSONField.null does not prevent nulls from being stored, achieving this must instead be handled with a validator.

Advanced Usage

By default python deserializes json into dict objects. This behavior differs from the standard json behavior because python dicts do not have ordered keys. To overcome this limitation and keep the sort order of OrderedDict keys the deserialisation can be adjusted on model initialisation:

import collections

class MyModel(models.Model):
    json = JSONField(load_kwargs={'object_pairs_hook': collections.OrderedDict})

Other Fields

jsonfield.JSONCharField

Subclasses models.CharField instead of models.TextField.

Running the tests

The test suite requires tox.

$ pip install tox

Then, run the tox command, which will run all test jobs.

$ tox

Or, to test just one job (for example Django 2.0 on Python 3.6):

$ tox -e py36-django20

Release Process

  • Update changelog
  • Update package version in setup.py
  • Check supported versions in setup.py and readme
  • Create git tag for version
  • Upload release to PyPI test server
  • Upload release to official PyPI server
$ pip install -U pip setuptools wheel twine
$ rm -rf dist/ build/
$ python setup.py sdist bdist_wheel
$ twine upload -r test dist/*
$ twine upload dist/*

Changes

Take a look at the changelog.

Comments
  • Problems withs PostGres 9.3, JSON Field and Query Distinct

    Problems withs PostGres 9.3, JSON Field and Query Distinct

    As mentionned in #47, using ".distinct()", which is used by the admin panel creating users but also in several other cases in django processes, triggers the same bug as #47 describes.

    A workaround has been found by @mkhattab : The work around for this in the Django admin is to subclass QuerySet and Manager and override the distinct method [to remove the JSON fields from it].

    What would be the definite fix ?

    opened by ewjoachim 20
  • Can not store ordinary Python strings in JSONfield

    Can not store ordinary Python strings in JSONfield

    Since version 0.9.4 I am not able to store ordinary Python strings in a JSONfield. Up to version 0.9.2 this worked without problems. Did I miss something? From my point of view, it should not matter, if I store ordinary strings, lists or dicts in a JSONfield.

    opened by jrief 20
  • AttributeError in Creator class

    AttributeError in Creator class

    I updated to Django 1.11 and I get AttributeError from subclassing.py:33.

    In Django 1.11 (django/db/models/options.py:890) the get method of the Creator class is implicitly called. The code in subclassing.py is a copy/paste from Django and seems to be <1.7 and Django's code has updated since then. Simply copying the code from Django 1.7 seems to solve my problems.

    opened by Majsvaffla 16
  • JSONField not deserialized when used from inside a polymorphic inherited model

    JSONField not deserialized when used from inside a polymorphic inherited model

    Hi,

    I have models like this:

    class BaseFeed(PolymorphicModel):
        errors = JSONField(default=list, blank=True)
    
    
    class RssAtomFeed(BaseFeed):
        pass
    

    Note: polymorphic is here. Say I do:

    f1 = BaseFeed.objects.create()
    f2 = RssAtomFeed.objects.create()
    

    Then I got:

    f1.errors
    []
    
    f2.errors
    u'[]'
    

    And this is pretty annoying because it will obviously fail in many places because all my code expects a list.

    I have read #92, but I didn't find any obvious/easy clue on how to solve this issue.

    Any other hint I can try?

    For the record, my models are much complex than this. You can view the base feed here and the rss/atom feed here. I've tried with or without default, with or without load_kwargs={'object_pairs_hook': collections.OrderedDict}, this doesn't change anything.

    I've tried the other JSONField implementation and the problem doesn't occur, even after a south migration (I'm on Django 1.6).

    regards,

    opened by Karmak23 11
  • Problem when iterating if default value is '

    Problem when iterating if default value is '""' on postgresql-9.3

    In [1]: from my_app.models import MyModel

    In [2]: [fo for fo in MyModel.objects.all()]

    ValidationError Traceback (most recent call last) in () ----> 1 [fo for fo in MyModel.objects.all()]

    /home/florent/.virtualenvs//local/lib/python2.7/site-packages/django/db/models/query.pyc in iter(self) 94 - Responsible for turning the rows into model objects. 95 """ ---> 96 self._fetch_all() 97 return iter(self._result_cache) 98

    /home/florent/.virtualenvs//local/lib/python2.7/site-packages/django/db/models/query.pyc in _fetch_all(self) 852 def _fetch_all(self): 853 if self._result_cache is None: --> 854 self._result_cache = list(self.iterator()) 855 if self._prefetch_related_lookups and not self._prefetch_done: 856 self._prefetch_related_objects()

    /home/florent/.virtualenvs//local/lib/python2.7/site-packages/django/db/models/query.pyc in iterator(self) 228 obj = model_cls(*_dict(zip(init_list, row_data))) 229 else: --> 230 obj = model(_row_data) 231 232 # Store the source database of the object

    /home/florent/.virtualenvs//local/lib/python2.7/site-packages/django/db/models/base.pyc in init(self, _args, *_kwargs) 345 # without changing the logic. 346 for val, field in zip(args, fields_iter): --> 347 setattr(self, field.attname, val) 348 else: 349 # Slower, kwargs-ready version.

    /home/florent/Work//django-jsonfield/jsonfield/subclassing.pyc in set(self, obj, value) 39 # we can definitively tell if a value has already been deserialized 40 # More: https://github.com/bradjasper/django-jsonfield/issues/33 ---> 41 obj.dict[self.field.name] = self.field.pre_init(value, obj) 42 43

    /home/florent/Work//django-jsonfield/jsonfield/fields.pyc in pre_init(self, value, obj) 75 return json.loads(value, **self.load_kwargs) 76 except ValueError: ---> 77 raise ValidationError(_("Enter valid JSON")) 78 79 return value

    ValidationError: [u'Enter valid JSON']

    opened by toopy 11
  • Serialize ''(empty) string to empty database value

    Serialize ''(empty) string to empty database value

    Uses ""(empty) string in database to store empty json ([]).

    You can use empty value as default for your table fields. You will have no overhead in database if you have no data.

    opened by Romamo 11
  • empty string or null in db

    empty string or null in db

    Hello,

    Wouldn't it be nice to gracefully handle empty string or NULL when reading from db? It could default to empty dictionary: {}.

    I've added the field to my model manually to the DB table and it did not work with empty string I had to update my_table set settings='{}'.

    I know I can set the '{}' via the south migrations, but it would still be a nice default behavior.

    opened by adrianandreias 9
  • DBError: could not identify an equality operator for type json when annotating a model with JSONField

    DBError: could not identify an equality operator for type json when annotating a model with JSONField

    I'm working in Django 1.5.4 with PostgreSQL 9.3, and i get this error on a query like this:

    ModelWithJsonField.objects.annotate(num=Count('field_to_count_by'))
    

    Traceback:

    Traceback (most recent call last):
      File "<console>", line 1, in <module>
      File "/Users/jondoe/Project/lib/python2.7/site-packages/django/db/models/query.py", line 93, in __repr__
        data = list(self[:REPR_OUTPUT_SIZE + 1])
      File "/Users/jondoe/Project/lib/python2.7/site-packages/django/db/models/query.py", line 108, in __len__
        self._result_cache.extend(self._iter)
      File "/Users/jondoe/Project/lib/python2.7/site-packages/django/db/models/query.py", line 317, in iterator
        for row in compiler.results_iter():
      File "/Users/jondoe/Project/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 775, in results_iter
        for rows in self.execute_sql(MULTI):
      File "/Users/jondoe/Project/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 840, in execute_sql
        cursor.execute(sql, params)
      File "/Users/jondoe/Project/lib/python2.7/site-packages/django/db/backends/util.py", line 41, in execute
        return self.cursor.execute(sql, params)
      File "/Users/jondoe/Project/lib/python2.7/site-packages/django/db/backends/postgresql_psycopg2/base.py", line 58, in execute
        six.reraise(utils.DatabaseError, utils.DatabaseError(*tuple(e.args)), sys.exc_info()[2])
      File "/Users/jondoe/Project/lib/python2.7/site-packages/django/db/backends/postgresql_psycopg2/base.py", line 54, in execute
        return self.cursor.execute(query, args)
    DatabaseError: could not identify an equality operator for type json
    

    Ideas? I saw another issue with similar error, going to try and find out what causes this.

    opened by neara 9
  • JSONField fails in MySQL

    JSONField fails in MySQL

    I recently added the JSONField to one of my Models and worked fine since both my local machine runs on Postgres and my Heroku instance too, since moving it over to production I have found that everytime I try and bring up an item containg a default value for MySQL it raises a validationerror.

    class Item(models.Model): ... ... json = JSONField(default='{}')

    raise ValidationError(msg) ValidationError: [u"'' is not a valid JSON string."]

    opened by appel268576 9
  • Issues between versions

    Issues between versions

    Hi,

    We have recently updated from jsonfield2 to jsonfield and we are experiencing issues when we have a dict data being passed in get_or_create().

    The idea is we were using jsonfield2, jsonfield2==3.0.2 and when we update to latest jsonfield==3.1.0 the issue appears. In order to reproduce it you need to have a model lets say:

    class MyModel:
        field_1 = models.CharField
        field_2 = JsonField()
    
       Meta:
          unique(field_1)
    
    MyModel.objects.create(field_1='test', field_2={'foo':'bar'})
    

    And while still on jsonfiield2==3.0.2 version create an entry in the database ( we use postgres ).

    Now uninstall jsonfield2 and install latest jsonfield and try to do

    MyModel.objects.get_or_create(field_1='test', field_2={'foo':'bar'}) - It will fail with IntegrityError and you will get "duplicate key value violates unique constraint" message. This is because if you just call MyModel.objects.get(field_1='test', field_2={'foo':'bar'}) - it raises ObjectDoesNotExist, and then the above tries to call create but there is already a entry in the database with those params and there is a unique constraint in our case.

    The most important thing is that this MyModel.objects.get(field_1='test', field_2={'foo':'bar'}) raises ObjectDoesNotExist when the entry clearly exists in the table

    opened by ghost 8
  • Support polymorphic objects

    Support polymorphic objects

    Fix polymorphic object issue: https://github.com/dmkoch/django-jsonfield/issues/101

    With polymorphic objects I get a AttributeError on obj.pk. obj.id works fine. pk and id has in the tests always the same value.

    opened by ubaumann 8
  • Don't autofill fields with null

    Don't autofill fields with null

    After 92613991d76429c65bd35aebea3470c0baf26520, a blank field is automatically populated with null. This is bad for null=True fields, since those should be left empty when unpopulated. This PR preserves the behaviour from before.

    opened by quantum5 0
  • Update tests

    Update tests

    I was testing some PR that you have for jsonfield

    https://github.com/rpkilby/jsonfield/pull/262 This fixes the warnings and the name of a file that causes the circle ci test to fail.

    https://github.com/rpkilby/jsonfield/pull/261 And this one adds test for django 3.1 and 3.2

    I think it would be nice to add them to the main branch. I tried both PR together and work fine (https://app.circleci.com/pipelines/github/MaferMazu/jsonfield?branch=mfmz%2Ffix-ci-and-test-django3.2)

    Hopefully you can add them so that the library remains updated.

    Any doubt or advance I am at your service.

    Thanks for your consideration.

    opened by MaferMazu 0
  • Filter warnings from Django during capture.

    Filter warnings from Django during capture.

    Django has warnings in asyncio that are also captured in the tests. Filter warnings from django.utils.asyncio module. This was fixed in Django but not yet released : https://github.com/django/django/commit/623c8cd8f41a99f22d39b264f7eaf7244417000b .

    Fedora issue: https://bugzilla.redhat.com/show_bug.cgi?id=1962449

    opened by tirkarthi 1
  • docs: fix simple typo, explicity -> explicitly

    docs: fix simple typo, explicity -> explicitly

    There is a small typo in README.rst.

    Should read explicitly rather than explicity.

    Semi-automated pull request generated by https://github.com/timgates42/meticulous/blob/master/docs/NOTE.md

    opened by timgates42 0
  • Not enough space to persist data

    Not enough space to persist data

    I was trying to save a big json object but i couldn't accomplish it since this field generate a Long type field on my Oracle DB. there is a solution to do it or it is neccesary apply a change.

    opened by jasonlazo 0
Owner
Ryan P Kilby
beep boop
Ryan P Kilby
No effort, no worry, maximum performance.

Django Cachalot Caches your Django ORM queries and automatically invalidates them. Documentation: http://django-cachalot.readthedocs.io Table of Conte

NoriPyt 980 Jan 06, 2023
Logan is a toolkit for building standalone Django applications

Logan Logan is a toolkit for running standalone Django applications. It provides you with tools to create a CLI runner, manage settings, and the abili

David Cramer 206 Jan 03, 2023
Dynamic, database-driven Django forms

Django Dataforms django-dataforms is a wrapper for the Django forms API that lets you dynamically define forms in a database, rather than hard-coding

35 Dec 16, 2022
User Authentication In Django/Ajax/Jquery

User Authentication In Django/Ajax/Jquery Demo: Authentication System Using Django/Ajax/Jquery Demo: Authentication System Using Django Overview The D

Suman Raj Khanal 10 Mar 26, 2022
Meta package to combine turbo-django and stimulus-django

Hotwire + Django This repository aims to help you integrate Hotwire with Django 🚀 Inspiration might be taken from @hotwired/hotwire-rails. We are sti

Hotwire for Django 31 Aug 09, 2022
Simple reproduction of connection leak with celery/django/gevent

Redis connection leak with celery/django/gevent Reproduces celery issue at https://github.com/celery/celery/issues/6819 using gevented django web serv

2 Apr 03, 2022
PWA is a simple Django app to develope and deploy a Progressive Web Application.

PWA PWA is a simple Django app to develope and deploy a Progressive Web Application. Detailed documentation is in the "docs" directory. Quick start Ad

Nima 6 Dec 09, 2022
Utilities to make function-based views cleaner, more efficient, and better tasting.

django-fbv Utilities to make Django function-based views cleaner, more efficient, and better tasting. 💥 📖 Complete documentation: https://django-fbv

Adam Hill 49 Dec 30, 2022
Simple XML-RPC and JSON-RPC server for modern Django

django-modern-rpc Build an XML-RPC and/or JSON-RPC server as part of your Django project. Major Django and Python versions are supported Main features

Antoine Lorence 82 Dec 04, 2022
A collection of models, views, middlewares, and forms to help secure a Django project.

Django-Security This package offers a number of models, views, middlewares and forms to facilitate security hardening of Django applications. Full doc

SD Elements 258 Jan 03, 2023
A calendaring app for Django. It is now stable, Please feel free to use it now. Active development has been taken over by bartekgorny.

Django-schedule A calendaring/scheduling application, featuring: one-time and recurring events calendar exceptions (occurrences changed or cancelled)

Tony Hauber 814 Dec 26, 2022
Актуальный сборник шаблонов для создания проектов и приложений на Django

О чем этот проект Этот репозиторий с шаблонами для быстрого создания Django проекта. В шаблоне проекта настроены следующий технологий: Django gunicorn

Denis Kustov 16 Oct 20, 2022
Backend with Django .

BackendCode - Cookies Documentation: https://docs.djangoproject.com/fr/3.2/intro/ By @tcotidiane33 & @yaya Models Premium class Pack(models.Model): n

just to do it 1 Jan 28, 2022
Run Django tests with testcontainers.

django-rdtwt (Run Django Tests With Testcontainers) This targets users who wish to forget setting up a database for tests. There's no manually startin

2 Jan 09, 2022
Management commands to help backup and restore your project database and media files

Django Database Backup This Django application provides management commands to help backup and restore your project database and media files with vari

687 Jan 04, 2023
Django Federated Login provides an authentication bridge between Django projects and OpenID-enabled identity providers.

Django Federated Login Django Federated Login provides an authentication bridge between Django projects and OpenID-enabled identity providers. The bri

Bouke Haarsma 18 Dec 29, 2020
Money fields for Django forms and models.

django-money A little Django app that uses py-moneyed to add support for Money fields in your models and forms. Django versions supported: 1.11, 2.1,

1.4k Jan 06, 2023
Login System Django

Login-System-Django Login System Using Django Tech Used Django Python Html Run Locally Clone project git clone https://link-to-project Get project for

Nandini Chhajed 6 Dec 12, 2021
An extremely fast JavaScript and CSS bundler and minifier

Website | Getting started | Documentation | Plugins | FAQ Why? Our current build tools for the web are 10-100x slower than they could be: The main goa

Evan Wallace 34.2k Jan 04, 2023
Use heroicons in your Django and Jinja templates.

heroicons Use heroicons in your Django and Jinja templates. Requirements Python 3.6 to 3.9 supported. Django 2.2 to 3.2 supported. Are your tests slow

Adam Johnson 52 Dec 14, 2022