A Django app for managing robots.txt files following the robots exclusion protocol

Related tags

Djangodjango-robots
Overview

Django Robots

Jazzband

This is a basic Django application to manage robots.txt files following the robots exclusion protocol, complementing the Django Sitemap contrib app.

For installation instructions, see the documentation install section; for instructions on how to use this application, and on what it provides, see the file "overview.txt" in the "docs/" directory or on ReadTheDocs: https://django-robots.readthedocs.io/

Supported Django versions

  • Django 3.1
  • Django 3.0
  • Django 2.2

For older Django versions (1.6-1.10) use django-robots==3.0. for Django 2 and above, use django-robots>=4.0.0.

Supported Python version

  • Python 3.6, 3.7, 3.8
Comments
  • sitemap not being included in robots.txt

    sitemap not being included in robots.txt

    I have tried both default discovery as well as specifying it manually using the ROBOTS_SITEMAP_URLS setting but I am still unable to get the site map to show up on the robots.txt

    opened by myusuf3 30
  • enable scheme in robots.txt host record

    enable scheme in robots.txt host record

    Just created a new setting to enable HTTPS or HTTP on robots.txt HOST record. Will be great if somebody will review and if everything is ok merge it to the project

    opened by tyapkov 17
  • Sitemap URL with double scheme

    Sitemap URL with double scheme

    I have a setup of Django CMS + Gunicorn + Nginx and when the robots.txt file is displayed, it shows the scheme of the URL twice.

    This is the robots file that is generated:

    User-agent: * Allow: / Disallow: /admin

    Host: https://www.chitwa.co.za Sitemap: https://https://www.chitwa.co.za/en/sitemap.xml

    bug 
    opened by lmanzurv 16
  • add django 3.2 and 4.0 support

    add django 3.2 and 4.0 support

    Add django 3.2 and 4.0 support

    I am facing the following warning while upgrading the django to 3.2 which take the default variable for DEFAULT_AUTO_FIELD ref

    web_1     | robots.Rule: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
    web_1     | 	HINT: Configure the DEFAULT_AUTO_FIELD setting or the AppConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
    web_1     | robots.Url: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
    web_1     | 	HINT: Configure the DEFAULT_AUTO_FIELD setting or the AppConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
    

    So in lieu of that I am facing the error in my CI/CD that is

    [TEST SUITE]./scripts/test/detect_missing_migrations.sh
    Error: one or more migrations are missing
    Migrations for 'robots': /opt/hostedtoolcache/Python/3.7.5/x64/lib/python3.7/site-packages/robots/migrations/0002_auto_20210923_1203.py - Alter field id on rule - Alter field id on url
    
    opened by umarmughal824 10
  • Update pypi with 3.1.0

    Update pypi with 3.1.0

    Hey, It seems like pypi still has 3.0 https://pypi.python.org/pypi/django-robots

    Can you guys also update pypi with 3.1.0 so it would be downlodable with pip?

    opened by gfhuseyinyilmaz 10
  • Feature request: wagtail support

    Feature request: wagtail support

    The current django-robots implementation is dependent on django.contrib.sites. Wagtail uses its own Site model, meaning django-robots currently is not compatible with Wagtail. It would be a nice feature to be able to use django-robots for django applications that cannot make use of the 'sites' framework, such as Wagtail-sites.

    opened by MikeOverkamp 9
  • Format crawl-delay to return an int

    Format crawl-delay to return an int

    Crawl delay is intended as an integer in most specifications (https://en.wikipedia.org/wiki/Robots_exclusion_standard#Crawl-delay_directive), no need to format as float

    opened by yakky 9
  • Django 4 is not supported

    Django 4 is not supported

    The error code is : File "/home/***/***/venv/lib/python3.9/site-packages/robots/models.py", line 4, in from django.utils.translation import ugettext_lazy as _ ImportError: cannot import name 'ugettext_lazy' from 'django.utils.translation' (/home/nour/LPM/venv/lib/python3.9/site-packages/django/utils/

    We must change ugettextlazy to gettextlazy in import section to solve the problem And THX <3 .

    opened by Nour833 7
  • Migration not working

    Migration not working

    Related to #124, a migration file now seems to be included in 5.0, but results (at least on my system) in the following error:

    psycopg2.errors.ObjectNotInPrerequisiteState: sequence must have same owner as table it is linked to

    opened by justin-ven 6
  • Tests no longer passes in older Django versions

    Tests no longer passes in older Django versions

    When you have Django 1.5.* and you run tests, django-robots 1.1 fails with message:

    File "/lib/python2.7/site-packages/robots/forms.py", line 7, in <module>
      class RuleAdminForm(forms.ModelForm):
    File "/lib/python2.7/site-packages/django/forms/models.py", line 221, in __new__
      raise FieldError(message)
    FieldError: Unknown field(s) (a, l, _) specified for Rule
    

    This is related to https://github.com/jezdez/django-robots/pull/30 where __all__ was added to support Django 1.8 which causes to break in older versions.

    opened by ReekenX 6
  • Needs version bump and new deploy to PyPi

    Needs version bump and new deploy to PyPi

    Just pip installed this but it doesn't have the fix for fields fix in robots/forms.py.

    I've worked around this by changing the entry in my requirements.txt file to -e git://github.com/jezdez/django-robots.git#egg=robots to install it from GitHub.

    opened by ryan-blunden 6
  • Solve MakeMigration issue with MySql

    Solve MakeMigration issue with MySql

    django-robots version 5.0 I am using above version of django-robots. When database is SQLite, things are working fine but when the database is MySQL, it gives following error while running migrate command.

    Error: django.db.utils.OperationalError: (3780, "Referencing column 'rule_id' and referenced column 'id' in foreign key constraint 'robots_rule_sites_rule_id_7921a799_fk_robots_rule_id' are incompatible.")

    I investigate and figure out that the BigAutoField is causing the problem.

    opened by pooja-narwade 0
  • Maintainers needed

    Maintainers needed

    Hi!

    If anyone is able to help out with maintenance, QA, writing tests, reviewing the pull request list, and triaging issues, this would be helpful! At this time I'm a bit short-handed.

    It's critical we get releases tested and that any fixes on the above are future issues working correctly.

    Outstanding issues, especially if you're familiar with migrations:

    • #124, #125. We have a PR at #134, which look decent to me
    • Same goes for #132 which as PR #133

    It's not clear to me - at this time - what the ramifications for creating migrations will be at scale for users across different django versions, who may have made their own migration, etc.

    opened by tony 1
  • fix problem with long Url.pattern from model_bakery and admin

    fix problem with long Url.pattern from model_bakery and admin

    fix #132

    • The MaxLengthValidator has problem, that it doesn't allow / with 254 other characters (which is generated when saved previously with long pattern). But I suppose it is very minor problem not worth adding more complexity to the code.
    opened by PetrDlouhy 5
  • Not able to create Url with model_bakery: too long pattern throws exception

    Not able to create Url with model_bakery: too long pattern throws exception

    If I try to create Url model with model_bakery without any parameters, I have got this error:

    django.db.utils.DataError: value too long for type character varying(255)
    

    The problem is, that pattern parameter (generated by model_bakery) gets lengthen in save() method by leading / and doesn't fit in the field anymore.

    The solution is very simple: add default="" to the pattern field.

    There is also second problem with the save method - if user enters pattern exactly 255 characters long in the admin, it will throw an exception rather than showing validation error.

    opened by PetrDlouhy 0
Releases(6.0b0)
  • 6.0b0(Oct 15, 2022)

    Breaking change: Missing DEFAULT_AUTO_FIELD

    This sets a DEFAULT_AUTO_FIELD to BigAutoField for robots in AppConfig. Previously, there was no app configuration or setting for this. (thank you @jan-szejko-steelseries in https://github.com/jazzband/django-robots/pull/134)

    Robots 5.0 included a BigAutoField migration file for django 3.2 support (see #112), however, this caused an extra migration to be created to set django.db.models.AutoField for certain users (see #124).

    Breaking: Migration issues

    New django sites should work fine.

    Existing sites that manually created migrations for robots 5.0:

    If you ran ./manage.py makemigrations to bypass the #112 issue, you may need to resolve this manually if a custom migration you made for robots was applied in django_migrations.

    Resources: - https://docs.djangoproject.com/en/4.1/releases/3.2/#customizing-type-of-auto-created-primary-keyshttps://docs.djangoproject.com/en/3.2/ref/settings/#std-setting-DEFAULT_AUTO_FIELD

    Packaging

    • build: Remove django south migrations by @tony in https://github.com/jazzband/django-robots/pull/138

    New Contributors

    • @pre-commit-ci made their first contribution in https://github.com/jazzband/django-robots/pull/126
    • @jan-szejko-steelseries made their first contribution in https://github.com/jazzband/django-robots/pull/134

    Full Changelog: https://github.com/jazzband/django-robots/compare/5.0...6.0b0

    Source code(tar.gz)
    Source code(zip)
  • 5.0(Jan 8, 2022)

    This may be the only prerelease. Will have shareholders from our django 4.0 threads give this a try first

    What's Changed

    • Use gettext_lazy for Python 3 code by @noisecapella in https://github.com/jazzband/django-robots/pull/99
    • Updated code style for isort v5 and black 20.8b1 by @smithdc1 in https://github.com/jazzband/django-robots/pull/104
    • Dropped support for Python 2.7 and 3.5 by @smithdc1 in https://github.com/jazzband/django-robots/pull/105
    • Migrate to GitHub Actions. by @jezdez in https://github.com/jazzband/django-robots/pull/106
    • Updated supported versions. Added Django 3.1 and Python 3.9. Dropped Django 2.2 and below. by @smithdc1 in https://github.com/jazzband/django-robots/pull/107
    • Rename Django's dev branch to main. by @jezdez in https://github.com/jazzband/django-robots/pull/110
    • add django 3.2 and 4.0 support by @umarmughal824 in https://github.com/jazzband/django-robots/pull/112
    • Add Python 3.10 support by @Andrew-Chen-Wang in https://github.com/jazzband/django-robots/pull/113
    • Jazzband: Synced file(s) with jazzband/.github by @jazzband-bot in https://github.com/jazzband/django-robots/pull/115
    • Update Django 4 documentation by @sergioisidoro in https://github.com/jazzband/django-robots/pull/119
    • Deprecate python 3.6 by @tony in https://github.com/jazzband/django-robots/pull/120
    • Add .pre-commit-config.yaml by @tony in https://github.com/jazzband/django-robots/pull/122
    • docs(CHANGES): Update for next release by @tony in https://github.com/jazzband/django-robots/pull/121

    New Contributors

    • @noisecapella made their first contribution in https://github.com/jazzband/django-robots/pull/99
    • @smithdc1 made their first contribution in https://github.com/jazzband/django-robots/pull/104
    • @jezdez made their first contribution in https://github.com/jazzband/django-robots/pull/106
    • @umarmughal824 made their first contribution in https://github.com/jazzband/django-robots/pull/112
    • @Andrew-Chen-Wang made their first contribution in https://github.com/jazzband/django-robots/pull/113
    • @sergioisidoro made their first contribution in https://github.com/jazzband/django-robots/pull/119

    Full Changelog: https://github.com/jazzband/django-robots/compare/4.0...5.0b1

    Source code(tar.gz)
    Source code(zip)
  • 5.0b1(Jan 6, 2022)

    This may be the only prerelease. Will have shareholders from our django 4.0 threads give this a try first

    What's Changed

    • Use gettext_lazy for Python 3 code by @noisecapella in https://github.com/jazzband/django-robots/pull/99
    • Updated code style for isort v5 and black 20.8b1 by @smithdc1 in https://github.com/jazzband/django-robots/pull/104
    • Dropped support for Python 2.7 and 3.5 by @smithdc1 in https://github.com/jazzband/django-robots/pull/105
    • Migrate to GitHub Actions. by @jezdez in https://github.com/jazzband/django-robots/pull/106
    • Updated supported versions. Added Django 3.1 and Python 3.9. Dropped Django 2.2 and below. by @smithdc1 in https://github.com/jazzband/django-robots/pull/107
    • Rename Django's dev branch to main. by @jezdez in https://github.com/jazzband/django-robots/pull/110
    • add django 3.2 and 4.0 support by @umarmughal824 in https://github.com/jazzband/django-robots/pull/112
    • Add Python 3.10 support by @Andrew-Chen-Wang in https://github.com/jazzband/django-robots/pull/113
    • Jazzband: Synced file(s) with jazzband/.github by @jazzband-bot in https://github.com/jazzband/django-robots/pull/115
    • Update Django 4 documentation by @sergioisidoro in https://github.com/jazzband/django-robots/pull/119
    • Deprecate python 3.6 by @tony in https://github.com/jazzband/django-robots/pull/120
    • Add .pre-commit-config.yaml by @tony in https://github.com/jazzband/django-robots/pull/122
    • docs(CHANGES): Update for next release by @tony in https://github.com/jazzband/django-robots/pull/121

    New Contributors

    • @noisecapella made their first contribution in https://github.com/jazzband/django-robots/pull/99
    • @smithdc1 made their first contribution in https://github.com/jazzband/django-robots/pull/104
    • @jezdez made their first contribution in https://github.com/jazzband/django-robots/pull/106
    • @umarmughal824 made their first contribution in https://github.com/jazzband/django-robots/pull/112
    • @Andrew-Chen-Wang made their first contribution in https://github.com/jazzband/django-robots/pull/113
    • @sergioisidoro made their first contribution in https://github.com/jazzband/django-robots/pull/119

    Full Changelog: https://github.com/jazzband/django-robots/compare/4.0...5.0b1

    Source code(tar.gz)
    Source code(zip)
  • 4.0(Jan 4, 2020)

    • Support for Django 2.1, 2.2, 3.0 and Python 3.7 and 3.8.
    • Updated the requirements for dropped support of six in Django 3.
    • Restructure test setup to use setuptools-scm and more modern Python patterns.
    Source code(tar.gz)
    Source code(zip)
Fast / fuzzy PostgreSQL counts for Django

Created by Stephen McDonald Introduction Up until PostgreSQL 9.2, COUNT queries generally required scanning every row in a database table. With millio

stephenmcd 85 Oct 25, 2021
pytest-django allows you to test your Django project/applications with the pytest testing tool.

pytest-django allows you to test your Django project/applications with the pytest testing tool.

pytest-dev 1.1k Dec 14, 2022
Login System Django

Login-System-Django Login System Using Django Tech Used Django Python Html Run Locally Clone project git clone https://link-to-project Get project for

Nandini Chhajed 6 Dec 12, 2021
An URL Shortener with Basic Features.

Simple Url Shortener What is that? Yet another url shortener built with Django framework. Preview HOW TO RUN? 1. Virtual Environment First create a vi

Ethem Turgut 6 Jan 25, 2022
Inject an ID into every log message from a Django request. ASGI compatible, integrates with Sentry, and works with Celery

Django GUID Now with ASGI support! Django GUID attaches a unique correlation ID/request ID to all your log outputs for every request. In other words,

snok 300 Dec 29, 2022
Django GUID attaches a unique correlation ID/request ID to all your log outputs for every request.

Django GUID Now with ASGI support! Django GUID attaches a unique correlation ID/request ID to all your log outputs for every request. In other words,

snok 300 Dec 29, 2022
mirage ~ ♪ extended django admin or manage.py command.

mirage ~ ♪ extended django admin or manage.py command. ⬇️ Installation Installing Mirage with Pipenv is recommended. pipenv install -d mirage-django-l

Shota Shimazu 6 Feb 14, 2022
Django application and library for importing and exporting data with admin integration.

django-import-export django-import-export is a Django application and library for importing and exporting data with included admin integration. Featur

2.6k Dec 26, 2022
A drop-in replacement for django's ImageField that provides a flexible, intuitive and easily-extensible interface for quickly creating new images from the one assigned to the field.

django-versatileimagefield A drop-in replacement for django's ImageField that provides a flexible, intuitive and easily-extensible interface for creat

Jonathan Ellenberger 490 Dec 13, 2022
It's the assignment 1 from the Python 2 course, that requires a ToDoApp with authentication using Django

It's the assignment 1 from the Python 2 course, that requires a ToDoApp with authentication using Django

0 Jan 20, 2022
User Authentication In Django/Ajax/Jquery

User Authentication In Django/Ajax/Jquery Demo: Authentication System Using Django/Ajax/Jquery Demo: Authentication System Using Django Overview The D

Suman Raj Khanal 10 Mar 26, 2022
Projeto Crud Django and Mongo

Projeto-Crud_Django_and_Mongo Configuração para rodar o projeto Download Project

Samuel Fernandes Oliveira 2 Jan 24, 2022
Django React Project Setup

Django-React-Project-Setup INSTALLATION: python -m pip install drps USAGE: in your cmd: python -m drps Starting fullstack project with Django and Reac

Ghazi Zabalawi 7 Feb 06, 2022
A reusable Django model field for storing ad-hoc JSON data

jsonfield jsonfield is a reusable model field that allows you to store validated JSON, automatically handling serialization to and from the database.

Ryan P Kilby 1.1k Jan 03, 2023
Sistema administrador de contranas desarrollador en Django

Sistema Contrasenas Desarrolado en Django Proyecto sistema de administracion de contraseñas, de la experiencia educativa Programacion Segura Descripci

Ibrain Rodriguez Espinoza 1 Sep 24, 2022
Django datatables with htmx.

Django datatables with htmx.

Regis Santos 7 Oct 23, 2022
Utility for working with recurring dates in Django.

django-recurrence django-recurrence is a utility for working with recurring dates in Django. Documentation is available at https://django-recurrence.r

408 Jan 06, 2023
Flashback is an awesome, retro IRC based app built using Django

Flashback Flashback is an awesome, retro IRC based app built using Django (and the Django Rest Framework) for the backend as well as React for the fro

Unloading Gnat 1 Dec 22, 2021
Bleach is an allowed-list-based HTML sanitizing library that escapes or strips markup and attributes

Bleach Bleach is an allowed-list-based HTML sanitizing library that escapes or strips markup and attributes. Bleach can also linkify text safely, appl

Mozilla 2.5k Dec 29, 2022
REST API con Python, Django y MySQL (GET, POST, PUT, DELETE)

django_api_mysql REST API con Python, Django y MySQL (GET, POST, PUT, DELETE) REST API con Python, Django y MySQL (GET, POST, PUT, DELETE)

Andrew 1 Dec 28, 2021