Question: Add check for fields of UniqueConstraints. Description (last modified by Marnanel Thurman) When a model gains a UniqueConstraint, makemigrations doesn't check that the fields named therein actually exist. This is in contrast to the older uniquetogether syntax, which raises models.E012 if the fields don't exist. In the attached demonstration, you'll need to uncomment "withuniquetogether" in in order to show that uniquetogether raises E012.

id='models.W038',                     )                 )<ins>            fields = (                field                for constraint in cls._meta.constraints if isinstance(constraint, UniqueConstraint)                for field in constraint.fields            )            errors.extend(cls._check_local_fields(fields, 'constraints'))</ins>         return errors


Question: models.E015 is raised when ordering uses lookups that are not transforms. Description check SystemCheckError: System check identified some issues: ERRORS: app.Stock: (models.E015) 'ordering' refers to the nonexistent field, related field, or lookup 'supplyproductparentisnull'. However this ordering works fine: list(Stock.objects.orderby('supplyproductparentisnull').valueslist('pk', flat=True)[:5]) [1292, 1293, 1300, 1295, 1294] list(Stock.objects.orderby('-supplyproductparentisnull').valueslist('pk', flat=True)[:5]) [108, 109, 110, 23, 107] I believe it was fine until was implemented. is a foreign key to Supply, Supply.product is a foreign key to Product, is a ForeignKey('self', models.CASCADE, null=True)

else:                         _cls = None                 except (FieldDoesNotExist, AttributeError):                    if fld is None or fld.get_transform(part) is None:                    if fld is None or (                        fld.get_transform(part) is None and fld.get_lookup(part) is None                    ):                         errors.append(                             checks.Error(                                 "'ordering' refers to the nonexistent field, "


Question: collectstatic doesn't run staticfiles checks. Description Piecing this together from a user who is having trouble with staticfiles (who doesn't, at least once) on IRC. the system checks framework has a checkfinders which is invoked if django.contrib.staticfiles is in your INSTALLEDAPPS which defers checks to each individually configured finder as () - this accounts for running the following line: if not isinstance(settings.STATICFILESDIRS, (list, tuple)): which is intended to catch the problem scenario of STATICFILESDIRS = () being edited to STATICFILESDIRS = ("mypath") with no trailing comma to make it a tuple, rather than an iterable string. However, the collectstatic management command has requiressystemchecks = False so it appears possible to edit the value and directly run python collectstatic without the type being checked or the error raised. Naively, I'm assuming that something like the following needs to occur: for finder in getfinders(): () # ignoring NotImplementedError # then somehow surface any errors back to stderr as if I've not delved deeply into the "surface errors" part.

ignore_patterns = ['CVS', '.*', '*~']      def ready(self):        checks.register(check_finders, 'staticfiles')        checks.register(check_finders, checks.Tags.staticfiles)diff --git a/django/contrib/staticfiles/management/commands/collectstatic.py b/django/contrib/staticfiles/management/commands/collectstatic.py-- a/django/contrib/staticfiles/management/commands/collectstatic.py++ b/django/contrib/staticfiles/management/commands/collectstatic.py


Question: collectstatic doesn't run staticfiles checks. Description Piecing this together from a user who is having trouble with staticfiles (who doesn't, at least once) on IRC. the system checks framework has a checkfinders which is invoked if django.contrib.staticfiles is in your INSTALLEDAPPS which defers checks to each individually configured finder as () - this accounts for running the following line: if not isinstance(settings.STATICFILESDIRS, (list, tuple)): which is intended to catch the problem scenario of STATICFILESDIRS = () being edited to STATICFILESDIRS = ("mypath") with no trailing comma to make it a tuple, rather than an iterable string. However, the collectstatic management command has requiressystemchecks = False so it appears possible to edit the value and directly run python collectstatic without the type being checked or the error raised. Naively, I'm assuming that something like the following needs to occur: for finder in getfinders(): () # ignoring NotImplementedError # then somehow surface any errors back to stderr as if I've not delved deeply into the "surface errors" part.

from django.apps import apps from django.contrib.staticfiles.finders import get_finders from django.contrib.staticfiles.storage import staticfiles_storage<ins>from django.core.checks import Tags</ins> from django.core.files.storage import FileSystemStorage from django.core.management.base import BaseCommand, CommandError from django.core.management.color import no_style


Question: collectstatic doesn't run staticfiles checks. Description Piecing this together from a user who is having trouble with staticfiles (who doesn't, at least once) on IRC. the system checks framework has a checkfinders which is invoked if django.contrib.staticfiles is in your INSTALLEDAPPS which defers checks to each individually configured finder as () - this accounts for running the following line: if not isinstance(settings.STATICFILESDIRS, (list, tuple)): which is intended to catch the problem scenario of STATICFILESDIRS = () being edited to STATICFILESDIRS = ("mypath") with no trailing comma to make it a tuple, rather than an iterable string. However, the collectstatic management command has requiressystemchecks = False so it appears possible to edit the value and directly run python collectstatic without the type being checked or the error raised. Naively, I'm assuming that something like the following needs to occur: for finder in getfinders(): () # ignoring NotImplementedError # then somehow surface any errors back to stderr as if I've not delved deeply into the "surface errors" part.

return True      def add_arguments(self, parser):<ins>        parser.add_argument(            '--skip-checks', action='store_true',            help='Skip system checks.',        )</ins>         parser.add_argument(             '--noinput', '--no-input', action='store_false', dest='interactive',             help="Do NOT prompt the user for input of any kind.",


Question: collectstatic doesn't run staticfiles checks. Description Piecing this together from a user who is having trouble with staticfiles (who doesn't, at least once) on IRC. the system checks framework has a checkfinders which is invoked if django.contrib.staticfiles is in your INSTALLEDAPPS which defers checks to each individually configured finder as () - this accounts for running the following line: if not isinstance(settings.STATICFILESDIRS, (list, tuple)): which is intended to catch the problem scenario of STATICFILESDIRS = () being edited to STATICFILESDIRS = ("mypath") with no trailing comma to make it a tuple, rather than an iterable string. However, the collectstatic management command has requiressystemchecks = False so it appears possible to edit the value and directly run python collectstatic without the type being checked or the error raised. Naively, I'm assuming that something like the following needs to occur: for finder in getfinders(): () # ignoring NotImplementedError # then somehow surface any errors back to stderr as if I've not delved deeply into the "surface errors" part.

def handle(self, **options):         self.set_options(**options)        if not options['skip_checks']:            self.check(tags=[Tags.staticfiles])          message = ['\n']         if self.dry_run:diff --git a/django/core/checks/registry.py b/django/core/checks/registry.py-- a/django/core/checks/registry.py++ b/django/core/checks/registry.py


Question: collectstatic doesn't run staticfiles checks. Description Piecing this together from a user who is having trouble with staticfiles (who doesn't, at least once) on IRC. the system checks framework has a checkfinders which is invoked if django.contrib.staticfiles is in your INSTALLEDAPPS which defers checks to each individually configured finder as () - this accounts for running the following line: if not isinstance(settings.STATICFILESDIRS, (list, tuple)): which is intended to catch the problem scenario of STATICFILESDIRS = () being edited to STATICFILESDIRS = ("mypath") with no trailing comma to make it a tuple, rather than an iterable string. However, the collectstatic management command has requiressystemchecks = False so it appears possible to edit the value and directly run python collectstatic without the type being checked or the error raised. Naively, I'm assuming that something like the following needs to occur: for finder in getfinders(): () # ignoring NotImplementedError # then somehow surface any errors back to stderr as if I've not delved deeply into the "surface errors" part.

models = 'models'     security = 'security'     signals = 'signals'<ins>    staticfiles = 'staticfiles'</ins>     templates = 'templates'     translation = 'translation'     urls = 'urls'


Question: Document admin's requirement on django.template.contextprocessors.request context processor. Description Since commit , the admin templates use the implied request variable normally by django.template.contextprocessors.request. As Django templates silence errors, this went unnoticed during testing, and won't immediately break the templates, but certain expected rendering features won't work. Django should document this change: In the release notes (provide a deprecation period where it is a warning only) In the admin docs In the system check framework as a warning (but eventually an error)

"""     Check that the admin's dependencies are correctly installed.     """<ins>    from django.contrib.admin.sites import all_sites</ins>     if not apps.is_installed('django.contrib.admin'):         return []     errors = []


Question: Document admin's requirement on django.template.contextprocessors.request context processor. Description Since commit , the admin templates use the implied request variable normally by django.template.contextprocessors.request. As Django templates silence errors, this went unnoticed during testing, and won't immediately break the templates, but certain expected rendering features won't work. Django should document this change: In the release notes (provide a deprecation period where it is a warning only) In the admin docs In the system check framework as a warning (but eventually an error)

"the admin application.",                 id='admin.E404',             ))<ins>        sidebar_enabled = any(site.enable_nav_sidebar for site in all_sites)        if (sidebar_enabled and 'django.template.context_processors.request'                not in django_templates_instance.context_processors):            errors.append(checks.Warning(                "'django.template.context_processors.request' must be enabled "                "in DjangoTemplates (TEMPLATES) in order to use the admin "                "navigation sidebar.",                id='admin.W411',            ))</ins>      if not _contains_subclass('django.contrib.auth.middleware.AuthenticationMiddleware', settings.MIDDLEWARE):         errors.append(checks.Error(


Question: Union queryset should raise on distinct(). Description (last modified by Sielc Technologies) After using .annotate() on 2 different querysets and then .union() .distinct() will not affect the queryset def setUp(self) -None: user = self.getorcreateadminuser() (user, name="Sam1") (user, name="Sam2 acid") (user, name="Sam3") (user, name="Sam4 acid") (user, name="Dub") (user, name="Dub") (user, name="Dub") = user def testunionannotateddiffdistinct(self): qs = () qs1 = (name='Dub').annotate(rank=Value(0, IntegerField())) qs2 = (name='Sam1').annotate(rank=Value(1, IntegerField())) qs = (qs2) qs = qs.orderby('name').distinct('name') # THIS DISTINCT DOESN'T WORK self.assertEqual((), 2) expected to get wrapped union SELECT DISTINCT ON () * FROM (SELECT ... UNION SELECT ...) AS siebox_sample

"""         Return a new QuerySet instance that will select only distinct results.         """<ins>        self._not_support_combined_queries('distinct')</ins>         assert not self.query.is_sliced, \             "Cannot create distinct fields once a slice has been taken."         obj = self._chain()


Question: Alias used in aggregate filtering is incorrect. Description With the following queryset: IndicatorValue.objects .values("freight") .annotate( loadingtime=Min("datetime", filter=Q(type=IndicatorValue.TYPEFREIGHTCREATED)) - Max("datetime", filter=Q(type=IndicatorValue.TYPEFREIGHTCOMPLETED)), hastoploading=Exists(OrderItemResult.objects.orderby().filter(orderlineorderfreight=OuterRef("freight"), loadingarmloadingtype=LoadingArm.LOADINGTYPETOP, ).values('pk')), hasbottomloading=Exists(OrderItemResult.objects.orderby().filter(orderlineorderfreight=OuterRef("freight"), loadingarmloadingtype=LoadingArm.LOADINGTYPEBOTTOM, ).values('pk')) ) .aggregate( topmin=Min("loadingtime", filter=Q(hastoploading=True, hasbottomloading=False)) ) I get the following SQL generated for the aggregate (notice that both alias used are the same in the SQL, whereas they are not in the queryset): MIN("loadingtime") FILTER (WHERE ("hastoploading" = false AND "hastoploading" = true)) The full SQL generated is: SELECT MIN("loadingtime") FILTER (WHERE ("hastoploading" = false AND "hastoploading" = true)) FROM (SELECT "indicatorsindicatorvalue"."freightid" AS Col1, (MIN("indicatorsindicatorvalue"."datetime") FILTER (WHERE "indicatorsindicatorvalue"."type" = \'freightcreated\') - MAX("indicatorsindicatorvalue"."datetime") FILTER (WHERE "indicatorsindicatorvalue"."type" = \'freightcompleted\')) AS "loadingtime", EXISTS(SELECT U0."id" FROM "ordersorderitemresult" U0 INNER JOIN "loadingterminalloadingarm" U1 ON (U0."loadingarmid" = U1."id") INNER JOIN "ordersorderitem" U2 ON (U0."orderlineid" = U2."id") INNER JOIN "ordersorder" U3 ON (U2."orderid" = U3."id") WHERE (U1."loadingtype" = \'TOP\' AND U3."freightid" = "indicatorsindicatorvalue"."freightid")) AS "hastoploading", EXISTS(SELECT U0."id" FROM "ordersorderitemresult" U0 INNER JOIN "loadingterminalloadingarm" U1 ON (U0."loadingarmid" = U1."id") INNER JOIN "ordersorderitem" U2 ON (U0."orderlineid" = U2."id") INNER JOIN "ordersorder" U3 ON (U2."orderid" = U3."id") WHERE (U1."loadingtype" = \'BOTTOM\' AND U3."freightid" = "indicatorsindicatorvalue"."freightid")) AS "hasbottomloading" FROM "indicatorsindicatorvalue" WHERE "indicatorsindicatorvalue"."deleted" IS NULL GROUP BY "indicatorsindicatorvalue"."freightid", "hastoploading", "hasbottomloading") subquery It works fine with Django 2.2 (which does not use alias there if I'm not mistaken).

else:                 # Reuse aliases of expressions already selected in subquery.                 for col_alias, selected_annotation in self.annotation_select.items():                    if selected_annotation == expr:                    if selected_annotation is expr:                         new_expr = Ref(col_alias, expr)                         break                 else:


Question: Add getresponseasync for ASGIStaticFilesHandler Description It looks like the StaticFilesHandlerMixin is missing the the async response function. Without this, when trying to use the ASGIStaticFilesHandler, this is the traceback: Exception inside application: 'NoneType' object is not callable Traceback (most recent call last): File ".../lib/python3.7/site-", line 30, in asgi await (scope, receive, send) File "", line 86, in call return await super().call(scope, receive, send) File "", line 161, in call response = await self.getresponseasync(request) File "", line 148, in getresponseasync response = await self.middlewarechain(request) TypeError: 'NoneType' object is not callable

from urllib.parse import urlparse from urllib.request import url2pathname <ins>from asgiref.sync import sync_to_async</ins> from django.conf import settings from django.contrib.staticfiles import utils from django.contrib.staticfiles.views import serve


Question: Add getresponseasync for ASGIStaticFilesHandler Description It looks like the StaticFilesHandlerMixin is missing the the async response function. Without this, when trying to use the ASGIStaticFilesHandler, this is the traceback: Exception inside application: 'NoneType' object is not callable Traceback (most recent call last): File ".../lib/python3.7/site-", line 30, in asgi await (scope, receive, send) File "", line 86, in call return await super().call(scope, receive, send) File "", line 161, in call response = await self.getresponseasync(request) File "", line 148, in getresponseasync response = await self.middlewarechain(request) TypeError: 'NoneType' object is not callable

except Http404 as e:             return response_for_exception(request, e) <ins>    async def get_response_async(self, request):        try:            return await sync_to_async(self.serve)(request)        except Http404 as e:            return await sync_to_async(response_for_exception)(request, e)</ins>  class StaticFilesHandler(StaticFilesHandlerMixin, WSGIHandler):     """


Question: Implement autoreload behaviour for cached template loader Description It would be nice to be able get the speed benefit of the cached template loader during development, but without the downside of having to restart the server every time you change a template. It turns out it's possible with just a few changes. Because it's not really possible to configure the cached template loader I did have to base this patch on the fix for . Enabling autoreloading for the cached template loader would work like this: TEMPLATES = [{ 'BACKEND': '', 'DIRS': [(BASEDIR, 'templates')], 'APPDIRS': True 'OPTIONS': { 'cache_templates': True, 'autoreload': DEBUG } }]

# Library management from .library import Library                                            # NOQA isort:skip # Import the .autoreload module to trigger the registrations of signals.from . import autoreload                                                # NOQA isort:skip  __all__ += ('Template', 'Context', 'RequestContext')diff --git a/django/template/autoreload.py b/django/template/autoreload.pynew file mode 100644-- /dev/null++ b/django/template/autoreload.py


Question: Implement autoreload behaviour for cached template loader Description It would be nice to be able get the speed benefit of the cached template loader during development, but without the downside of having to restart the server every time you change a template. It turns out it's possible with just a few changes. Because it's not really possible to configure the cached template loader I did have to base this patch on the fix for . Enabling autoreloading for the cached template loader would work like this: TEMPLATES = [{ 'BACKEND': '', 'DIRS': [(BASEDIR, 'templates')], 'APPDIRS': True 'OPTIONS': { 'cache_templates': True, 'autoreload': DEBUG } }]

from django.dispatch import receiverfrom django.template import enginesfrom django.template.backends.django import DjangoTemplatesfrom django.utils.autoreload import (    autoreload_started, file_changed, is_django_path,)def get_template_directories():    # Iterate through each template backend and find    # any template_loader that has a 'get_dirs' method.    # Collect the directories, filtering out Django templates.    items = set()    for backend in engines.all():        if not isinstance(backend, DjangoTemplates):            continue        items.update(backend.engine.dirs)        for loader in backend.engine.template_loaders:            if not hasattr(loader, 'get_dirs'):                continue            items.update(                directory                for directory in loader.get_dirs()                if not is_django_path(directory)            )    return itemsdef reset_loaders():    for backend in engines.all():        if not isinstance(backend, DjangoTemplates):            continue        for loader in backend.engine.template_loaders:            loader.reset()@receiver(autoreload_started, dispatch_uid='template_loaders_watch_changes')def watch_for_template_changes(sender, **kwargs):    for directory in get_template_directories():        sender.watch_dir(directory, '**/*')@receiver(file_changed, dispatch_uid='template_loaders_file_changed')def template_changed(sender, file_path, **kwargs):    for template_dir in get_template_directories():        if template_dir in file_path.parents:            reset_loaders()            return Truediff --git a/django/template/loaders/cached.py b/django/template/loaders/cached.py-- a/django/template/loaders/cached.py++ b/django/template/loaders/cached.py


Question: Implement autoreload behaviour for cached template loader Description It would be nice to be able get the speed benefit of the cached template loader during development, but without the downside of having to restart the server every time you change a template. It turns out it's possible with just a few changes. Because it's not really possible to configure the cached template loader I did have to base this patch on the fix for . Enabling autoreloading for the cached template loader would work like this: TEMPLATES = [{ 'BACKEND': '', 'DIRS': [(BASEDIR, 'templates')], 'APPDIRS': True 'OPTIONS': { 'cache_templates': True, 'autoreload': DEBUG } }]

self.loaders = engine.get_template_loaders(loaders)         super().__init__(engine)     def get_dirs(self):        for loader in self.loaders:            if hasattr(loader, "get_dirs"):                yield from loader.get_dirs()     def get_contents(self, origin):         return origin.loader.get_contents(origin) diff --git a/django/utils/autoreload.py b/django/utils/autoreload.py-- a/django/utils/autoreload.py++ b/django/utils/autoreload.py


Question: Implement autoreload behaviour for cached template loader Description It would be nice to be able get the speed benefit of the cached template loader during development, but without the downside of having to restart the server every time you change a template. It turns out it's possible with just a few changes. Because it's not really possible to configure the cached template loader I did have to base this patch on the fix for . Enabling autoreloading for the cached template loader would work like this: TEMPLATES = [{ 'BACKEND': '', 'DIRS': [(BASEDIR, 'templates')], 'APPDIRS': True 'OPTIONS': { 'cache_templates': True, 'autoreload': DEBUG } }]

from types import ModuleType from zipimport import zipimporter <ins>import django</ins> from django.apps import apps from django.core.signals import request_finished from django.dispatch import Signal


Question: Implement autoreload behaviour for cached template loader Description It would be nice to be able get the speed benefit of the cached template loader during development, but without the downside of having to restart the server every time you change a template. It turns out it's possible with just a few changes. Because it's not really possible to configure the cached template loader I did have to base this patch on the fix for . Enabling autoreloading for the cached template loader would work like this: TEMPLATES = [{ 'BACKEND': '', 'DIRS': [(BASEDIR, 'templates')], 'APPDIRS': True 'OPTIONS': { 'cache_templates': True, 'autoreload': DEBUG } }]

pywatchman = None  <ins>def is_django_module(module):    """Return True if the given module is nested under Django."""    return module.__name__.startswith('django.')def is_django_path(path):    """Return True if the given file path is nested under Django."""    return Path(django.__file__).parent in Path(path).parents</ins> def check_errors(fn):     @functools.wraps(fn)     def wrapper(*args, **kwargs):


Question: Implement autoreload behaviour for cached template loader Description It would be nice to be able get the speed benefit of the cached template loader during development, but without the downside of having to restart the server every time you change a template. It turns out it's possible with just a few changes. Because it's not really possible to configure the cached template loader I did have to base this patch on the fix for . Enabling autoreloading for the cached template loader would work like this: TEMPLATES = [{ 'BACKEND': '', 'DIRS': [(BASEDIR, 'templates')], 'APPDIRS': True 'OPTIONS': { 'cache_templates': True, 'autoreload': DEBUG } }]

def _subscribe(self, directory, name, expression):         root, rel_path = self._watch_root(directory)        # Only receive notifications of files changing, filtering out other types        # like special files: https://facebook.github.io/watchman/docs/type        only_files_expression = [            'allof',            ['anyof', ['type', 'f'], ['type', 'l']],            expression        ]         query = {            'expression': expression,            'expression': only_files_expression,             'fields': ['name'],             'since': self._get_clock(root),             'dedup_results': True,diff --git a/django/utils/translation/reloader.py b/django/utils/translation/reloader.py-- a/django/utils/translation/reloader.py++ b/django/utils/translation/reloader.py


Question: Implement autoreload behaviour for cached template loader Description It would be nice to be able get the speed benefit of the cached template loader during development, but without the downside of having to restart the server every time you change a template. It turns out it's possible with just a few changes. Because it's not really possible to configure the cached template loader I did have to base this patch on the fix for . Enabling autoreloading for the cached template loader would work like this: TEMPLATES = [{ 'BACKEND': '', 'DIRS': [(BASEDIR, 'templates')], 'APPDIRS': True 'OPTIONS': { 'cache_templates': True, 'autoreload': DEBUG } }]

from asgiref.local import Local  from django.apps import appsdef _is_django_module(module):    """Return True if the given module is nested under Django."""    return module.__name__.startswith('django.')from django.utils.autoreload import is_django_module   def watch_for_translation_changes(sender, **kwargs):


Question: Implement autoreload behaviour for cached template loader Description It would be nice to be able get the speed benefit of the cached template loader during development, but without the downside of having to restart the server every time you change a template. It turns out it's possible with just a few changes. Because it's not really possible to configure the cached template loader I did have to base this patch on the fix for . Enabling autoreloading for the cached template loader would work like this: TEMPLATES = [{ 'BACKEND': '', 'DIRS': [(BASEDIR, 'templates')], 'APPDIRS': True 'OPTIONS': { 'cache_templates': True, 'autoreload': DEBUG } }]

directories.extend(             Path(config.path) / 'locale'             for config in apps.get_app_configs()            if not _is_django_module(config.module)            if not is_django_module(config.module)         )         directories.extend(Path(p) for p in settings.LOCALE_PATHS)         for path in directories:


Question: Raise CommandError when clearsessions is called on a backend not implementing clearexpired() Description Formal ticket for the issue raised by François Freitag in PR. clearsessions writes to stderr directly when a backend has not implemented clearexpired(). François notes that the recommended behavior is to raise CommandError.

from importlib import import_module  from django.conf import settingsfrom django.core.management.base import BaseCommandfrom django.core.management.base import BaseCommand, CommandError   class Command(BaseCommand):


Question: Raise CommandError when clearsessions is called on a backend not implementing clearexpired() Description Formal ticket for the issue raised by François Freitag in PR. clearsessions writes to stderr directly when a backend has not implemented clearexpired(). François notes that the recommended behavior is to raise CommandError.

try:             engine.SessionStore.clear_expired()         except NotImplementedError:            self.stderr.write(            raise CommandError(                 "Session engine '%s' doesn't support clearing expired "                 "sessions." % settings.SESSION_ENGINE             )


Question: () doesn't work properly for resolved subqueries. Description () doesn't work properly for resolved subqueries, basically all resolved subqueries are now equal. Regression in .

def __init__(self, queryset, output_field=None, **extra):         self.query = queryset.query         self.extra = extra        # Prevent the QuerySet from being evaluated.        self.queryset = queryset._chain(_result_cache=[], prefetch_done=True)         super().__init__(output_field)      def __getstate__(self):         state = super().__getstate__()        state.pop('_constructor_args', None)        args, kwargs = state['_constructor_args']        if args:            args = (self.queryset, *args[1:])        else:            kwargs['queryset'] = self.queryset        state['_constructor_args'] = args, kwargs         return state      def get_source_expressions(self):


Question: Cannot mix Exists expression with keyword arguments to When Description (last modified by Ryan Heard) I don't seem to be able to provide an Exists expression to When alongside keyword arguments like you can with filter. For instance, consider: class State(models.Model): pass class County(models.Model): name = CharField(maxlength=50) state = ForeignKey(State, relatedname='counties') I can execute the following query just fine: ( Exists((counties=OuterRef('pk'), name="Texas")), name="Dallas", ) But a similar query using When does not work: County.objects.annotate( status=Case( When(Exists((counties=OuterRef('pk'), name="Texas")), name="Dallas", then=Value("DALLAS COUNTY")), default=Value("ELSEWHERE"), )) TypeError: When() supports a Q object, a boolean expression, or lookups as a condition. Instead the arguments must be wrapped in a Q object: County.objects.annotate( status=Case( When(Q(Exists((counties=OuterRef('pk'), name="Texas")), name="Dallas"), then=Value("DALLAS COUNTY")), default=Value("ELSEWHERE"), )) This is inconvenient and inconsistent with how filter works, as shown. When's init method can be modified to allow similar input as filter. Code is in a branch in my repo, but as this is my first time contributing to Django, I want to make sure I open a ticket and get feedback first. Also I wasn't sure how to classify this. I wasn't sure if it was a bug, as I wasn't sure if it was designed this way.

conditional = False      def __init__(self, condition=None, then=None, **lookups):        if lookups and condition is None:            condition, lookups = Q(**lookups), None        if lookups:            if condition is None:                condition, lookups = Q(**lookups), None            elif getattr(condition, 'conditional', False):                condition, lookups = Q(condition, **lookups), None         if condition is None or not getattr(condition, 'conditional', False) or lookups:             raise TypeError(                 'When() supports a Q object, a boolean expression, or lookups '


Question: Check that and UniqueConstraint.condition don't span joins. Description (last modified by Mariusz Felisiak) Similar to but for check and condition. Not everyone is familiar with the fact database level constraint cannot span across tables and might be tempted to do class Person(models.Model): age = models.PositiveSmallIntegerField() parent = models.ForeignKey(self) class Meta: constraints = { CheckConstraint( name='ageltparent', check=Q(agelt=parentage) ), } Which we'll happily create migrations for but we'll then crash because we prevent JOINs when resolving check.

from django.db.models.functions import Coalesce from django.db.models.manager import Manager from django.db.models.options import Optionsfrom django.db.models.query import Qfrom django.db.models.query import F, Q from django.db.models.signals import (     class_prepared, post_init, post_save, pre_init, pre_save, )


Question: Check that and UniqueConstraint.condition don't span joins. Description (last modified by Mariusz Felisiak) Similar to but for check and condition. Not everyone is familiar with the fact database level constraint cannot span across tables and might be tempted to do class Person(models.Model): age = models.PositiveSmallIntegerField() parent = models.ForeignKey(self) class Meta: constraints = { CheckConstraint( name='ageltparent', check=Q(agelt=parentage) ), } Which we'll happily create migrations for but we'll then crash because we prevent JOINs when resolving check.

return errors <ins>    @classmethod    def _get_expr_references(cls, expr):        if isinstance(expr, Q):            for child in expr.children:                if isinstance(child, tuple):                    lookup, value = child                    yield tuple(lookup.split(LOOKUP_SEP))                    yield from cls._get_expr_references(value)                else:                    yield from cls._get_expr_references(child)        elif isinstance(expr, F):            yield tuple(expr.name.split(LOOKUP_SEP))        elif hasattr(expr, 'get_source_expressions'):            for src_expr in expr.get_source_expressions():                yield from cls._get_expr_references(src_expr)</ins>     @classmethod     def _check_constraints(cls, databases):         errors = []


Question: Check that and UniqueConstraint.condition don't span joins. Description (last modified by Mariusz Felisiak) Similar to but for check and condition. Not everyone is familiar with the fact database level constraint cannot span across tables and might be tempted to do class Person(models.Model): age = models.PositiveSmallIntegerField() parent = models.ForeignKey(self) class Meta: constraints = { CheckConstraint( name='ageltparent', check=Q(agelt=parentage) ), } Which we'll happily create migrations for but we'll then crash because we prevent JOINs when resolving check.

id='models.W039',                     )                 )            fields = chain.from_iterable(            fields = set(chain.from_iterable(                 (*constraint.fields, *constraint.include)                 for constraint in cls._meta.constraints if isinstance(constraint, UniqueConstraint)            )            ))            references = set()            for constraint in cls._meta.constraints:                if isinstance(constraint, UniqueConstraint):                    if (                        connection.features.supports_partial_indexes or                        'supports_partial_indexes' not in cls._meta.required_db_features                    ) and isinstance(constraint.condition, Q):                        references.update(cls._get_expr_references(constraint.condition))                elif isinstance(constraint, CheckConstraint):                    if (                        connection.features.supports_table_check_constraints or                        'supports_table_check_constraints' not in cls._meta.required_db_features                    ) and isinstance(constraint.check, Q):                        references.update(cls._get_expr_references(constraint.check))            for field_name, *lookups in references:                # pk is an alias that won't be found by opts.get_field.                if field_name != 'pk':                    fields.add(field_name)                if not lookups:                    # If it has no lookups it cannot result in a JOIN.                    continue                try:                    if field_name == 'pk':                        field = cls._meta.pk                    else:                        field = cls._meta.get_field(field_name)                    if not field.is_relation or field.many_to_many or field.one_to_many:                        continue                except FieldDoesNotExist:                    continue                # JOIN must happen at the first lookup.                first_lookup = lookups[0]                if (                    field.get_transform(first_lookup) is None and                    field.get_lookup(first_lookup) is None                ):                    errors.append(                        checks.Error(                            "'constraints' refers to the joined field '%s'."                            % LOOKUP_SEP.join([field_name] + lookups),                            obj=cls,                            id='models.E041',                        )                    )             errors.extend(cls._check_local_fields(fields, 'constraints'))         return errors


Question: Geodjango js template should use for float values to avoid DECIMALSEPARATOR ruin the js syntax Description should use |safe on float values to avoid DECIMALSEPARATOR (and probably other settings in this category) ruin the js syntax by adding unexpected characters instead of dot.

elif isinstance(value, bool):  # Make sure booleans don't get treated as numbers         return str(value)     elif isinstance(value, (decimal.Decimal, float, int)):<ins>        if use_l10n is False:            return str(value)</ins>         return number_format(value, use_l10n=use_l10n)     elif isinstance(value, datetime.datetime):         return date_format(value, 'DATETIME_FORMAT', use_l10n=use_l10n)


Question: orderby() with expressions crashes on union() querysets. Description (last modified by Mariusz Felisiak) I have read the recent tickets about unions and orderby (, , ) , and my bug is slightly different, so I hope it's not a duplicate. Let's consider two similar models: class EntityA(models.Model): namea = models.CharField(maxlength=128, null=True) dta = models.DateTimeField(null=True) class EntityB(models.Model): nameb = models.CharField(maxlength=128, null=True) dtb = models.DateTimeField(null=True) (namea="a") (namea="qwerty", ()) (nameb="random", ()) (nameb="b") qsa = (name=F("namea"), dt=F("dta")) qsb = (name=F("nameb"), dt=F("dtb")) queryset = (qsb) I can use a simple ORDER BY clause: queryset.orderby("-dt") And everything will work, no problem here. What I actually want is the same query, but with a NULLS LAST Usually the query becomes: queryset.orderby(F("dt").desc(nulls_last=True)) but that raises a DatabaseError: ORDER BY term does not match any column in the result set. I know unions can handle only a few clauses, but ORDER BY is one of them, so I'm unsure whether this is the expected behaviour or not. If it's expected, then the raised exception could be more explicit.

from django.core.exceptions import EmptyResultSet, FieldError from django.db import DatabaseError, NotSupportedError from django.db.models.constants import LOOKUP_SEPfrom django.db.models.expressions import OrderBy, Random, RawSQL, Ref, Valuefrom django.db.models.expressions import F, OrderBy, Random, RawSQL, Ref, Value from django.db.models.functions import Cast from django.db.models.query_utils import Q, select_related_descend from django.db.models.sql.constants import (


Question: orderby() with expressions crashes on union() querysets. Description (last modified by Mariusz Felisiak) I have read the recent tickets about unions and orderby (, , ) , and my bug is slightly different, so I hope it's not a duplicate. Let's consider two similar models: class EntityA(models.Model): namea = models.CharField(maxlength=128, null=True) dta = models.DateTimeField(null=True) class EntityB(models.Model): nameb = models.CharField(maxlength=128, null=True) dtb = models.DateTimeField(null=True) (namea="a") (namea="qwerty", ()) (nameb="random", ()) (nameb="b") qsa = (name=F("namea"), dt=F("dta")) qsb = (name=F("nameb"), dt=F("dtb")) queryset = (qsb) I can use a simple ORDER BY clause: queryset.orderby("-dt") And everything will work, no problem here. What I actually want is the same query, but with a NULLS LAST Usually the query becomes: queryset.orderby(F("dt").desc(nulls_last=True)) but that raises a DatabaseError: ORDER BY term does not match any column in the result set. I know unions can handle only a few clauses, but ORDER BY is one of them, so I'm unsure whether this is the expected behaviour or not. If it's expected, then the raised exception could be more explicit.

resolved = expr.resolve_expression(self.query, allow_joins=True, reuse=None)             if self.query.combinator:                 src = resolved.get_source_expressions()[0]                expr_src = expr.get_source_expressions()[0]                 # Relabel order by columns to raw numbers if this is a combined                 # query; necessary since the columns can't be referenced by the                 # fully qualified name and the simple column names may collide.                 for idx, (sel_expr, _, col_alias) in enumerate(self.select):                     if is_ref and col_alias == src.refs:                         src = src.source                    elif col_alias:                    elif col_alias and not (                        isinstance(expr_src, F) and col_alias == expr_src.name                    ):                         continue                     if src == sel_expr:                         resolved.set_source_expressions([RawSQL('%d' % (idx + 1), ())])


Question: ().delete() subquery usage performance regression Description Lock tests are failing with Django-MySQL on Django 3.1:  . The tests run ().delete(). Django 3.0 generates this SQL: DELETE FROM Django 3.1 generates this SQL: DELETE FROM WHERE . IN (SELECT . FROM ) The subquery is a blocker for using LOCK TABLES along with delete() - as per the mysql docs: You cannot refer to a locked table multiple times in a single query using the same name. Use aliases instead, and obtain a separate lock for the table and each alias: mysqlLOCK TABLE t WRITE, t AS t1 READ; mysqlINSERT INTO t SELECT FROM t; ERROR 1100: Table 't' was not locked with LOCK TABLES mysqlINSERT INTO t SELECT FROM t AS t1; Since there's no alias on the subquery, there's no way to lock it. Additionally this change is a performance regression. I benchmarked with MariaDB 10.3 and filling an InnoDB a table of 100k rows via the [sequence storage engine ]. Using the old DELETE FROM, deletion takes 0.2 seconds: root [19]create table t(c int primary key); Query OK, 0 rows affected (0.079 sec) root [16]insert into t select from seq1to100000; Query OK, rows affected (0.252 sec) Records: Duplicates: 0 Warnings: 0 root [17]delete from t; Query OK, rows affected (0.200 sec) Using DELETE FROM WHERE id IN (...), deletion takes 7.5 seconds: root [18]drop table t; Query OK, 0 rows affected (0.008 sec) root [19]create table t(c int primary key); Query OK, 0 rows affected (0.079 sec) root [20]insert into t select from seq1to_100000; Query OK, rows affected (0.594 sec) Records: Duplicates: 0 Warnings: 0 root [21]delete from t where c in (select c from t); Query OK, rows affected (7.543 sec) root [22]drop table t; Query OK, 0 rows affected (0.013 sec) Yes in theory the optimizer should be able to find this, and it may even be fixed in later MySQL/MariaDB versions. But I think we shouldn't change the SQL here.

class SQLDeleteCompiler(SQLCompiler):     @cached_property     def single_alias(self):<ins>        # Ensure base table is in aliases.        self.query.get_initial_alias()</ins>         return sum(self.query.alias_refcount[t] > 0 for t in self.query.alias_map) == 1      def _as_sql(self, query):


Question: Add hint to the E410 about AuthenticationMiddleware. Description Given an empty MIDDLEWARE in your project, you try and run the admin. The system checks framework kicks in, giving you: (admin.E408) '' must be in MIDDLEWARE in order to use the admin application. (admin.E409) 'django.contrib.messages.middleware.MessageMiddleware' must be in MIDDLEWARE in order to use the admin application. (admin.E410) 'django.contrib.sessions.middleware.SessionMiddleware' must be in MIDDLEWARE in order to use the admin application. You add each of those middlewares to the stack, and run the application: File "", line 23, in processrequest ) % ("CLASSES" if settings.MIDDLEWARE is None else "") AssertionError: The Django authentication middleware requires session middleware to be installed. Edit your MIDDLEWARE setting to insert 'django.contrib.sessions.middleware.SessionMiddleware' before '' Ignoring the fact that it should be a system check, and it's an unfortunate side-effect of spanning 2 separate contrib apps that it isn't, if the order of the errors output was updated, we'd not see this exception in this specific scenario.

errors.append(checks.Error(             "'django.contrib.sessions.middleware.SessionMiddleware' must "             "be in MIDDLEWARE in order to use the admin application.",<ins>            hint=(                "Insert "                "'django.contrib.sessions.middleware.SessionMiddleware' "                "before "                "'django.contrib.auth.middleware.AuthenticationMiddleware'."            ),</ins>             id='admin.E410',         ))     return errors


Question: Make () strip dashes and underscores Description (last modified by Elinaldo do Nascimento Monteiro) Bug generation slug Example: from import text text.slugify("This is a test ---") output: this-is-a-test- Improvement after correction from import text text.slugify("_This is a test ---") output: this-is-a-test PR

@keep_lazy_text def slugify(value, allow_unicode=False):     """    Convert to ASCII if 'allow_unicode' is False. Convert spaces to hyphens.    Remove characters that aren't alphanumerics, underscores, or hyphens.    Convert to lowercase. Also strip leading and trailing whitespace.    Convert to ASCII if 'allow_unicode' is False. Convert spaces or repeated    dashes to single dashes. Remove characters that aren't alphanumerics,    underscores, or hyphens. Convert to lowercase. Also strip leading and    trailing whitespace, dashes, and underscores.     """     value = str(value)     if allow_unicode:         value = unicodedata.normalize('NFKC', value)     else:         value = unicodedata.normalize('NFKD', value).encode('ascii', 'ignore').decode('ascii')    value = re.sub(r'[^\w\s-]', '', value.lower()).strip()    return re.sub(r'[-\s]+', '-', value)    value = re.sub(r'[^\w\s-]', '', value.lower())    return re.sub(r'[-\s]+', '-', value).strip('-_')   def camel_case_to_spaces(value):


Question: Constant expressions of an ExpressionWrapper object are incorrectly placed at the GROUP BY clause Description I have a function that expects an arbitrary Query expression and constructs a query on a Postgres db def execQuery(expr): expr = ExpressionWrapper(expr, outputfield=IntegerField()) return Model.objects.annotate(exprres=expr).values('exprres', 'columna').annotate(sum=Sum('columnb')) However, when the given expr is a constant expression (e.g., Value(3)), Django generates an SQL query that contains this constant expression in its GROUP BY clause. SELECT "model"."columna", 3 AS "exprres", SUM("model"."columnb") AS "sum" FROM "model" GROUP BY "model"."columna", 3 This leads to an exception because in Postgres, the query above is invalid: aggregate functions are not allowed in GROUP BY LINE 1: SELECT "model"."columna", 3 AS "exprres", SUM("model"."col... Note that when the given query expression is not wrapped by the ExpressionWrapper object, Django correctly identifies and omits the constant from the GROUP BY clause. For example, the query below runs correctly. def execQuery(expr): return Model.objects.annotate(exprres=Value(3, outputfield=IntegerField())).values('exprres', 'columna').annotate(sum=Sum('columnb')) SELECT "model"."columna", 3 AS "exprres", SUM("model"."columnb") AS "sum" FROM "model" GROUP BY "model"."columna"

def get_source_expressions(self):         return [self.expression] <ins>    def get_group_by_cols(self, alias=None):        return self.expression.get_group_by_cols(alias=alias)</ins>     def as_sql(self, compiler, connection):         return self.expression.as_sql(compiler, connection)


Question: Memcached key validation raises InvalidCacheKey with clunky message. Description On Django 2.2.13 the code for memcachekeywarnings in has a bad format string that results in raising an exception rather than just producing a warning. This can be reproduced with a memcached key with a space in it, e.g. "foo bar". This code was present before the 2.2.13 release, but becomes more exposed with that release, since it begins validating cache keys. I think it's as simple as removing the , CacheKeyWarning.

if ord(char) < 33 or ord(char) == 127:             yield (                 'Cache key contains characters that will cause errors if '                'used with memcached: %r' % key, CacheKeyWarning                'used with memcached: %r' % key             )             break


Question: DecimalField.topython() raises TypeError on dict values. Description A call to DecimalField.topython() with a dictionary as the value parameter produces TypeError instead of ValidationError. This is a problem, for example, when you try to save a model object, and a decimal field got set to a dictionary by mistake. The TypeError exception that comes back makes it hard to track the problem to the field if the object has a lot of fields. I am proposing a patch to fix it:

return self.context.create_decimal_from_float(value)         try:             return decimal.Decimal(value)        except decimal.InvalidOperation:        except (decimal.InvalidOperation, TypeError, ValueError):             raise exceptions.ValidationError(                 self.error_messages['invalid'],                 code='invalid',


Question: Queryset raises NotSupportedError when RHS has filterable=False attribute. Description (last modified by Nicolas Baccelli) I'm migrating my app to django 3.0.7 and I hit a strange behavior using a model class with a field labeled filterable class ProductMetaDataType(models.Model): label = models.CharField(maxlength=255, unique=True, blank=False, null=False) filterable = models.BooleanField(default=False, verbosename=("filterable")) class Meta: applabel = "adminpricing" verbosename = ("product meta data type") verbosenameplural = ("product meta data types") def str(self): return class ProductMetaData(models.Model): id = models.BigAutoField(primarykey=True) product = models.ForeignKey( Produit, null=False, blank=False, ondelete=models.CASCADE ) value = models.TextField(null=False, blank=False) marketplace = models.ForeignKey( Plateforme, null=False, blank=False, ondelete=models.CASCADE ) datecreated = models.DateTimeField(null=True, ) metadatatype = models.ForeignKey( ProductMetaDataType, null=False, blank=False, ondelete=models.CASCADE ) class Meta: applabel = "adminpricing" verbosename = ("product meta data") verbosenameplural = ("product meta datas") Error happened when filtering ProductMetaData with a metadatatype : (value="Dark Vador", metadatatype=self.brandmetadata) Error traceback : Traceback (most recent call last): File "", line 481, in testcheckpolicywarningbyfields for p in ( File "/usr/local/lib/python3.8/site-", line 82, in managermethod return getattr(self.getqueryset(), name)(args, kwargs) File "/usr/local/lib/python3.8/site-", line 904, in filter return self.filterorexclude(False, args, kwargs) File "/usr/local/lib/python3.8/site-", line 923, in filterorexclude (Q(args, *kwargs)) File "/usr/local/lib/python3.8/site-", line 1351, in addq clause, = (qobject, self.usedaliases) File "/usr/local/lib/python3.8/site-", line 1378, in addq childclause, neededinner = self.buildfilter( File "/usr/local/lib/python3.8/site-", line 1264, in buildfilter self.checkfilterable(value) File "/usr/local/lib/python3.8/site-", line 1131, in checkfilterable raise NotSupportedError( ProductMetaDataType is disallowed in the filter clause. I changed label to filterabletest and it fixed this issue This should be documented or fix.

def check_filterable(self, expression):         """Raise an error if expression cannot be used in a WHERE clause."""        if not getattr(expression, 'filterable', True):        if (            hasattr(expression, 'resolve_expression') and            not getattr(expression, 'filterable', True)        ):             raise NotSupportedError(                 expression.__class__.__name__ + ' is disallowed in the filter '                 'clause.'


Question: Avoid passing NULL to the IN lookup Description Currently prefetchrelated on a FK passes the NULL through to the database for e.g. authorid IN (NULL, 2). Passing NULL is always unnecessary, since it's not allowed in FK's. There's a small risk from passing NULL that it could lead to incorrect with complex prefetch querysets using PK refs because of NULL's weirdness in SQL. For example with these models: from import models class Author(models.Model): pass class Book(models.Model): author = models.ForeignKey(Author, null=True, ondelete=models.DONOTHING) Prefetching authors on Books, when at least one Book has author=None, uses IN (..., NULL, ...) in the query: In [1]: from import Author, Book In [2]: a1 = () In [3]: (author=a1) Out[3]: <Book: Book object (3)In [4]: (author=None) Out[4]: <Book: Book object (4)In [5]: Book.objects.prefetchrelated('author') Out[5]: <QuerySet [<Book: Book object (3), <Book: Book object (4)]In [6]: from import connection In [7]: connection.queries Out[7]: [{'sql': 'INSERT INTO "coreauthor" ("id") VALUES (NULL)', 'time': '0.001'}, {'sql': 'INSERT INTO "corebook" ("authorid") VALUES (2)', 'time': '0.001'}, {'sql': 'INSERT INTO "corebook" ("authorid") VALUES (NULL)', 'time': '0.001'}, {'sql': 'SELECT "corebook"."id", "corebook"."authorid" FROM "corebook" LIMIT 21', 'time': '0.000'}, {'sql': 'SELECT "coreauthor"."id" FROM "coreauthor" WHERE "coreauthor"."id" IN (NULL, 2)', 'time': '0.000'}] Maybe this could generally be extended to use of _in with non-nullable fields?

)          if self.rhs_is_direct_value():            # Remove None from the list as NULL is never equal to anything.             try:                 rhs = OrderedSet(self.rhs)                rhs.discard(None)             except TypeError:  # Unhashable items in self.rhs                rhs = self.rhs                rhs = [r for r in self.rhs if r is not None]              if not rhs:                 raise EmptyResultSet


Question: Improve makemigrations warning message when calling without an active database connection. Description I was looking at the gis install instructions and I came across an error when running makemigrations. (Error is I had not entered password correctly). However, the error message that is generated is a bit odd, it has a full stop on a new line and shows ( I was also able to get the same error message on a clean project, with a postgres database (not gis) and an incorrect password. I'm not sure if this is a 'bug' but it doesn't look quite right? (gis) PS C:\Users\smith\gis\geodjangopython .\ makemigrations c:\users\smith\pycharmprojects\django2\django\core\management\commands\ RuntimeWarning: Got an error checking a consistent migration history performed for database connection 'default': fe_sendauth: no password supplied . ( No changes detected (gis) PS C:\Users\smith\gis\geodjango

except OperationalError as error:                     warnings.warn(                         "Got an error checking a consistent migration history "                        "performed for database connection '%s': %s."                        "performed for database connection '%s': %s"                         % (alias, error),                         RuntimeWarning,                     )


Question: Self referencing foreign key doesn't correctly order by a relation "id" field. Description Initially discovered on 2.2.10 but verified still happens on 3.0.6. Given the following models: class OneModel(models.Model): class Meta: ordering = ("-id",) id = models.BigAutoField(primarykey=True) root = models.ForeignKey("OneModel", ondelete=models.CASCADE, null=True) oneval = models.BigIntegerField(null=True) class TwoModel(models.Model): id = models.BigAutoField(primarykey=True) record = models.ForeignKey(OneModel, ondelete=models.CASCADE) twoval = models.BigIntegerField(null=True) The following queryset gives unexpected results and appears to be an incorrect SQL query: qs = (recordonevalin=[1,2,3]) qs = qs.orderby("recordrootid") print() SELECT "oriontwomodel"."id", "oriontwomodel"."recordid", "oriontwomodel"."twoval" FROM "oriontwomodel" INNER JOIN "oriononemodel" ON ("oriontwomodel"."recordid" = "oriononemodel"."id") LEFT OUTER JOIN "oriononemodel" T3 ON ("oriononemodel"."rootid" = T3."id") WHERE "oriononemodel"."oneval" IN (1, 2, 3) ORDER BY T3."id" DESC The query has an unexpected DESCENDING sort. That appears to come from the default sort order on the OneModel class, but I would expect the orderby() to take prececence. The the query has two JOINS, which is unnecessary. It appears that, since is a foreign key to itself, that is causing it to do the unnecessary extra join. In fact, testing a model where root is a foreign key to a third model doesn't show the problem behavior. Note also that the queryset with orderby("recordroot") gives the exact same SQL. This queryset gives correct results and what looks like a pretty optimal SQL: qs = (recordonevalin=[1,2,3]) qs = qs.orderby("recordrootid") print() SELECT "oriontwomodel"."id", "oriontwomodel"."recordid", "oriontwomodel"."twoval" FROM "oriontwomodel" INNER JOIN "oriononemodel" ON ("oriontwomodel"."recordid" = "oriononemodel"."id") WHERE "oriononemodel"."oneval" IN (1, 2, 3) ORDER BY "oriononemodel"."rootid" ASC So is this a potential bug or a misunderstanding on my part? Another queryset that works around the issue and gives a reasonable SQL query and expected results: qs = (recordonevalin=[1,2,3]) qs = qs.annotate(rootid=F("recordrootid")) qs = qs.orderby("rootid") print() SELECT "oriontwomodel"."id", "oriontwomodel"."recordid", "oriontwomodel"."twoval" FROM "oriontwomodel" INNER JOIN "oriononemodel" ON ("oriontwomodel"."recordid" = "oriononemodel"."id") WHERE "oriononemodel"."oneval" IN (1, 2, 3) ORDER BY "oriononemodel"."zeroid" ASC ASCENDING sort, and a single INNER JOIN, as I'd expect. That actually works for my use because I need that output column anyway. One final oddity; with the original queryset but the inverted sort orderby(): qs = (recordonevalin=[1,2,3]) qs = qs.orderby("-recordrootid") print() SELECT "oriontwomodel"."id", "oriontwomodel"."recordid", "oriontwomodel"."twoval" FROM "oriontwomodel" INNER JOIN "oriononemodel" ON ("oriontwomodel"."recordid" = "oriononemodel"."id") LEFT OUTER JOIN "oriononemodel" T3 ON ("oriononemodel"."rootid" = T3."id") WHERE "oriononemodel"."oneval" IN (1, 2, 3) ORDER BY T3."id" ASC One gets the query with the two JOINs but an ASCENDING sort order. I was not under the impression that sort orders are somehow relative to the class level sort order, eg: does specifing orderby("-record_rootid") invert the class sort order? Testing that on a simple case doesn't show that behavior at all. Thanks for any assistance and clarification.

# If we get to this point and the field is a relation to another model,         # append the default ordering for that model unless it is the pk         # shortcut or the attribute name of the field that is specified.        if field.is_relation and opts.ordering and getattr(field, 'attname', None) != name and name != 'pk':        if (            field.is_relation and            opts.ordering and            getattr(field, 'attname', None) != pieces[-1] and            name != 'pk'        ):             # Firstly, avoid infinite loops.             already_seen = already_seen or set()             join_tuple = tuple(getattr(self.query.alias_map[j], 'join_cols', None) for j in joins)


Question: Argon2id should be supported and become the default variety for Argon2PasswordHasher Description (last modified by Si Feng) There were three important changes in the upstream argon2-cffi library since Django 1.10 was released with Argon2 support: (Nov 10, 2016) argon2id support was :  (Apr 9, 2018) argon2id became its default type:  (Aug 18, 2018) its hasher's default memory cost changed from 512 to , and parallelism from 2 to 8, per RFC draft recommendations:  When Django 1.10 was released, only argon2d and argon2i were available, hence the hard-coded argon2i variety in Argon2PasswordHasher. Though Django does not use its hasher, the memorycost = 512 and parallelism = 2 in Argon2PasswordHasher were simply copied from argon2-cffi's hasher back then. Now we should sync Django with upstream updates.

library = 'argon2'      time_cost = 2    memory_cost = 512    parallelism = 2    memory_cost = 102400    parallelism = 8      def encode(self, password, salt):         argon2 = self._load_library()        params = self.params()         data = argon2.low_level.hash_secret(             password.encode(),             salt.encode(),            time_cost=self.time_cost,            memory_cost=self.memory_cost,            parallelism=self.parallelism,            hash_len=argon2.DEFAULT_HASH_LENGTH,            type=argon2.low_level.Type.I,            time_cost=params.time_cost,            memory_cost=params.memory_cost,            parallelism=params.parallelism,            hash_len=params.hash_len,            type=params.type,         )         return self.algorithm + data.decode('ascii')


Question: Argon2id should be supported and become the default variety for Argon2PasswordHasher Description (last modified by Si Feng) There were three important changes in the upstream argon2-cffi library since Django 1.10 was released with Argon2 support: (Nov 10, 2016) argon2id support was :  (Apr 9, 2018) argon2id became its default type:  (Aug 18, 2018) its hasher's default memory cost changed from 512 to , and parallelism from 2 to 8, per RFC draft recommendations:  When Django 1.10 was released, only argon2d and argon2i were available, hence the hard-coded argon2i variety in Argon2PasswordHasher. Though Django does not use its hasher, the memorycost = 512 and parallelism = 2 in Argon2PasswordHasher were simply copied from argon2-cffi's hasher back then. Now we should sync Django with upstream updates.

algorithm, rest = encoded.split('$', 1)         assert algorithm == self.algorithm         try:            return argon2.low_level.verify_secret(                ('$' + rest).encode('ascii'),                password.encode(),                type=argon2.low_level.Type.I,            )            return argon2.PasswordHasher().verify('$' + rest, password)         except argon2.exceptions.VerificationError:             return False


Question: Argon2id should be supported and become the default variety for Argon2PasswordHasher Description (last modified by Si Feng) There were three important changes in the upstream argon2-cffi library since Django 1.10 was released with Argon2 support: (Nov 10, 2016) argon2id support was :  (Apr 9, 2018) argon2id became its default type:  (Aug 18, 2018) its hasher's default memory cost changed from 512 to , and parallelism from 2 to 8, per RFC draft recommendations:  When Django 1.10 was released, only argon2d and argon2i were available, hence the hard-coded argon2i variety in Argon2PasswordHasher. Though Django does not use its hasher, the memorycost = 512 and parallelism = 2 in Argon2PasswordHasher were simply copied from argon2-cffi's hasher back then. Now we should sync Django with upstream updates.

}      def must_update(self, encoded):        (algorithm, variety, version, time_cost, memory_cost, parallelism,            salt, data) = self._decode(encoded)        algorithm, rest = encoded.split('$', 1)         assert algorithm == self.algorithm         argon2 = self._load_library()        return (            argon2.low_level.ARGON2_VERSION != version or            self.time_cost != time_cost or            self.memory_cost != memory_cost or            self.parallelism != parallelism        )        current_params = argon2.extract_parameters('$' + rest)        new_params = self.params()        # Set salt_len to the salt_len of the current parameters because salt        # is explicitly passed to argon2.        new_params.salt_len = current_params.salt_len        return current_params != new_params      def harden_runtime(self, password, encoded):         # The runtime for Argon2 is too complicated to implement a sensible         # hardening algorithm.         pass     def params(self):        argon2 = self._load_library()        # salt_len is a noop, because we provide our own salt.        return argon2.Parameters(            type=argon2.low_level.Type.ID,            version=argon2.low_level.ARGON2_VERSION,            salt_len=argon2.DEFAULT_RANDOM_SALT_LENGTH,            hash_len=argon2.DEFAULT_HASH_LENGTH,            time_cost=self.time_cost,            memory_cost=self.memory_cost,            parallelism=self.parallelism,        )     def _decode(self, encoded):         """         Split an encoded hash and return: (


Question: Argon2id should be supported and become the default variety for Argon2PasswordHasher Description (last modified by Si Feng) There were three important changes in the upstream argon2-cffi library since Django 1.10 was released with Argon2 support: (Nov 10, 2016) argon2id support was :  (Apr 9, 2018) argon2id became its default type:  (Aug 18, 2018) its hasher's default memory cost changed from 512 to , and parallelism from 2 to 8, per RFC draft recommendations:  When Django 1.10 was released, only argon2d and argon2i were available, hence the hard-coded argon2i variety in Argon2PasswordHasher. Though Django does not use its hasher, the memorycost = 512 and parallelism = 2 in Argon2PasswordHasher were simply copied from argon2-cffi's hasher back then. Now we should sync Django with upstream updates.

parallelism, salt, data,         ).         """        bits = encoded.split('$')        if len(bits) == 5:            # Argon2 < 1.3            algorithm, variety, raw_params, salt, data = bits            version = 0x10        else:            assert len(bits) == 6            algorithm, variety, raw_version, raw_params, salt, data = bits            assert raw_version.startswith('v=')            version = int(raw_version[len('v='):])        params = dict(bit.split('=', 1) for bit in raw_params.split(','))        assert len(params) == 3 and all(x in params for x in ('t', 'm', 'p'))        time_cost = int(params['t'])        memory_cost = int(params['m'])        parallelism = int(params['p'])        argon2 = self._load_library()        algorithm, rest = encoded.split('$', 1)        params = argon2.extract_parameters('$' + rest)        variety, *_, salt, data = rest.split('$')         return (            algorithm, variety, version, time_cost, memory_cost, parallelism,            salt, data,            algorithm, variety, params.version, params.time_cost,            params.memory_cost, params.parallelism, salt, data,         )


Question: django-admin runserver mostly does not work on Windows Description I frequently encounter the problem that running "django-admin runserver" on Windows fails: <some-venv\scripts\ can't open file '<some-venv\test\Scripts\django-admin': [Errno 2] No such file or directory The command works if run with --noreload. There are probably other conditions that must be met for it to fail, or work. I think it makes a difference, for example, if Django was installed with pip or setuptools. The .exe "console scripts " are different between the two; one type has a bit of readable Python at the end, the other does not. What I do know is that the problem is either caused or enabled by the generated console scripts (the .exe wrappers in venv/Scripts) stripping the .exe extension from their own [0] when passing that to Python to run. If I sabotage this by changing the regex in the file itself (where present), or by adding the extension again in before running the command line, it works fine. To be honest, I have no idea whether this is a bug in Django, or pip, or setuptools, or distutils, or something else entirely. I have spent the last two days trying to figure out what exactly goes wrong, and where, with little success. I'm reporting it here because Django is the only place I could find where it is possible to kludge the code into working. I'll be happy to take my business elsewhere if someone can point the way.

on reloading.     """     import django.__main__    django_main_path = Path(django.__main__.__file__)    py_script = Path(sys.argv[0])      args = [sys.executable] + ['-W%s' % o for o in sys.warnoptions]    if sys.argv[0] == django.__main__.__file__:    if py_script == django_main_path:         # The server was started with `python -m django runserver`.         args += ['-m', 'django']         args += sys.argv[1:]    elif not py_script.exists():        # sys.argv[0] may not exist for several reasons on Windows.        # It may exist with a .exe extension or have a -script.py suffix.        exe_entrypoint = py_script.with_suffix('.exe')        if exe_entrypoint.exists():            # Should be executed directly, ignoring sys.executable.            return [exe_entrypoint, *sys.argv[1:]]        script_entrypoint = py_script.with_name('%s-script.py' % py_script.name)        if script_entrypoint.exists():            # Should be executed as usual.            return [*args, script_entrypoint, *sys.argv[1:]]        raise RuntimeError('Script %s does not exist.' % py_script)     else:         args += sys.argv     return args


Question: compilemessages needlessly runs msgfmt on unchanged .po files Description I have a project where running django-admin compilemessages takes 1.75 seconds. Running it again, when all the .mo files already exists and are up-to-date, also takes 1.75 seconds. I propose that is changed so that it only invokes msgfmt when it would do anything useful. This can be implemented by checking the mtime of the .po file and the corresponding .mo file. (If statting the .mo file fails, treat that as if the mtime was 0.) Only submit the command to the executor if the mtime of the .po file is greater than that of the .mo file. In effect: don't do anything if the .mo file is newer than the .po file. There is one issue with this: the way the code currently uses the iswritable function. Since it modifies the mtime of the .mo file, you would have to perform the stat of the .mo file before you check if it is writable. (Or, you could just remove the iswritable function and its use. That feature is, in my opinion, of dubious value, and it doesn't appear to be documented.) After I made the changes above, the runtime in the common case where nothing needs to be done was reduced from 1.75 seconds to 0.2 seconds. (Unfortunately, I doubt that I will be able to get a Corporate Contributor License Agreement signed, so I can unfortunately not contribute my change.) 75 seconds may not be much, but when a CI system does it repeatedly, it adds up.

import concurrent.futures import glob import os<ins>from pathlib import Path</ins>  from django.core.management.base import BaseCommand, CommandError from django.core.management.utils import (


Question: compilemessages needlessly runs msgfmt on unchanged .po files Description I have a project where running django-admin compilemessages takes 1.75 seconds. Running it again, when all the .mo files already exists and are up-to-date, also takes 1.75 seconds. I propose that is changed so that it only invokes msgfmt when it would do anything useful. This can be implemented by checking the mtime of the .po file and the corresponding .mo file. (If statting the .mo file fails, treat that as if the mtime was 0.) Only submit the command to the executor if the mtime of the .po file is greater than that of the .mo file. In effect: don't do anything if the .mo file is newer than the .po file. There is one issue with this: the way the code currently uses the iswritable function. Since it modifies the mtime of the .mo file, you would have to perform the stat of the .mo file before you check if it is writable. (Or, you could just remove the iswritable function and its use. That feature is, in my opinion, of dubious value, and it doesn't appear to be documented.) After I made the changes above, the runtime in the common case where nothing needs to be done was reduced from 1.75 seconds to 0.2 seconds. (Unfortunately, I doubt that I will be able to get a Corporate Contributor License Agreement signed, so I can unfortunately not contribute my change.) 75 seconds may not be much, but when a CI system does it repeatedly, it adds up.

def has_bom(fn):    with open(fn, 'rb') as f:    with fn.open('rb') as f:         sample = f.read(4)     return sample.startswith((codecs.BOM_UTF8, codecs.BOM_UTF16_LE, codecs.BOM_UTF16_BE))


Question: compilemessages needlessly runs msgfmt on unchanged .po files Description I have a project where running django-admin compilemessages takes 1.75 seconds. Running it again, when all the .mo files already exists and are up-to-date, also takes 1.75 seconds. I propose that is changed so that it only invokes msgfmt when it would do anything useful. This can be implemented by checking the mtime of the .po file and the corresponding .mo file. (If statting the .mo file fails, treat that as if the mtime was 0.) Only submit the command to the executor if the mtime of the .po file is greater than that of the .mo file. In effect: don't do anything if the .mo file is newer than the .po file. There is one issue with this: the way the code currently uses the iswritable function. Since it modifies the mtime of the .mo file, you would have to perform the stat of the .mo file before you check if it is writable. (Or, you could just remove the iswritable function and its use. That feature is, in my opinion, of dubious value, and it doesn't appear to be documented.) After I made the changes above, the runtime in the common case where nothing needs to be done was reduced from 1.75 seconds to 0.2 seconds. (Unfortunately, I doubt that I will be able to get a Corporate Contributor License Agreement signed, so I can unfortunately not contribute my change.) 75 seconds may not be much, but when a CI system does it repeatedly, it adds up.

with concurrent.futures.ThreadPoolExecutor() as executor:             futures = []             for i, (dirpath, f) in enumerate(locations):                po_path = Path(dirpath) / f                mo_path = po_path.with_suffix('.mo')                try:                    if mo_path.stat().st_mtime >= po_path.stat().st_mtime:                        if self.verbosity > 0:                            self.stdout.write(                                'File “%s” is already compiled and up to date.'                                % po_path                            )                        continue                except FileNotFoundError:                    pass                 if self.verbosity > 0:                     self.stdout.write('processing file %s in %s' % (f, dirpath))                po_path = os.path.join(dirpath, f)                 if has_bom(po_path):                     self.stderr.write(                         'The %s file has a BOM (Byte Order Mark). Django only '


Question: compilemessages needlessly runs msgfmt on unchanged .po files Description I have a project where running django-admin compilemessages takes 1.75 seconds. Running it again, when all the .mo files already exists and are up-to-date, also takes 1.75 seconds. I propose that is changed so that it only invokes msgfmt when it would do anything useful. This can be implemented by checking the mtime of the .po file and the corresponding .mo file. (If statting the .mo file fails, treat that as if the mtime was 0.) Only submit the command to the executor if the mtime of the .po file is greater than that of the .mo file. In effect: don't do anything if the .mo file is newer than the .po file. There is one issue with this: the way the code currently uses the iswritable function. Since it modifies the mtime of the .mo file, you would have to perform the stat of the .mo file before you check if it is writable. (Or, you could just remove the iswritable function and its use. That feature is, in my opinion, of dubious value, and it doesn't appear to be documented.) After I made the changes above, the runtime in the common case where nothing needs to be done was reduced from 1.75 seconds to 0.2 seconds. (Unfortunately, I doubt that I will be able to get a Corporate Contributor License Agreement signed, so I can unfortunately not contribute my change.) 75 seconds may not be much, but when a CI system does it repeatedly, it adds up.

)                     self.has_errors = True                     continue                base_path = os.path.splitext(po_path)[0]                  # Check writability on first location                if i == 0 and not is_writable(base_path + '.mo'):                if i == 0 and not is_writable(mo_path):                     self.stderr.write(                         'The po files under %s are in a seemingly not writable location. '                         'mo files will not be updated/created.' % dirpath


Question: compilemessages needlessly runs msgfmt on unchanged .po files Description I have a project where running django-admin compilemessages takes 1.75 seconds. Running it again, when all the .mo files already exists and are up-to-date, also takes 1.75 seconds. I propose that is changed so that it only invokes msgfmt when it would do anything useful. This can be implemented by checking the mtime of the .po file and the corresponding .mo file. (If statting the .mo file fails, treat that as if the mtime was 0.) Only submit the command to the executor if the mtime of the .po file is greater than that of the .mo file. In effect: don't do anything if the .mo file is newer than the .po file. There is one issue with this: the way the code currently uses the iswritable function. Since it modifies the mtime of the .mo file, you would have to perform the stat of the .mo file before you check if it is writable. (Or, you could just remove the iswritable function and its use. That feature is, in my opinion, of dubious value, and it doesn't appear to be documented.) After I made the changes above, the runtime in the common case where nothing needs to be done was reduced from 1.75 seconds to 0.2 seconds. (Unfortunately, I doubt that I will be able to get a Corporate Contributor License Agreement signed, so I can unfortunately not contribute my change.) 75 seconds may not be much, but when a CI system does it repeatedly, it adds up.

self.has_errors = True                     return                 args = [self.program] + self.program_options + [                    '-o', base_path + '.mo', base_path + '.po'                ]                args = [self.program, *self.program_options, '-o', mo_path, po_path]                 futures.append(executor.submit(popen_wrapper, args))              for future in concurrent.futures.as_completed(futures):


Question: sometimes fails with 'NoneType' object is not subscriptable Description (last modified by Guillermo Bonvehí) I'm sporadically getting some cache errors using database backend. The error is: 'NoneType' object is not subscriptable And the backtrace: /usr/local/lib/python3.7/site-→ getresponse /usr/local/lib/python3.7/site-→ render /usr/local/lib/python3.7/site-→ callback /usr/local/lib/python3.7/site-→ processresponse /usr/local/lib/python3.7/site-→ learncachekey /usr/local/lib/python3.7/site-→ set /usr/local/lib/python3.7/site-→ baseset /usr/local/lib/python3.7/site-→ cull This is using Django 2.2.11 but I see the same code is in master.  cursor.execute( () % table, [cullnum]) cursor.execute("DELETE FROM %s " "WHERE cachekey < %%s" % table, [cursor.fetchone()[0]]) From what I can understand, the cursor after running () command is not returning any data, so cursor.fetchone()[0] afterwards fails. I guess a simple check to see if it contains data would be enough, may apply for an easy picking. Edit: Wording

cursor.execute(                     connection.ops.cache_key_culling_sql() % table,                     [cull_num])                cursor.execute("DELETE FROM %s "                               "WHERE cache_key < %%s" % table,                               [cursor.fetchone()[0]])                last_cache_key = cursor.fetchone()                if last_cache_key:                    cursor.execute(                        'DELETE FROM %s WHERE cache_key < %%s' % table,                        [last_cache_key[0]],                    )      def clear(self):         db = router.db_for_write(self.cache_model_class)


Question: formsets with candelete=True shouldn't add delete field to extra forms Description Current behavior of formsets with candelete=True is to add a delete field to every form. This behavior differs from that expected, however (why would one want a delete option on an "add" form?), as well as that of the builtin admin. I've included a patch on , but haven't bothered with patching tests yet.

def add_fields(self, form, index):         """A hook for adding extra fields on to each form instance."""        initial_form_count = self.initial_form_count()         if self.can_order:             # Only pre-fill the ordering field for initial forms.            if index is not None and index < self.initial_form_count():            if index is not None and index < initial_form_count:                 form.fields[ORDERING_FIELD_NAME] = IntegerField(                     label=_('Order'),                     initial=index + 1,


Question: formsets with candelete=True shouldn't add delete field to extra forms Description Current behavior of formsets with candelete=True is to add a delete field to every form. This behavior differs from that expected, however (why would one want a delete option on an "add" form?), as well as that of the builtin admin. I've included a patch on , but haven't bothered with patching tests yet.

required=False,                     widget=self.get_ordering_widget(),                 )        if self.can_delete:        if self.can_delete and (self.can_delete_extra or index < initial_form_count):             form.fields[DELETION_FIELD_NAME] = BooleanField(label=_('Delete'), required=False)      def add_prefix(self, index):


Question: formsets with candelete=True shouldn't add delete field to extra forms Description Current behavior of formsets with candelete=True is to add a delete field to every form. This behavior differs from that expected, however (why would one want a delete option on an "add" form?), as well as that of the builtin admin. I've included a patch on , but haven't bothered with patching tests yet.

def formset_factory(form, formset=BaseFormSet, extra=1, can_order=False,                     can_delete=False, max_num=None, validate_max=False,                    min_num=None, validate_min=False, absolute_max=None):                    min_num=None, validate_min=False, absolute_max=None,                    can_delete_extra=True):     """Return a FormSet for the given form class."""     if min_num is None:         min_num = DEFAULT_MIN_NUM


Question: formsets with candelete=True shouldn't add delete field to extra forms Description Current behavior of formsets with candelete=True is to add a delete field to every form. This behavior differs from that expected, however (why would one want a delete option on an "add" form?), as well as that of the builtin admin. I've included a patch on , but haven't bothered with patching tests yet.

'extra': extra,         'can_order': can_order,         'can_delete': can_delete,        'can_delete_extra': can_delete_extra,         'min_num': min_num,         'max_num': max_num,         'absolute_max': absolute_max,diff --git a/django/forms/models.py b/django/forms/models.py-- a/django/forms/models.py++ b/django/forms/models.py


Question: formsets with candelete=True shouldn't add delete field to extra forms Description Current behavior of formsets with candelete=True is to add a delete field to every form. This behavior differs from that expected, however (why would one want a delete option on an "add" form?), as well as that of the builtin admin. I've included a patch on , but haven't bothered with patching tests yet.

widgets=None, validate_max=False, localized_fields=None,                          labels=None, help_texts=None, error_messages=None,                          min_num=None, validate_min=False, field_classes=None,                         absolute_max=None):                         absolute_max=None, can_delete_extra=True):     """Return a FormSet class for the given Django model class."""     meta = getattr(form, 'Meta', None)     if (getattr(meta, 'fields', fields) is None and


Question: formsets with candelete=True shouldn't add delete field to extra forms Description Current behavior of formsets with candelete=True is to add a delete field to every form. This behavior differs from that expected, however (why would one want a delete option on an "add" form?), as well as that of the builtin admin. I've included a patch on , but haven't bothered with patching tests yet.

FormSet = formset_factory(form, formset, extra=extra, min_num=min_num, max_num=max_num,                               can_order=can_order, can_delete=can_delete,                               validate_min=validate_min, validate_max=validate_max,                              absolute_max=absolute_max)                              absolute_max=absolute_max, can_delete_extra=can_delete_extra)     FormSet.model = model     return FormSet


Question: formsets with candelete=True shouldn't add delete field to extra forms Description Current behavior of formsets with candelete=True is to add a delete field to every form. This behavior differs from that expected, however (why would one want a delete option on an "add" form?), as well as that of the builtin admin. I've included a patch on , but haven't bothered with patching tests yet.

widgets=None, validate_max=False, localized_fields=None,                           labels=None, help_texts=None, error_messages=None,                           min_num=None, validate_min=False, field_classes=None,                          absolute_max=None):                          absolute_max=None, can_delete_extra=True):     """     Return an ``InlineFormSet`` for the given kwargs.


Question: formsets with candelete=True shouldn't add delete field to extra forms Description Current behavior of formsets with candelete=True is to add a delete field to every form. This behavior differs from that expected, however (why would one want a delete option on an "add" form?), as well as that of the builtin admin. I've included a patch on , but haven't bothered with patching tests yet.

'error_messages': error_messages,         'field_classes': field_classes,         'absolute_max': absolute_max,<ins>        'can_delete_extra': can_delete_extra,</ins>     }     FormSet = modelformset_factory(model, **kwargs)     FormSet.fk = fk


Question: ForeignKey.validate() should validate using the base manager. Description ForeignKey.validate() should validate using the base manager instead of the default manager. Consider the models: class ArticleManager(models.Manage): def getqueryset(self): qs = super().getqueryset() return (archived=False) class Article(models.Model): title = models.CharField(maxlength=100) archived = models.BooleanField(default=False) # Don't include archived articles by default. objects = ArticleManager() class FavoriteAricles(models.Model): article = models.ForeignKey(Article, ondelete=models.CASCADE) In the example, now consider a form that allows users to pick a favorite article including archived articles. class FavoriteAriclesForm(forms.ModelForm): class Meta: model = FavoriteArticle fields = 'all' def init(self, args, kwargs): super().init(args, kwargs) # Use the base manager instead of the default manager to allow archived articles. ['article'].queryset = () The above form will never validate as True when a user selects an archived article. This is because the ForeignKey validation always uses defaultmanager instead of basemanager. The user facing error message is "article instance with id 123 does not exist." (quite confusing to typical users). The code for this validation is here:  The FavoriteAriclesForm is specifically designed to use a different manager, but the ForeignKey validation makes this difficult. In this example scenario, it is not acceptable to change the model's default manager as the default should avoid archived articles in other typical scenarios. Suggested solution: the ForeignKey validation should use the basemanager instead which does not include the default filters.

return          using = router.db_for_read(self.remote_field.model, instance=model_instance)        qs = self.remote_field.model._default_manager.using(using).filter(        qs = self.remote_field.model._base_manager.using(using).filter(             **{self.remote_field.field_name: value}         )         qs = qs.complex_filter(self.get_limit_choices_to())


Question: Support "%V" format in WeekArchiveView. Description (Docs for WeekArchiveView are misleading about %W) - closed 4 years ago mentioned support for %V week format. Since python 3.6, %G, %u and %V ISO 8601 formatters were to strptime. WeekArchiveView should add %V to the list of accepted week formatters. This would require as well the special case to change the year format to %G, or simply ValueError in datefrom_string should mention the message passed from datetime.datetime.strptime: ISO week directive '%V' is incompatible with the year directive '%Y'. Use the ISO year '%G'.

The first day according to the week format is 0 and the last day is 6.         """         week_format = self.get_week_format()        if week_format == '%W':                 # week starts on Monday        if week_format in {'%W', '%V'}:         # week starts on Monday             return date.weekday()         elif week_format == '%U':               # week starts on Sunday             return (date.weekday() + 1) % 7


Question: Support "%V" format in WeekArchiveView. Description (Docs for WeekArchiveView are misleading about %W) - closed 4 years ago mentioned support for %V week format. Since python 3.6, %G, %u and %V ISO 8601 formatters were to strptime. WeekArchiveView should add %V to the list of accepted week formatters. This would require as well the special case to change the year format to %G, or simply ValueError in datefrom_string should mention the message passed from datetime.datetime.strptime: ISO week directive '%V' is incompatible with the year directive '%Y'. Use the ISO year '%G'.

date_field = self.get_date_field()         week_format = self.get_week_format()        week_choices = {'%W': '1', '%U': '0'}        week_choices = {'%W': '1', '%U': '0', '%V': '1'}         try:             week_start = week_choices[week_format]         except KeyError:


Question: Support "%V" format in WeekArchiveView. Description (Docs for WeekArchiveView are misleading about %W) - closed 4 years ago mentioned support for %V week format. Since python 3.6, %G, %u and %V ISO 8601 formatters were to strptime. WeekArchiveView should add %V to the list of accepted week formatters. This would require as well the special case to change the year format to %G, or simply ValueError in datefrom_string should mention the message passed from datetime.datetime.strptime: ISO week directive '%V' is incompatible with the year directive '%Y'. Use the ISO year '%G'.

week_format,                 ', '.join(sorted(week_choices)),             ))        date = _date_from_string(year, self.get_year_format(),                                 week_start, '%w',                                 week, week_format)        year_format = self.get_year_format()        if week_format == '%V' and year_format != '%G':            raise ValueError(                "ISO week directive '%s' is incompatible with the year "                "directive '%s'. Use the ISO year '%%G' instead." % (                    week_format, year_format,                )            )        date = _date_from_string(year, year_format, week_start, '%w', week, week_format)         since = self._make_date_lookup_arg(date)         until = self._make_date_lookup_arg(self._get_next_week(date))         lookup_kwargs = {


Question: makemigrations crashes for ForeignKey with mixed-case app name. Description When i run "python3 migrate" on Django 3.1b1 shows me that error (Please, note that the code works well in 3.0) ValueError: The field DJRegLogin.Content.category was declared with a lazy reference to 'djreglogin.category', but app 'djreglogin' isn't installed. (Conflict Part) class Category(models.Model): title = models.CharField(maxlength=100, dbindex=True) slug = models.SlugField(maxlength=100, dbindex=True) class Meta: verbosename = 'Category' verbosenameplural = 'Categories' def str(self): return def getabsoluteurl(self): return reverse('viewblogcategory', None, kwargs={'slug': }) class Content(models.Model): title = models.CharField(maxlength=100, unique=True) slug = models.SlugField(maxlength=100, unique=True) body = RichTextField(configname='default') posted = models.DateTimeField(dbindex=True, autonowadd=True) sites = models.ManyToManyField(Site) ip = models.GenericIPAddressField(editable=False) category = models.ForeignKey(Category, ondelete=models.CASCADE) user = models.ForeignKey(User, ondelete=models.CASCADE, null=False, blank=False, editable=False) status = models.CharField(maxlength=10, choices=STATUSCHOICES, default='draft') def str(self): return def getabsoluteurl(self): return reverse('viewblogpost', None, kwargs={'slug': }) (Related to issue part) INSTALLEDAPPS = [ 'DJRegLogin', '', '', 'django.contrib.contenttypes', 'django.contrib.sessions', 'django.contrib.messages', 'django.contrib.staticfiles', 'socialdjango', 'ckeditor', '', 'django.contrib.flatpages', 'django.contrib.sitemaps', ] from import AppConfig class DJRegLoginConfig(AppConfig): name = 'DJRegLogin' verbosename = "Contents"

if self.remote_field.parent_link:             kwargs['parent_link'] = self.remote_field.parent_link         if isinstance(self.remote_field.model, str):            kwargs['to'] = self.remote_field.model.lower()            if '.' in self.remote_field.model:                app_label, model_name = self.remote_field.model.split('.')                kwargs['to'] = '%s.%s' % (app_label, model_name.lower())            else:                kwargs['to'] = self.remote_field.model.lower()         else:             kwargs['to'] = self.remote_field.model._meta.label_lower         # If swappable is True, then see if we're actually pointing to the target


Question: Add errors when UniqueConstraint.deferrable is combined with index specific parameters Description UniqueConstraint should throw an error when deferrable is combined with parameters which require the usage of an explicit unique index through CREATE UNIQUE INDEX. This was missed when UniqueConstraint.include and UniqueConstraint.opclasses was . We should also add documentation about this incompatibility.

raise ValueError(                 'UniqueConstraint with conditions cannot be deferred.'             )<ins>        if include and deferrable:            raise ValueError(                'UniqueConstraint with include fields cannot be deferred.'            )        if opclasses and deferrable:            raise ValueError(                'UniqueConstraint with opclasses cannot be deferred.'            )</ins>         if not isinstance(deferrable, (type(None), Deferrable)):             raise ValueError(                 'UniqueConstraint.deferrable must be a Deferrable instance.'


Question: SQL generated for negated F() expressions is incorrect Description Consider the following model definition. from import models class Rectangle(models.Model): length = models.IntegerField(null=True) width = models.IntegerField(null=True) We make the following queries: Q1) Rectangles that are squares. Q2a) Rectangles that are not squares (length != width). Q2b) Rectangles that are not squares (width != length). Queries Q2a and Q2b semantically mean the same. However, the ORM generates different SQL for these two queries. import django () from import F from import Rectangle print '--- Q1: Get the rectangles that are squares' print (length=F('width')).values('pk').query print '--- Q2a: Get the rectangles that are not squares' print Rectangle.objects.exclude(length=F('width')).values('pk').query print '--- Q2b: Get the rectangles that are not squares' print Rectangle.objects.exclude(width=F('length')).values('pk').query The generated SQL is --- Q1: Get the rectangles that are squares SELECT "testapprectangle"."id" FROM "testapprectangle" WHERE "testapprectangle"."length" = ("testapprectangle"."width") --- Q2a: Get the rectangles that are not squares SELECT "testapprectangle"."id" FROM "testapprectangle" WHERE NOT ("testapprectangle"."length" = ("testapprectangle"."width") AND "testapprectangle"."length" IS NOT NULL) --- Q2b: Get the rectangles that are not squares SELECT "testapprectangle"."id" FROM "testapprectangle" WHERE NOT ("testapprectangle"."width" = ("testapprectangle"."length") AND "testapprectangle"."width" IS NOT NULL) Note the asymmetry between Q2a and Q2b. These queries will return different results. Reddit user /u/charettes set up this useful SQL fiddle with the above mentioned schema and test values: !12//4 Here's my reddit post on this issue:

require_outer = lookup_type == 'isnull' and condition.rhs is True and not current_negated         if current_negated and (lookup_type != 'isnull' or condition.rhs is False) and condition.rhs is not None:             require_outer = True            if (lookup_type != 'isnull' and (                    self.is_nullable(targets[0]) or                    self.alias_map[join_list[-1]].join_type == LOUTER)):            if lookup_type != 'isnull':                 # The condition added here will be SQL like this:                 # NOT (col IS NOT NULL), where the first NOT is added in                 # upper layers of code. The reason for addition is that if col