Question: DatabaseCache backend doesn't quote all fields in queries Description Snowflake requires all fields to be quoted, otherwise they're treated as uppercase. The attached patch works with stable/3.2.x, but I'll have to review it once django-snowflake development is caught up to Django's main branch.

else:             connection = connections[db]             table = connection.ops.quote_name(self._table)            cursor.execute("DELETE FROM %s WHERE expires < %%s" % table,                           [connection.ops.adapt_datetimefield_value(now)])            cursor.execute('DELETE FROM %s WHERE %s < %%s' % (                table,                connection.ops.quote_name('expires'),            ), [connection.ops.adapt_datetimefield_value(now)])             deleted_count = cursor.rowcount             remaining_num = num - deleted_count             if remaining_num > self._max_entries:


Question: DatabaseCache backend doesn't quote all fields in queries Description Snowflake requires all fields to be quoted, otherwise they're treated as uppercase. The attached patch works with stable/3.2.x, but I'll have to review it once django-snowflake development is caught up to Django's main branch.

last_cache_key = cursor.fetchone()                 if last_cache_key:                     cursor.execute(                        'DELETE FROM %s WHERE cache_key < %%s' % table,                        'DELETE FROM %s WHERE %s < %%s' % (                            table,                            connection.ops.quote_name('cache_key'),                        ),                         [last_cache_key[0]],                     ) diff --git a/django/db/backends/base/operations.py b/django/db/backends/base/operations.py-- a/django/db/backends/base/operations.py++ b/django/db/backends/base/operations.py


Question: DatabaseCache backend doesn't quote all fields in queries Description Snowflake requires all fields to be quoted, otherwise they're treated as uppercase. The attached patch works with stable/3.2.x, but I'll have to review it once django-snowflake development is caught up to Django's main branch.

This is used by the 'db' cache backend to determine where to start         culling.         """        return "SELECT cache_key FROM %s ORDER BY cache_key LIMIT 1 OFFSET %%s"        cache_key = self.quote_name('cache_key')        return f'SELECT {cache_key} FROM %s ORDER BY {cache_key} LIMIT 1 OFFSET %%s'      def unification_cast_sql(self, output_field):         """diff --git a/django/db/backends/oracle/operations.py b/django/db/backends/oracle/operations.py-- a/django/db/backends/oracle/operations.py++ b/django/db/backends/oracle/operations.py


Question: DatabaseCache backend doesn't quote all fields in queries Description Snowflake requires all fields to be quoted, otherwise they're treated as uppercase. The attached patch works with stable/3.2.x, but I'll have to review it once django-snowflake development is caught up to Django's main branch.

}      def cache_key_culling_sql(self):        return 'SELECT cache_key FROM %s ORDER BY cache_key OFFSET %%s ROWS FETCH FIRST 1 ROWS ONLY'        cache_key = self.quote_name('cache_key')        return (            f'SELECT {cache_key} '            f'FROM %s '            f'ORDER BY {cache_key} OFFSET %%s ROWS FETCH FIRST 1 ROWS ONLY'        )      def date_extract_sql(self, lookup_type, field_name):         if lookup_type == 'week_day':


Question: path()/repath() should raise a TypeError when kwargs is not a dict. Description Apparently, however many years into using Django, I'm still capable of making a "newbie" mistake and getting confused. So perhaps other actual new users encounter similar, especially given the lack of typing specifiers. I defined a URL like so: urlpatterns = [ path("path/to/thing", (), "myview"), ] which ... well, you either spot the issue immediately or you don't, and end up with the following. If you try and resolve() the path (eg: by making a request in your browser), you'll get something like: In [3]: resolve("/path/to/thing") in resolve(path, urlconf) 22 if urlconf is None: 23 urlconf = geturlconf() ---24 return getresolver(urlconf).resolve(path) 25 26 in resolve(self, path) 586 for pattern in self.urlpatterns: 587 try: --588 submatch = pattern.resolve(newpath) 589 except Resolver404 as e: 590 self.extendtried(tried, pattern, e.args[0].get('tried')) in resolve(self, path) 388 newpath, args, kwargs = match 389 # Pass any extrakwargs as kwargs. --390 (self.defaultargs) 391 return ResolverMatch(self.callback, args, kwargs, , route=str(self.pattern)) 392 ValueError: dictionary update sequence element has length 1; 2 is required The crux of the issue being that I meant to give the URL a name, and it's a super unfortunate history that kwargs comes before the name argument (because nearly everyone gives a URL a name, but passing static kwargs is comparatively infrequent). So what's actually happened is that kwargs = "myview" and eventually self.defaultargs = "myview". If I update to path("path/to/thing", (), "myview", name="myview"), leaving the type incorrect, I can get the following error via reverse, too: In [4]: reverse("myview") in reverse(viewname, urlconf, args, kwargs, currentapp) 84 resolver = getnsresolver(nspattern, resolver, tuple(())) 85 ---86 return resolver.reversewithprefix(view, prefix, args, kwargs) 87 88 in reversewithprefix(self, lookupview, prefix, args, **kwargs) 669 if set(kwargs).symmetricdifference(params).difference(defaults): 670 continue --671 if any((k, v) != v for k, v in ()): 672 continue 673 candidatesubs = kwargs AttributeError: 'str' object has no attribute 'items' Both of these suggest that either there should be a type-guard in path to assert it's dict-ish (if not None), or a system check on URLPattern to raise a friendly message. Well, they actually continue to suggest to me that everything after the view argument should be keyword-only, or that kwargs should come later, but I suspect those to be a harder sell ;) This is specifically around the kwargs, but it doesn't look like there's any guarding on the name either, and I feel like a name of {'test': 'test'} (i.e. accidentally swapped both positionals) is likely to bite & cause an issue somewhere.

def _path(route, view, kwargs=None, name=None, Pattern=None):     from django.views import View <ins>    if kwargs is not None and not isinstance(kwargs, dict):        raise TypeError(            f'kwargs argument must be a dict, but got {kwargs.__class__.__name__}.'        )</ins>     if isinstance(view, (list, tuple)):         # For include(...) processing.         pattern = Pattern(route, is_endpoint=False)


Question: Deprecate cachedproperty's name argument Description Django 2.2 is the last version to support Python 3.5 where cachedproperty's name argument is required. Following the release of Django 4.0, most apps will drop support for Django 2.2 (and hence Python 3.5), so Django 4.0 can deprecate the name argument without giving warnings that aren't actionable as long as apps want to keep support for Python 3.5.

import copy import itertools import operator<ins>import warnings</ins> from functools import total_ordering, wraps


Question: Deprecate cachedproperty's name argument Description Django 2.2 is the last version to support Python 3.5 where cachedproperty's name argument is required. Following the release of Django 4.0, most apps will drop support for Django 2.2 (and hence Python 3.5), so Django 4.0 can deprecate the name argument without giving warnings that aren't actionable as long as apps want to keep support for Python 3.5.

A cached property can be made out of an existing method:     (e.g. ``url = cached_property(get_absolute_url)``).<del>    The optional ``name`` argument is obsolete as of Python 3.6 and will be    deprecated in Django 4.0 (#30127).</del>     """     name = None


Question: Deprecate cachedproperty's name argument Description Django 2.2 is the last version to support Python 3.5 where cachedproperty's name argument is required. Following the release of Django 4.0, most apps will drop support for Django 2.2 (and hence Python 3.5), so Django 4.0 can deprecate the name argument without giving warnings that aren't actionable as long as apps want to keep support for Python 3.5.

)      def __init__(self, func, name=None):<ins>        from django.utils.deprecation import RemovedInDjango50Warning        if name is not None:            warnings.warn(                "The name argument is deprecated as it's unnecessary as of "                "Python 3.6.",                RemovedInDjango50Warning,                stacklevel=2,            )</ins>         self.real_func = func         self.__doc__ = getattr(func, '__doc__')


Question: URLField throws ValueError instead of ValidationError on clean Description forms.URLField( ).clean('////] results in: ValueError: Invalid IPv6 URL Traceback (most recent call last): File "", line 22, in TestOneInput File "", line 350, in testformsURLField File "", line 151, in clean File "", line 136, in runvalidators File "", line 130, in call File "", line 440, in urlsplit

raise ValidationError(self.message, code=self.code, params={'value': value})          # Then check full URL        try:            splitted_url = urlsplit(value)        except ValueError:            raise ValidationError(self.message, code=self.code, params={'value': value})         try:             super().__call__(value)         except ValidationError as e:             # Trivial case failed. Try for possible IDN domain             if value:                try:                    scheme, netloc, path, query, fragment = urlsplit(value)                except ValueError:  # for example, "Invalid IPv6 URL"                    raise ValidationError(self.message, code=self.code, params={'value': value})                scheme, netloc, path, query, fragment = splitted_url                 try:                     netloc = punycode(netloc)  # IDN -> ACE                 except UnicodeError:  # invalid domain part


Question: URLField throws ValueError instead of ValidationError on clean Description forms.URLField( ).clean('////] results in: ValueError: Invalid IPv6 URL Traceback (most recent call last): File "", line 22, in TestOneInput File "", line 350, in testformsURLField File "", line 151, in clean File "", line 136, in runvalidators File "", line 130, in call File "", line 440, in urlsplit

raise         else:             # Now verify IPv6 in the netloc part            host_match = re.search(r'^\[(.+)\](?::\d{1,5})?$', urlsplit(value).netloc)            host_match = re.search(r'^\[(.+)\](?::\d{1,5})?$', splitted_url.netloc)             if host_match:                 potential_ip = host_match[1]                 try:


Question: URLField throws ValueError instead of ValidationError on clean Description forms.URLField( ).clean('////] results in: ValueError: Invalid IPv6 URL Traceback (most recent call last): File "", line 22, in TestOneInput File "", line 350, in testformsURLField File "", line 151, in clean File "", line 136, in runvalidators File "", line 130, in call File "", line 440, in urlsplit

# section 3.1. It's defined to be 255 bytes or less, but this includes         # one byte for the length of the name and one byte for the trailing dot         # that's used to indicate absolute names in DNS.        if len(urlsplit(value).hostname) > 253:        if splitted_url.hostname is None or len(splitted_url.hostname) > 253:             raise ValidationError(self.message, code=self.code, params={'value': value})


Question: fails to handle broken data Description (last modified by Florian Apolloner) The actual input string was 'P3(3D' === Uncaught Python exception: === ValueError: could not convert string to float: '3(3' Traceback (most recent call last): File "", line 22, in TestOneInput File "", line 294, in testformsDurationField File "", line 149, in clean File "", line 502, in topython File "", line 154, in parse_duration File "", line 154, in <dictcomp

iso8601_duration_re = _lazy_re_compile(     r'^(?P<sign>[-+]?)'     r'P'    r'(?:(?P<days>\d+(.\d+)?)D)?'    r'(?:(?P<days>\d+([\.,]\d+)?)D)?'     r'(?:T'    r'(?:(?P<hours>\d+(.\d+)?)H)?'    r'(?:(?P<minutes>\d+(.\d+)?)M)?'    r'(?:(?P<seconds>\d+(.\d+)?)S)?'    r'(?:(?P<hours>\d+([\.,]\d+)?)H)?'    r'(?:(?P<minutes>\d+([\.,]\d+)?)M)?'    r'(?:(?P<seconds>\d+([\.,]\d+)?)S)?'     r')?'     r'$' )


Question: nevercache()/cachecontrol() decorators raise error on duck-typed requests. Description The cache decorators cachecontrol, nevercache and sensitivepostparameters no longer work with Django REST framework because they strictly check for an HttpRequest instance.

from functools import wraps <del>from django.http import HttpRequest</del> from django.middleware.cache import CacheMiddleware from django.utils.cache import add_never_cache_headers, patch_cache_control from django.utils.decorators import decorator_from_middleware_with_args


Question: nevercache()/cachecontrol() decorators raise error on duck-typed requests. Description The cache decorators cachecontrol, nevercache and sensitivepostparameters no longer work with Django REST framework because they strictly check for an HttpRequest instance.

def _cache_controller(viewfunc):         @wraps(viewfunc)         def _cache_controlled(request, *args, **kw):            if not isinstance(request, HttpRequest):            # Ensure argument looks like a request.            if not hasattr(request, 'META'):                 raise TypeError(                     "cache_control didn't receive an HttpRequest. If you are "                     "decorating a classmethod, be sure to use "


Question: nevercache()/cachecontrol() decorators raise error on duck-typed requests. Description The cache decorators cachecontrol, nevercache and sensitivepostparameters no longer work with Django REST framework because they strictly check for an HttpRequest instance.

"""     @wraps(view_func)     def _wrapped_view_func(request, *args, **kwargs):        if not isinstance(request, HttpRequest):        # Ensure argument looks like a request.        if not hasattr(request, 'META'):             raise TypeError(                 "never_cache didn't receive an HttpRequest. If you are "                 "decorating a classmethod, be sure to use @method_decorator."


Question: ExpressionWrapper for ~Q(pkin=[]) crashes. Description (last modified by Stefan Brand) Problem Description I'm reducing some Q objects (similar to what is described in ticket:. Everything is fine for the case where the result is ExpressionWrapper(Q(pkin=[])). However, when I reduce to ExpressionWrapper(~Q(pkin=[])) the query breaks. Symptoms Working for ExpressionWrapper(Q(pkin=[])) print(queryset.annotate(foo=ExpressionWrapper(Q(pkin=[]), outputfield=BooleanField())).values("foo").query) SELECT 0 AS "foo" FROM "table" Not working for ExpressionWrapper(~Q(pkin=[])) print(queryset.annotate(foo=ExpressionWrapper(~Q(pkin=[]), outputfield=BooleanField())).values("foo").query) SELECT AS "foo" FROM "table"

defaults = {'form_class': form_class, 'required': False}         return super().formfield(**{**defaults, **kwargs}) <ins>    def select_format(self, compiler, sql, params):        sql, params = super().select_format(compiler, sql, params)        # Filters that match everything are handled as empty strings in the        # WHERE clause, but in SELECT or GROUP BY list they must use a        # predicate that's always True.        if sql == '':            sql = '1'        return sql, params</ins>  class CharField(Field):     description = _("String (up to %(max_length)s)")


Question: Delete nonexistent migrations from djangomigrations table Description Django adds a djangomigrations table to the database which list all of the migrations that have been applied (and when). With the introduction of squashmigrations, it is possible for this table to contain a lot of old migrations that no longer exist. This can be problematic if naming duplication occurs: Example: I have an app with: myapp/migrations/ I squash and delete replaced migrations: myapp/migrations/ I create a new migration and use poor naming: myapp/migrations/ My new migration never runs because the djangomigrations table thinks it has already been applied. I propose truncation of the djangomigrations table so that it includes only migrations that actually exist in the django project. This could be done automatically (when executor runs, or inside the migrate mcommand). Or have its own mcommand that requires it be run manually. I prefer the automatic approach though. Pros: Cleans up old data that just bloats the database. Protects users from the trap mentioned above where a new migration is created with the same name as one that was applied in the past. Cons: A loss of historical information. Note: Need to be careful with implementation to avoid a possible new trap if a user squashes migrations and then proceeds to delete the replaced migrations before running the squashed migrations on their database -django will think squashed migrations haven't been applied and will attempt to reapply them. This can be remedied simply by not removing migrations mentioned in replaces lists of other migrations from djangomigrations (ie. we'd consider replaced migrations as still existing, even if their actual files have already been removed).

'--check', action='store_true', dest='check_unapplied',             help='Exits with a non-zero status if unapplied migrations exist.',         )<ins>        parser.add_argument(            '--prune', action='store_true', dest='prune',            help='Delete nonexistent migrations from the django_migrations table.',        )</ins>      @no_translations     def handle(self, *args, **options):


Question: Delete nonexistent migrations from djangomigrations table Description Django adds a djangomigrations table to the database which list all of the migrations that have been applied (and when). With the introduction of squashmigrations, it is possible for this table to contain a lot of old migrations that no longer exist. This can be problematic if naming duplication occurs: Example: I have an app with: myapp/migrations/ I squash and delete replaced migrations: myapp/migrations/ I create a new migration and use poor naming: myapp/migrations/ My new migration never runs because the djangomigrations table thinks it has already been applied. I propose truncation of the djangomigrations table so that it includes only migrations that actually exist in the django project. This could be done automatically (when executor runs, or inside the migrate mcommand). Or have its own mcommand that requires it be run manually. I prefer the automatic approach though. Pros: Cleans up old data that just bloats the database. Protects users from the trap mentioned above where a new migration is created with the same name as one that was applied in the past. Cons: A loss of historical information. Note: Need to be careful with implementation to avoid a possible new trap if a user squashes migrations and then proceeds to delete the replaced migrations before running the squashed migrations on their database -django will think squashed migrations haven't been applied and will attempt to reapply them. This can be remedied simply by not removing migrations mentioned in replaces lists of other migrations from djangomigrations (ie. we'd consider replaced migrations as still existing, even if their actual files have already been removed).

else:             targets = executor.loader.graph.leaf_nodes() <ins>        if options['prune']:            if not options['app_label']:                raise CommandError(                    'Migrations can be pruned only when an app is specified.'                )            if self.verbosity > 0:                self.stdout.write('Pruning migrations:', self.style.MIGRATE_HEADING)            to_prune = set(executor.loader.applied_migrations) - set(executor.loader.disk_migrations)            squashed_migrations_with_deleted_replaced_migrations = [                migration_key                for migration_key, migration_obj in executor.loader.replacements.items()                if any(replaced in to_prune for replaced in migration_obj.replaces)            ]            if squashed_migrations_with_deleted_replaced_migrations:                self.stdout.write(self.style.NOTICE(                    "  Cannot use --prune because the following squashed "                    "migrations have their 'replaces' attributes and may not "                    "be recorded as applied:"                ))                for migration in squashed_migrations_with_deleted_replaced_migrations:                    app, name = migration                    self.stdout.write(f'    {app}.{name}')                self.stdout.write(self.style.NOTICE(                    "  Re-run 'manage.py migrate' if they are not marked as "                    "applied, and remove 'replaces' attributes in their "                    "Migration classes."                ))            else:                to_prune = sorted(                    migration                    for migration in to_prune                    if migration[0] == app_label                )                if to_prune:                    for migration in to_prune:                        app, name = migration                        if self.verbosity > 0:                            self.stdout.write(self.style.MIGRATE_LABEL(                                f'  Pruning {app}.{name}'                            ), ending='')                        executor.recorder.record_unapplied(app, name)                        if self.verbosity > 0:                            self.stdout.write(self.style.SUCCESS(' OK'))                elif self.verbosity > 0:                    self.stdout.write('  No migrations to prune.')</ins>         plan = executor.migration_plan(targets)         exit_dry = plan and options['check_unapplied']


Question: Delete nonexistent migrations from djangomigrations table Description Django adds a djangomigrations table to the database which list all of the migrations that have been applied (and when). With the introduction of squashmigrations, it is possible for this table to contain a lot of old migrations that no longer exist. This can be problematic if naming duplication occurs: Example: I have an app with: myapp/migrations/ I squash and delete replaced migrations: myapp/migrations/ I create a new migration and use poor naming: myapp/migrations/ My new migration never runs because the djangomigrations table thinks it has already been applied. I propose truncation of the djangomigrations table so that it includes only migrations that actually exist in the django project. This could be done automatically (when executor runs, or inside the migrate mcommand). Or have its own mcommand that requires it be run manually. I prefer the automatic approach though. Pros: Cleans up old data that just bloats the database. Protects users from the trap mentioned above where a new migration is created with the same name as one that was applied in the past. Cons: A loss of historical information. Note: Need to be careful with implementation to avoid a possible new trap if a user squashes migrations and then proceeds to delete the replaced migrations before running the squashed migrations on their database -django will think squashed migrations haven't been applied and will attempt to reapply them. This can be remedied simply by not removing migrations mentioned in replaces lists of other migrations from djangomigrations (ie. we'd consider replaced migrations as still existing, even if their actual files have already been removed).

return         if exit_dry:             sys.exit(1)<ins>        if options['prune']:            return</ins>          # At this point, ignore run_syncdb if there aren't any apps to sync.         run_syncdb = options['run_syncdb'] and executor.loader.unmigrated_apps


Question: Add ability to know where an object was deleted from to pre/post delete signals Description Since deleting objects deletes related objects, it would be good to know the origin of the delete. It lets the signal listeners know why an object is being delete. If it's deleted from a (), you know which instance initiated the delete and if it's from a () you know which queryset initiated the delete, with the models in that queryset. Using this you know if the instance is being deleted directly or because it's related to another instance.

return [], {}, set(), []     else:         using = router.db_for_write(obj._meta.model)    collector = NestedObjects(using=using)    collector = NestedObjects(using=using, origin=objs)     collector.collect(objs)     perms_needed = set() diff --git a/django/contrib/contenttypes/management/commands/remove_stale_contenttypes.py b/django/contrib/contenttypes/management/commands/remove_stale_contenttypes.py-- a/django/contrib/contenttypes/management/commands/remove_stale_contenttypes.py++ b/django/contrib/contenttypes/management/commands/remove_stale_contenttypes.py


Question: Add ability to know where an object was deleted from to pre/post delete signals Description Since deleting objects deletes related objects, it would be good to know the origin of the delete. It lets the signal listeners know why an object is being delete. If it's deleted from a (), you know which instance initiated the delete and if it's from a () you know which queryset initiated the delete, with the models in that queryset. Using this you know if the instance is being deleted directly or because it's related to another instance.

ct_info = []                     for ct in to_remove:                         ct_info.append('    - Content type for %s.%s' % (ct.app_label, ct.model))                        collector = NoFastDeleteCollector(using=using)                        collector = NoFastDeleteCollector(using=using, origin=ct)                         collector.collect([ct])                          for obj_type, objs in collector.data.items():diff --git a/django/db/models/base.py b/django/db/models/base.py-- a/django/db/models/base.py++ b/django/db/models/base.py


Question: Add ability to know where an object was deleted from to pre/post delete signals Description Since deleting objects deletes related objects, it would be good to know the origin of the delete. It lets the signal listeners know why an object is being delete. If it's deleted from a (), you know which instance initiated the delete and if it's from a () you know which queryset initiated the delete, with the models in that queryset. Using this you know if the instance is being deleted directly or because it's related to another instance.

"to None." % (self._meta.object_name, self._meta.pk.attname)             )         using = using or router.db_for_write(self.__class__, instance=self)        collector = Collector(using=using)        collector = Collector(using=using, origin=self)         collector.collect([self], keep_parents=keep_parents)         return collector.delete() diff --git a/django/db/models/deletion.py b/django/db/models/deletion.py-- a/django/db/models/deletion.py++ b/django/db/models/deletion.py


Question: Add ability to know where an object was deleted from to pre/post delete signals Description Since deleting objects deletes related objects, it would be good to know the origin of the delete. It lets the signal listeners know why an object is being delete. If it's deleted from a (), you know which instance initiated the delete and if it's from a () you know which queryset initiated the delete, with the models in that queryset. Using this you know if the instance is being deleted directly or because it's related to another instance.

class Collector:    def __init__(self, using):    def __init__(self, using, origin=None):         self.using = using        # A Model or QuerySet object.        self.origin = origin         # Initially, {model: {instances}}, later values become lists.         self.data = defaultdict(set)         # {model: {(field, value): {instances}}}


Question: Add ability to know where an object was deleted from to pre/post delete signals Description Since deleting objects deletes related objects, it would be good to know the origin of the delete. It lets the signal listeners know why an object is being delete. If it's deleted from a (), you know which instance initiated the delete and if it's from a () you know which queryset initiated the delete, with the models in that queryset. Using this you know if the instance is being deleted directly or because it's related to another instance.

for model, obj in self.instances_with_model():                 if not model._meta.auto_created:                     signals.pre_delete.send(                        sender=model, instance=obj, using=self.using                        sender=model, instance=obj, using=self.using,                        origin=self.origin,                     )              # fast deletes


Question: Add ability to know where an object was deleted from to pre/post delete signals Description Since deleting objects deletes related objects, it would be good to know the origin of the delete. It lets the signal listeners know why an object is being delete. If it's deleted from a (), you know which instance initiated the delete and if it's from a () you know which queryset initiated the delete, with the models in that queryset. Using this you know if the instance is being deleted directly or because it's related to another instance.

if not model._meta.auto_created:                     for obj in instances:                         signals.post_delete.send(                            sender=model, instance=obj, using=self.using                            sender=model, instance=obj, using=self.using,                            origin=self.origin,                         )          # update collected instancesdiff --git a/django/db/models/query.py b/django/db/models/query.py-- a/django/db/models/query.py++ b/django/db/models/query.py


Question: Add ability to know where an object was deleted from to pre/post delete signals Description Since deleting objects deletes related objects, it would be good to know the origin of the delete. It lets the signal listeners know why an object is being delete. If it's deleted from a (), you know which instance initiated the delete and if it's from a () you know which queryset initiated the delete, with the models in that queryset. Using this you know if the instance is being deleted directly or because it's related to another instance.

del_query.query.select_related = False         del_query.query.clear_ordering(force=True)         collector = Collector(using=del_query.db)        collector = Collector(using=del_query.db, origin=self)         collector.collect(del_query)         deleted, _rows_count = collector.delete()


Question: MigrationRecorder does not obey dbrouter allowmigrate rules Description Hi, We have a multi-db setup. We have one connection that is for the django project, and several connections that talk to other dbs for information (ie models with managed = False). Django should only create tables in the first connection, never in any of the other connections. We have a simple router that does the following: class Router(object): def allowmigrate(self, db, model): if db == 'default': return True return False Current Behaviour We run our functional tests and the migrate command is called against each connection when the test databases are created (see , setupdatabases, line 300-ish, which calls , createtestdb, line 377-ish) When this migrate runs, it tries to apply our migrations, which tries to record that a migration has been applied (see , applymigration, which has several calls to self.recorder.recordapplied). The first thing that recordapplied does is a call to self.ensureschema() (see , recordapplied, lien 66-ish). ensureschema checks to see if the Migration model is in the tables in the connection. If it does not find the table then it tries to create the table. I believe that this is incorrect behaviour when a dbrouter has been provided. If using the router above, my expectation would be that the table is not created on any connection other than the 'default' connection. Looking at the other methods on the MigrationRecorder, I would expect that there will be similar issues with appliedmigrations and record_unapplied.

(un)applied and in a second step run all the database operations.         """         # The django_migrations table must be present to record applied        # migrations.        self.recorder.ensure_schema()        # migrations, but don't create it if there are no migrations to apply.        if plan == []:            if not self.recorder.has_table():                return self._create_project_state(with_applied_migrations=False)        else:            self.recorder.ensure_schema()          if plan is None:             plan = self.migration_plan(targets)


Question: Optimize multiple AlterFooTogether operations into one Description Hi, In we split the AlterFooTogether (AlterUniqueTogether and AlterIndexTogether) operations into two types of operations. First, a migration will have operations to remove constraints, and then other operations adds the new constraints. This allows field alterations to work as expected during in between operations. In some cases, this introduced two operations that can actually easily be reduced to one. See for instance the test case:  Example: operations = [ migrations.AlterUniqueTogether( name='mymodel', uniquetogether=set(), ), migrations.AlterIndexTogether( name='mymodel', indextogether=set(), ), migrations.AlterUniqueTogether( name='mymodel', uniquetogether={("col",)}, ), migrations.AlterIndexTogether( name='mymodel', indextogether={("col",)}, ), ] should be optimized to operations = [ migrations.AlterUniqueTogether( name='mymodel', uniquetogether={("col",)}, ), migrations.AlterIndexTogether( name='mymodel', indextogether={("col",)}, ), ] So that we don't do two operations on each constraint, but only one.

def reduce(self, operation, app_label):         return (             super().reduce(operation, app_label) or            not operation.references_model(self.name, app_label)            self.can_reduce_through(operation, app_label)         )     def can_reduce_through(self, operation, app_label):        return not operation.references_model(self.name, app_label)  class CreateModel(ModelOperation):     """Create a model's table."""


Question: Optimize multiple AlterFooTogether operations into one Description Hi, In we split the AlterFooTogether (AlterUniqueTogether and AlterIndexTogether) operations into two types of operations. First, a migration will have operations to remove constraints, and then other operations adds the new constraints. This allows field alterations to work as expected during in between operations. In some cases, this introduced two operations that can actually easily be reduced to one. See for instance the test case:  Example: operations = [ migrations.AlterUniqueTogether( name='mymodel', uniquetogether=set(), ), migrations.AlterIndexTogether( name='mymodel', indextogether=set(), ), migrations.AlterUniqueTogether( name='mymodel', uniquetogether={("col",)}, ), migrations.AlterIndexTogether( name='mymodel', indextogether={("col",)}, ), ] should be optimized to operations = [ migrations.AlterUniqueTogether( name='mymodel', uniquetogether={("col",)}, ), migrations.AlterIndexTogether( name='mymodel', indextogether={("col",)}, ), ] So that we don't do two operations on each constraint, but only one.

def migration_name_fragment(self):         return 'alter_%s_%s' % (self.name_lower, self.option_name) <ins>    def can_reduce_through(self, operation, app_label):        return (            super().can_reduce_through(operation, app_label) or (                isinstance(operation, AlterTogetherOptionOperation) and                type(operation) is not type(self)            )        )</ins>  class AlterUniqueTogether(AlterTogetherOptionOperation):     """


Question: Add a management command to optimize a migration Description (last modified by Raphael Gaschignard) Because the migration optimizer still has a bit of trouble with reducing operations, you often want to edit squashed migrations and re-run it through the optimizer. The attached patch contains an implementation of a management command, optimizemigration, that will help speed up this process. optimizemigration appname migrationname reads a single migration, pass it through the optimizer, and then rewrite the result to disk. Unlike squashmigrations, this ignores things like the squashed-ness of the migration.

<ins>import sysfrom django.apps import appsfrom django.core.management.base import BaseCommand, CommandErrorfrom django.core.management.utils import run_formattersfrom django.db import migrationsfrom django.db.migrations.exceptions import AmbiguityErrorfrom django.db.migrations.loader import MigrationLoaderfrom django.db.migrations.optimizer import MigrationOptimizerfrom django.db.migrations.writer import MigrationWriterfrom django.utils.version import get_docs_versionclass Command(BaseCommand):    help = "Optimizes the operations for the named migration."    def add_arguments(self, parser):        parser.add_argument(            "app_label",            help="App label of the application to optimize the migration for.",        )        parser.add_argument(            "migration_name", help="Migration name to optimize the operations for."        )        parser.add_argument(            "--check",            action="store_true",            help="Exit with a non-zero status if the migration can be optimized.",        )    def handle(self, *args, **options):        verbosity = options["verbosity"]        app_label = options["app_label"]        migration_name = options["migration_name"]        check = options["check"]        # Validate app_label.        try:            apps.get_app_config(app_label)        except LookupError as err:            raise CommandError(str(err))        # Load the current graph state.        loader = MigrationLoader(None)        if app_label not in loader.migrated_apps:            raise CommandError(f"App '{app_label}' does not have migrations.")        # Find a migration.        try:            migration = loader.get_migration_by_prefix(app_label, migration_name)        except AmbiguityError:            raise CommandError(                f"More than one migration matches '{migration_name}' in app "                f"'{app_label}'. Please be more specific."            )        except KeyError:            raise CommandError(                f"Cannot find a migration matching '{migration_name}' from app "                f"'{app_label}'."            )        # Optimize the migration.        optimizer = MigrationOptimizer()        new_operations = optimizer.optimize(migration.operations, migration.app_label)        if len(migration.operations) == len(new_operations):            if verbosity > 0:                self.stdout.write("No optimizations possible.")            return        else:            if verbosity > 0:                self.stdout.write(                    "Optimizing from %d operations to %d operations."                    % (len(migration.operations), len(new_operations))                )            if check:                sys.exit(1)        # Set the new migration optimizations.        migration.operations = new_operations        # Write out the optimized migration file.        writer = MigrationWriter(migration)        migration_file_string = writer.as_string()        if writer.needs_manual_porting:            if migration.replaces:                raise CommandError(                    "Migration will require manual porting but is already a squashed "                    "migration.\nTransition to a normal migration first: "                    "https://docs.djangoproject.com/en/%s/topics/migrations/"                    "#squashing-migrations" % get_docs_version()                )            # Make a new migration with those operations.            subclass = type(                "Migration",                (migrations.Migration,),                {                    "dependencies": migration.dependencies,                    "operations": new_operations,                    "replaces": [(migration.app_label, migration.name)],                },            )            optimized_migration_name = "%s_optimized" % migration.name            optimized_migration = subclass(optimized_migration_name, app_label)            writer = MigrationWriter(optimized_migration)            migration_file_string = writer.as_string()            if verbosity > 0:                self.stdout.write(                    self.style.MIGRATE_HEADING("Manual porting required") + "\n"                    "  Your migrations contained functions that must be manually "                    "copied over,\n"                    "  as we could not safely copy their implementation.\n"                    "  See the comment at the top of the optimized migration for "                    "details."                )        with open(writer.path, "w", encoding="utf-8") as fh:            fh.write(migration_file_string)        run_formatters([writer.path])        if verbosity > 0:            self.stdout.write(                self.style.MIGRATE_HEADING(f"Optimized migration {writer.path}")            )</ins>


Question: Micro-optimisation for Value.resolveoutputfield (by modifying ) Description Currently, when you do something like annotate(x=Value('test')) that will eventually probably call down into Value.resolveoutputfield() and run the following code: if isinstance(, str): return fields.CharField() which is innocuous enough. However, CharField currently expects that self.maxlength is always a non null value of sensible data, and AFAIK this is caught for users at system-check time as a requirement for use. So what currently happens is that the CharField internally gets granted a MaxLengthValidator which cannot work and must be demonstrably extraneous (i.e. validators aren't used the outputfield, at least for Value) x = Value('test') y = x.resolveoutputfield() y.validators [ at 0x105e3d940] y.clean('1', modelinstance=None) in compare(self, a, b): TypeError: '' not supported between instances of 'int' and 'NoneType' Further compounding this is that MaxLengthValidator is decorated by (both directly and indirectly via BaseValidator ...?). So, baseline (as of ): In [1]: from import Value In [2]: x = Value('test') In [3]: %timeit x.resolveoutputfield() 1 µs ± 39.6 ns per loop (mean ± std. dev. of 7 runs, loops each) (Note: a previous run was faster at 7.6µs, so normal CPU workfload flux is in effect). We can see how much of the time is because of (see my comment here on a PR about deconstructible being a source to potentially optimise away) by just commenting it out from both validator classes: In [1]: from import Value In [2]: x = Value('test') In [3]: %timeit x.resolveoutputfield() 96 µs ± 130 ns per loop (mean ± std. dev. of 7 runs, loops each) But ignoring the class instantiation altogether is faster, easier and more correct at this juncture: In [1]: from import Value In [2]: x = Value('test') In [3]: %timeit x.resolveoutputfield() 86 µs ± 45.4 ns per loop (mean ± std. dev. of 7 runs, loops each) So roughly a 2µs improvement. How do we get to that? Change the to: if self.maxlength is not None: (validators.MaxLengthValidator(self.max_length)) which incidentally and happily is the same process taken by for precedent. I have a branch locally with this change, and all existing tests currently pass. I'll push it to CI once I get a ticket number out of this submission, and see if it causes any issues elsewhere, and we can decide if it can be accepted from there.

def __init__(self, *args, db_collation=None, **kwargs):         super().__init__(*args, **kwargs)         self.db_collation = db_collation        self.validators.append(validators.MaxLengthValidator(self.max_length))        if self.max_length is not None:            self.validators.append(validators.MaxLengthValidator(self.max_length))      def check(self, **kwargs):         databases = kwargs.get('databases') or []


Question: Adding nullable OneToOneField crashes on SQLite. Description This new sqlite3 error has cropped up between building django-oauth-toolkit between Django 4.0 and main branch for migrations.AddField of a OneToOneField (see ): self = object at 0x10b8038b0query = 'ALTER TABLE "oauth2provideraccesstoken" ADD COLUMN "sourcerefreshtokenid" bigint NULL UNIQUE REFERENCES "oauth2providerrefreshtoken" ("id") DEFERRABLE INITIALLY DEFERRED' params = [] def execute(self, query, params=None): if params is None: return Database.Cursor.execute(self, query) query = self.convertquery(query) E Cannot add a UNIQUE column Here's the relevant migration snippet: migrations.AddField( modelname='AccessToken', name='sourcerefreshtoken', field=models.OneToOneField(blank=True, null=True, , to=oauth2settings.REFRESHTOKENMODEL, relatedname="refreshedaccess_token"), ), I see there have been a lot of sqlite3 changes in since the 4.0 release....

def add_field(self, model, field):         """Create a field on a model."""        # Fields with default values cannot by handled by ALTER TABLE ADD        # COLUMN statement because DROP DEFAULT is not supported in        # ALTER TABLE.        if not field.null or self.effective_default(field) is not None:        if (            # Primary keys and unique fields are not supported in ALTER TABLE            # ADD COLUMN.            field.primary_key or field.unique or            # Fields with default values cannot by handled by ALTER TABLE ADD            # COLUMN statement because DROP DEFAULT is not supported in            # ALTER TABLE.            not field.null or self.effective_default(field) is not None        ):             self._remake_table(model, create_field=field)         else:             super().add_field(model, field)


Question: Deferred fields incorrect when following prefetches back to the "parent" object Description Given the following models: class User(models.Model): email = models.EmailField() kind = models.CharField( maxlength=10, choices=[("ADMIN", "Admin"), ("REGULAR", "Regular")] ) class Profile(models.Model): fullname = models.CharField(maxlength=255) user = models.OneToOneField(User, ondelete=models.CASCADE) I'd expect the following test case to pass: def testonlyrelatedqueryset(self): user = ( email="", kind="ADMIN", ) (user=user, fullname="Test Tester") queryset = ("email").prefetchrelated( Prefetch( "profile", queryset=Profile.objects.prefetchrelated( Prefetch("user", ("kind")) ), ) ) with self.assertNumQueries(3): user = () with self.assertNumQueries(0): self.assertEqual(, "ADMIN") The second assertNumQueries actually fails with: AssertionError: 1 != 0 : 1 queries executed, 0 expected Captured queries were: "testsuser"."id", "testsuser"."kind" FROM "testsuser" WHERE "testsuser"."id" = 1 This is exactly the query I'd expect to see if kind on the inner User queryset had been deferred, which it hasn't. The three queries executed when iterating the main queryset (ie when executing user = ()) look correct: "testsuser"."id", "testsuser"."email" FROM "testsuser" ORDER BY "testsuser"."id" ASC LIMIT 1 "testsprofile"."id", "testsprofile"."fullname", "testsprofile"."userid" FROM "testsprofile" WHERE "testsprofile"."userid" IN (1) "testsuser"."id", "testsuser"."kind" FROM "testsuser" WHERE "testsuser"."id" IN (1) Printing () returns {'kind'}. It looks to me like Django is correctly evaluating the set of deferred fields when executing the "inner" User queryset, but somehow the instances are inheriting the set of fields they "think" have been deferred from the outer User queryset, so when the attribute is accessed it causes a database query to be executed. It appears that this also happens if the relationship between Profile and User is a ForeignKey rather than a OneToOneField (in that case, a query is executed when accessing ()[0]). I'm happy to attempt to tackle this if someone can (a) confirm it's actually a bug and (b) point me in the right direction! Thanks :)

# Since we just bypassed this class' get_queryset(), we must manage             # the reverse relation manually.             for rel_obj in queryset:                instance = instances_dict[rel_obj_attr(rel_obj)]                setattr(rel_obj, self.field.name, instance)                if not self.field.is_cached(rel_obj):                    instance = instances_dict[rel_obj_attr(rel_obj)]                    setattr(rel_obj, self.field.name, instance)             cache_name = self.field.remote_field.get_cache_name()             return queryset, rel_obj_attr, instance_attr, False, cache_name, False


Question: Technical 404 debug page reported incorrect view name for CBVs. Description Technical 404 debug page reported incorrect view name for CBVs, e.g. Raised by: <class ''instead of Raised by: Regression in . Thanks Keryn Knight for the report.

obj = resolver_match.func          if hasattr(obj, 'view_class'):            caller = obj.view_class        elif hasattr(obj, '__name__'):            obj = obj.view_class        if hasattr(obj, '__name__'):             caller = obj.__name__         elif hasattr(obj, '__class__') and hasattr(obj.__class__, '__name__'):             caller = obj.__class__.__name__


Question: ResolverMatch repr is incorrect for Class Based Views Description The following test applies cleanly to 3.2.9, and AFAIK would apply roughly correctly all the way back to when CBVs were introduced (I can't easily test without going back to a super old Python and finding the test cases, which have moved around): """ add to class: """ def testclassbasedrepr(self): self.assertEqual( repr(resolve('/redirect/')), "ResolverMatch(, " "args=(), kwargs={}, urlname=None, appnames=[], " "namespaces=[], route=redirect/)", ) The funcpath as AFAIK always been a representation to the fully qualified dotted callable where possible, that is for a CBV it's the CBV module + the class name. As of 4.0, the funcpath has become , I believe because of removing the use of updatewrapper and intentionally not setting the name and qualname in favour using the viewclass attribute, as per the comment viewclass should be used to robustly determine the name of the view (see Pull Request) Unfortunately I think that means the detection of class based views in ResolverMatch no longer works correctly, and this can probably only be resolved by making ResolverMatch CBV aware again by embedding detection of viewclass therein. Noted it as a question in this PR for ticket , but hoisting it here properly to be considered separately. The fix appears to ostensibly the same as for (see PR) class ResolverMatch: def init(...): # ... if hasattr(func, 'viewclass'): func = func.viewclass if not hasattr(func, 'name'): # ... I have a branch which I shall push shortly to confirm it works generally.

self.namespaces = [x for x in namespaces if x] if namespaces else []         self.namespace = ':'.join(self.namespaces) <ins>        if hasattr(func, 'view_class'):            func = func.view_class</ins>         if not hasattr(func, '__name__'):             # A class-based view             self._func_path = func.__class__.__module__ + '.' + func.__class__.__name__


Question: Model () should be immutable. Description changes value when a field is assigned to a model class. This code crashes with an AssertionError: from import models f = models.CharField(max_length=200) d = {f: 1} class Book(models.Model): title = f assert f in d The bug was introduced in . It's unlikely to have been encountered because there are few use cases to put a field in a dict before it's assigned to a model class. But I found a reason to do so whilst implementing and the behaviour had me stumped for a little. IMO we can revert the hash change from . Objects with the same hash are still checked for equality, which was fixed in that ticket. But it's bad if an object's hash changes, since it breaks its use in dicts.

return NotImplemented      def __hash__(self):        return hash((            self.creation_counter,            self.model._meta.app_label if hasattr(self, 'model') else None,            self.model._meta.model_name if hasattr(self, 'model') else None,        ))        return hash(self.creation_counter)      def __deepcopy__(self, memodict):         # We don't have to deepcopy very much here, since most things are not


Question: simplifyregex() doesn't handle non-capturing groups Description (last modified by Mariusz Felisiak) While using Django REST Framework's Schema generator, I found out they're using simplifyregex(); however, current version has a few shortcomings, namely non-capturing groups are broken. simplifyregex() doesn't handle non-capturing groups Description (last modified by Mariusz Felisiak) While using Django REST Framework's Schema generator, I found out they're using simplifyregex(); however, current version has a few shortcomings, namely non-capturing groups are broken.

for name, urlbase in ROLES.items():         create_reference_role(name, urlbase) # Match the beginning of a named or unnamed group.# Match the beginning of a named, unnamed, or non-capturing groups. named_group_matcher = _lazy_re_compile(r'\(\?P(<\w+>)') unnamed_group_matcher = _lazy_re_compile(r'\(')non_capturing_group_matcher = _lazy_re_compile(r'\(\?\:')   def replace_metacharacters(pattern):


Question: simplifyregex() doesn't handle non-capturing groups Description (last modified by Mariusz Felisiak) While using Django REST Framework's Schema generator, I found out they're using simplifyregex(); however, current version has a few shortcomings, namely non-capturing groups are broken. simplifyregex() doesn't handle non-capturing groups Description (last modified by Mariusz Felisiak) While using Django REST Framework's Schema generator, I found out they're using simplifyregex(); however, current version has a few shortcomings, namely non-capturing groups are broken.

final_pattern += pattern[:start] + '<var>'         prev_end = end     return final_pattern + pattern[prev_end:]def remove_non_capturing_groups(pattern):    r"""    Find non-capturing groups in the given `pattern` and remove them, e.g.    1. (?P\w+)/b/(?:\w+)c(?:\w+) => (?P\\w+)/b/c    2. ^(?:\w+(?:\w+))a => ^a    3. ^a(?:\w+)/b(?:\w+) => ^a/b    """    group_start_end_indices = _find_groups(pattern, non_capturing_group_matcher)    final_pattern, prev_end = '', None    for start, end, _ in group_start_end_indices:        final_pattern += pattern[prev_end:start]        prev_end = end    return final_pattern + pattern[prev_end:]diff --git a/django/contrib/admindocs/views.py b/django/contrib/admindocs/views.py-- a/django/contrib/admindocs/views.py++ b/django/contrib/admindocs/views.py


Question: simplifyregex() doesn't handle non-capturing groups Description (last modified by Mariusz Felisiak) While using Django REST Framework's Schema generator, I found out they're using simplifyregex(); however, current version has a few shortcomings, namely non-capturing groups are broken. simplifyregex() doesn't handle non-capturing groups Description (last modified by Mariusz Felisiak) While using Django REST Framework's Schema generator, I found out they're using simplifyregex(); however, current version has a few shortcomings, namely non-capturing groups are broken.

from django.contrib.admin.views.decorators import staff_member_required from django.contrib.admindocs import utils from django.contrib.admindocs.utils import (    replace_metacharacters, replace_named_groups, replace_unnamed_groups,    remove_non_capturing_groups, replace_metacharacters, replace_named_groups,    replace_unnamed_groups, ) from django.core.exceptions import ImproperlyConfigured, ViewDoesNotExist from django.db import models


Question: simplifyregex() doesn't handle non-capturing groups Description (last modified by Mariusz Felisiak) While using Django REST Framework's Schema generator, I found out they're using simplifyregex(); however, current version has a few shortcomings, namely non-capturing groups are broken. simplifyregex() doesn't handle non-capturing groups Description (last modified by Mariusz Felisiak) While using Django REST Framework's Schema generator, I found out they're using simplifyregex(); however, current version has a few shortcomings, namely non-capturing groups are broken.

example, turn "^(?P<sport_slug>\w+)/athletes/(?P<athlete_slug>\w+)/$"     into "/<sport_slug>/athletes/<athlete_slug>/".     """<ins>    pattern = remove_non_capturing_groups(pattern)</ins>     pattern = replace_named_groups(pattern)     pattern = replace_unnamed_groups(pattern)     pattern = replace_metacharacters(pattern)


Question: Unify reverse foreign key and m2m unsaved model querying Description Currently when querying unsaved reverse relations the behavior differs. Using model: class Foo(models.Model): fk = models.ForeignKey('self', relatedname='fkrev') m2m = models.ManyToManyField('self') and test case: print(Foo()()) print(Foo()()) We get [] from the first filter, but an error ValueError: "<Foo: Foo object" needs to have a value for field "fromfoo" before this many-to-many relationship can be used. from the second filter. So, m2m fields can't be filtered if the object isn't saved, but reverse fk fields can be filtered. There is a (slightly stale) patch for which makes fk fields and m2m fields work consistently. The changes in behavior are: Nullable many-to-many and foreign key relations will return an empty queryset when the relation field is null. For many-to-many this was previously an error (no change for foreign keys). Trying to add objects to a foreign key relation when the relation field is null is now an error (no change for m2m). Trying to use a relation of any kind when the object isn't saved is now an error (no change for m2m). I think we can squeeze these changes in as bug-fixes. These are slight backwards compatibility changes, but to me it seems that almost always the changes will be visible only in code that isn't working as intended. If these are seen as something likely breaking working code, then I don't see any way to make the APIs consistent. The patch is available from:

self.core_filters = {self.field.name: instance} <ins>            # Even if this relation is not to pk, we require still pk value.            # The wish is that the instance has been already saved to DB,            # although having a pk value isn't a guarantee of that.            if self.instance.pk is None:                raise ValueError(                    f"{instance.__class__.__name__!r} instance needs to have a primary "                    f"key value before this relationship can be used."                )</ins>         def __call__(self, *, manager):             manager = getattr(self.model, manager)             manager_class = create_reverse_many_to_one_manager(manager.__class__, rel)


Question: Unify reverse foreign key and m2m unsaved model querying Description Currently when querying unsaved reverse relations the behavior differs. Using model: class Foo(models.Model): fk = models.ForeignKey('self', relatedname='fkrev') m2m = models.ManyToManyField('self') and test case: print(Foo()()) print(Foo()()) We get [] from the first filter, but an error ValueError: "<Foo: Foo object" needs to have a value for field "fromfoo" before this many-to-many relationship can be used. from the second filter. So, m2m fields can't be filtered if the object isn't saved, but reverse fk fields can be filtered. There is a (slightly stale) patch for which makes fk fields and m2m fields work consistently. The changes in behavior are: Nullable many-to-many and foreign key relations will return an empty queryset when the relation field is null. For many-to-many this was previously an error (no change for foreign keys). Trying to add objects to a foreign key relation when the relation field is null is now an error (no change for m2m). Trying to use a relation of any kind when the object isn't saved is now an error (no change for m2m). I think we can squeeze these changes in as bug-fixes. These are slight backwards compatibility changes, but to me it seems that almost always the changes will be visible only in code that isn't working as intended. If these are seen as something likely breaking working code, then I don't see any way to make the APIs consistent. The patch is available from:

do_not_call_in_templates = True <ins>        def _check_fk_val(self):            for field in self.field.foreign_related_fields:                if getattr(self.instance, field.attname) is None:                    raise ValueError(                        f'"{self.instance!r}" needs to have a value for field '                        f'"{field.attname}" before this relationship can be used.'                    )</ins>         def _apply_rel_filters(self, queryset):             """             Filter the queryset for the instance this manager is bound to.


Question: Unify reverse foreign key and m2m unsaved model querying Description Currently when querying unsaved reverse relations the behavior differs. Using model: class Foo(models.Model): fk = models.ForeignKey('self', relatedname='fkrev') m2m = models.ManyToManyField('self') and test case: print(Foo()()) print(Foo()()) We get [] from the first filter, but an error ValueError: "<Foo: Foo object" needs to have a value for field "fromfoo" before this many-to-many relationship can be used. from the second filter. So, m2m fields can't be filtered if the object isn't saved, but reverse fk fields can be filtered. There is a (slightly stale) patch for which makes fk fields and m2m fields work consistently. The changes in behavior are: Nullable many-to-many and foreign key relations will return an empty queryset when the relation field is null. For many-to-many this was previously an error (no change for foreign keys). Trying to add objects to a foreign key relation when the relation field is null is now an error (no change for m2m). Trying to use a relation of any kind when the object isn't saved is now an error (no change for m2m). I think we can squeeze these changes in as bug-fixes. These are slight backwards compatibility changes, but to me it seems that almost always the changes will be visible only in code that isn't working as intended. If these are seen as something likely breaking working code, then I don't see any way to make the APIs consistent. The patch is available from:

return queryset, rel_obj_attr, instance_attr, False, cache_name, False          def add(self, *objs, bulk=True):<ins>            self._check_fk_val()</ins>             self._remove_prefetched_objects()             db = router.db_for_write(self.model, instance=self.instance)


Question: Unify reverse foreign key and m2m unsaved model querying Description Currently when querying unsaved reverse relations the behavior differs. Using model: class Foo(models.Model): fk = models.ForeignKey('self', relatedname='fkrev') m2m = models.ManyToManyField('self') and test case: print(Foo()()) print(Foo()()) We get [] from the first filter, but an error ValueError: "<Foo: Foo object" needs to have a value for field "fromfoo" before this many-to-many relationship can be used. from the second filter. So, m2m fields can't be filtered if the object isn't saved, but reverse fk fields can be filtered. There is a (slightly stale) patch for which makes fk fields and m2m fields work consistently. The changes in behavior are: Nullable many-to-many and foreign key relations will return an empty queryset when the relation field is null. For many-to-many this was previously an error (no change for foreign keys). Trying to add objects to a foreign key relation when the relation field is null is now an error (no change for m2m). Trying to use a relation of any kind when the object isn't saved is now an error (no change for m2m). I think we can squeeze these changes in as bug-fixes. These are slight backwards compatibility changes, but to me it seems that almost always the changes will be visible only in code that isn't working as intended. If these are seen as something likely breaking working code, then I don't see any way to make the APIs consistent. The patch is available from:

add.alters_data = True          def create(self, **kwargs):<ins>            self._check_fk_val()</ins>             kwargs[self.field.name] = self.instance             db = router.db_for_write(self.model, instance=self.instance)             return super(RelatedManager, self.db_manager(db)).create(**kwargs)


Question: Unify reverse foreign key and m2m unsaved model querying Description Currently when querying unsaved reverse relations the behavior differs. Using model: class Foo(models.Model): fk = models.ForeignKey('self', relatedname='fkrev') m2m = models.ManyToManyField('self') and test case: print(Foo()()) print(Foo()()) We get [] from the first filter, but an error ValueError: "<Foo: Foo object" needs to have a value for field "fromfoo" before this many-to-many relationship can be used. from the second filter. So, m2m fields can't be filtered if the object isn't saved, but reverse fk fields can be filtered. There is a (slightly stale) patch for which makes fk fields and m2m fields work consistently. The changes in behavior are: Nullable many-to-many and foreign key relations will return an empty queryset when the relation field is null. For many-to-many this was previously an error (no change for foreign keys). Trying to add objects to a foreign key relation when the relation field is null is now an error (no change for m2m). Trying to use a relation of any kind when the object isn't saved is now an error (no change for m2m). I think we can squeeze these changes in as bug-fixes. These are slight backwards compatibility changes, but to me it seems that almost always the changes will be visible only in code that isn't working as intended. If these are seen as something likely breaking working code, then I don't see any way to make the APIs consistent. The patch is available from:

create.alters_data = True          def get_or_create(self, **kwargs):<ins>            self._check_fk_val()</ins>             kwargs[self.field.name] = self.instance             db = router.db_for_write(self.model, instance=self.instance)             return super(RelatedManager, self.db_manager(db)).get_or_create(**kwargs)


Question: Unify reverse foreign key and m2m unsaved model querying Description Currently when querying unsaved reverse relations the behavior differs. Using model: class Foo(models.Model): fk = models.ForeignKey('self', relatedname='fkrev') m2m = models.ManyToManyField('self') and test case: print(Foo()()) print(Foo()()) We get [] from the first filter, but an error ValueError: "<Foo: Foo object" needs to have a value for field "fromfoo" before this many-to-many relationship can be used. from the second filter. So, m2m fields can't be filtered if the object isn't saved, but reverse fk fields can be filtered. There is a (slightly stale) patch for which makes fk fields and m2m fields work consistently. The changes in behavior are: Nullable many-to-many and foreign key relations will return an empty queryset when the relation field is null. For many-to-many this was previously an error (no change for foreign keys). Trying to add objects to a foreign key relation when the relation field is null is now an error (no change for m2m). Trying to use a relation of any kind when the object isn't saved is now an error (no change for m2m). I think we can squeeze these changes in as bug-fixes. These are slight backwards compatibility changes, but to me it seems that almost always the changes will be visible only in code that isn't working as intended. If these are seen as something likely breaking working code, then I don't see any way to make the APIs consistent. The patch is available from:

get_or_create.alters_data = True          def update_or_create(self, **kwargs):<ins>            self._check_fk_val()</ins>             kwargs[self.field.name] = self.instance             db = router.db_for_write(self.model, instance=self.instance)             return super(RelatedManager, self.db_manager(db)).update_or_create(**kwargs)


Question: Unify reverse foreign key and m2m unsaved model querying Description Currently when querying unsaved reverse relations the behavior differs. Using model: class Foo(models.Model): fk = models.ForeignKey('self', relatedname='fkrev') m2m = models.ManyToManyField('self') and test case: print(Foo()()) print(Foo()()) We get [] from the first filter, but an error ValueError: "<Foo: Foo object" needs to have a value for field "fromfoo" before this many-to-many relationship can be used. from the second filter. So, m2m fields can't be filtered if the object isn't saved, but reverse fk fields can be filtered. There is a (slightly stale) patch for which makes fk fields and m2m fields work consistently. The changes in behavior are: Nullable many-to-many and foreign key relations will return an empty queryset when the relation field is null. For many-to-many this was previously an error (no change for foreign keys). Trying to add objects to a foreign key relation when the relation field is null is now an error (no change for m2m). Trying to use a relation of any kind when the object isn't saved is now an error (no change for m2m). I think we can squeeze these changes in as bug-fixes. These are slight backwards compatibility changes, but to me it seems that almost always the changes will be visible only in code that isn't working as intended. If these are seen as something likely breaking working code, then I don't see any way to make the APIs consistent. The patch is available from:

def remove(self, *objs, bulk=True):                 if not objs:                     return<ins>                self._check_fk_val()</ins>                 val = self.field.get_foreign_related_value(self.instance)                 old_ids = set()                 for obj in objs:


Question: Unify reverse foreign key and m2m unsaved model querying Description Currently when querying unsaved reverse relations the behavior differs. Using model: class Foo(models.Model): fk = models.ForeignKey('self', relatedname='fkrev') m2m = models.ManyToManyField('self') and test case: print(Foo()()) print(Foo()()) We get [] from the first filter, but an error ValueError: "<Foo: Foo object" needs to have a value for field "fromfoo" before this many-to-many relationship can be used. from the second filter. So, m2m fields can't be filtered if the object isn't saved, but reverse fk fields can be filtered. There is a (slightly stale) patch for which makes fk fields and m2m fields work consistently. The changes in behavior are: Nullable many-to-many and foreign key relations will return an empty queryset when the relation field is null. For many-to-many this was previously an error (no change for foreign keys). Trying to add objects to a foreign key relation when the relation field is null is now an error (no change for m2m). Trying to use a relation of any kind when the object isn't saved is now an error (no change for m2m). I think we can squeeze these changes in as bug-fixes. These are slight backwards compatibility changes, but to me it seems that almost always the changes will be visible only in code that isn't working as intended. If these are seen as something likely breaking working code, then I don't see any way to make the APIs consistent. The patch is available from:

remove.alters_data = True              def clear(self, *, bulk=True):<ins>                self._check_fk_val()</ins>                 self._clear(self, bulk)              clear.alters_data = True


Question: Unify reverse foreign key and m2m unsaved model querying Description Currently when querying unsaved reverse relations the behavior differs. Using model: class Foo(models.Model): fk = models.ForeignKey('self', relatedname='fkrev') m2m = models.ManyToManyField('self') and test case: print(Foo()()) print(Foo()()) We get [] from the first filter, but an error ValueError: "<Foo: Foo object" needs to have a value for field "fromfoo" before this many-to-many relationship can be used. from the second filter. So, m2m fields can't be filtered if the object isn't saved, but reverse fk fields can be filtered. There is a (slightly stale) patch for which makes fk fields and m2m fields work consistently. The changes in behavior are: Nullable many-to-many and foreign key relations will return an empty queryset when the relation field is null. For many-to-many this was previously an error (no change for foreign keys). Trying to add objects to a foreign key relation when the relation field is null is now an error (no change for m2m). Trying to use a relation of any kind when the object isn't saved is now an error (no change for m2m). I think we can squeeze these changes in as bug-fixes. These are slight backwards compatibility changes, but to me it seems that almost always the changes will be visible only in code that isn't working as intended. If these are seen as something likely breaking working code, then I don't see any way to make the APIs consistent. The patch is available from:

_clear.alters_data = True          def set(self, objs, *, bulk=True, clear=False):<ins>            self._check_fk_val()</ins>             # Force evaluation of `objs` in case it's a queryset whose value             # could be affected by `manager.clear()`. Refs #19816.             objs = tuple(objs)


Question: () generates invalid SQL. Description (last modified by M1ha Shvn) Since this commit (...) method returns incorrect SQL removing first and last symbols instead of absent breakets. Adding Subquery().query.subquery = True attribute fixes the problem. From my point of view, it should be set in Subquery constructor. from import connection from import App q = Subquery(()) print(str(q.query)) print(q.assql(q.query.getcompiler('default'), connection)) q.query.subquery = True print(q.assql(q.query.getcompiler('default'), connection)) ('(SELECT "appsapp"."id", "appsapp"."name" FROM "apps_app")', ())

def __init__(self, queryset, output_field=None, **extra):         # Allow the usage of both QuerySet and sql.Query objects.        self.query = getattr(queryset, 'query', queryset)        self.query = getattr(queryset, 'query', queryset).clone()        self.query.subquery = True         self.extra = extra         super().__init__(output_field)


Question: File upload crash when a file extension contains null characters. Description A 2.5M file uploaded with a raw null byte anyplace after the . in its filename means that Django attempts to create a tempfile with that same "extension," which errors out with ValueError: embedded null byte. It's almost certainly a violation of RFC to have a filename with a raw null byte in it, but it shouldn't result in a 500 when parsing the form. Here's code to generate a bad request: #!/usr/bin/env python3 import io import requests contents = io.StringIO("." (1024 1024 * 3)) files = {"docfile": (b"!", contents, "text/plain")} req = requests.Request("POST", "http://localhost:8000/", files=files, data={}) prepared = req.prepare() body = assert isinstance(body, bytes) = body.replace(b"!", b"\x00") requests.Session().send(prepared) produces an error with the view: from django import forms from import HttpResponseRedirect from django.shortcuts import render from import csrfexempt class UploadFileForm(forms.Form): docfile = forms.FileField() def index(request): if == 'POST': form = UploadFileForm(request.POST, request.FILES) if form.is_valid(): print(repr(request.FILES['docfile'])) return HttpResponseRedirect('/') else: print("Not valid!") return HttpResponseRedirect('/') else: form = UploadFileForm() return render(request, '', {'form': form}) I'm not sure what the goal is of preserving the "extension" of the uploaded file in the tempfile that is made; if that's important enough a behaviour to keep, some amount of escaping on the parsed-out extension may be necessary.

file_name = html.unescape(file_name)         file_name = file_name.rsplit('/')[-1]         file_name = file_name.rsplit('\\')[-1]<ins>        # Remove non-printable characters.        file_name = ''.join([char for char in file_name if char.isprintable()])</ins>          if file_name in {'', '.', '..'}:             return None


Question: Support prefetchrelated() with Queryset.iterator() Description (last modified by Asif Saifuddin Auvi) I was surprised when I found out that prefetchrelated calls are ignored when using Queryset.iterator. I noticed in the docs here  that it is because the "these two optimizations do not make sense together." That may have been true in the past, but it is definitely not the case now. The iterator allows chunking (by default 2000) and it would be very helpful to prefetch related for each chunk.

import copy import operator import warningsfrom itertools import chainfrom itertools import chain, islice  import django from django.conf import settings


Question: Support prefetchrelated() with Queryset.iterator() Description (last modified by Asif Saifuddin Auvi) I was surprised when I found out that prefetchrelated calls are ignored when using Queryset.iterator. I noticed in the docs here  that it is because the "these two optimizations do not make sense together." That may have been true in the past, but it is definitely not the case now. The iterator allows chunking (by default 2000) and it would be very helpful to prefetch related for each chunk.

from django.db.models.sql.constants import CURSOR, GET_ITERATOR_CHUNK_SIZE from django.db.models.utils import create_namedtuple_class, resolve_callables from django.utils import timezone<ins>from django.utils.deprecation import RemovedInDjango50Warning</ins> from django.utils.functional import cached_property, partition  # The maximum number of results to fetch in a get() query.


Question: Support prefetchrelated() with Queryset.iterator() Description (last modified by Asif Saifuddin Auvi) I was surprised when I found out that prefetchrelated calls are ignored when using Queryset.iterator. I noticed in the docs here  that it is because the "these two optimizations do not make sense together." That may have been true in the past, but it is definitely not the case now. The iterator allows chunking (by default 2000) and it would be very helpful to prefetch related for each chunk.

####################################      def _iterator(self, use_chunked_fetch, chunk_size):        yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)        iterable = self._iterable_class(            self,            chunked_fetch=use_chunked_fetch,            chunk_size=chunk_size or 2000,        )        if not self._prefetch_related_lookups or chunk_size is None:            yield from iterable            return        iterator = iter(iterable)        while results := list(islice(iterator, chunk_size)):            prefetch_related_objects(results, *self._prefetch_related_lookups)            yield from results     def iterator(self, chunk_size=2000):    def iterator(self, chunk_size=None):         """         An iterator over the results from applying this QuerySet to the        database.        """        if chunk_size <= 0:        database. chunk_size must be provided for QuerySets that prefetch        related objects. Otherwise, a default chunk_size of 2000 is supplied.        """        if chunk_size is None:            if self._prefetch_related_lookups:                # When the deprecation ends, replace with:                # raise ValueError(                #     'chunk_size must be provided when using '                #     'QuerySet.iterator() after prefetch_related().'                # )                warnings.warn(                    'Using QuerySet.iterator() after prefetch_related() '                    'without specifying chunk_size is deprecated.',                    category=RemovedInDjango50Warning,                    stacklevel=2,                )        elif chunk_size <= 0:             raise ValueError('Chunk size must be strictly positive.')         use_chunked_fetch = not connections[self.db].settings_dict.get('DISABLE_SERVER_SIDE_CURSORS')         return self._iterator(use_chunked_fetch, chunk_size)


Question: Migration autodetector crashes on models with field named order, but not using orderwithrespectto. Description (last modified by Fabian Büchler) The commit  introduced a new function ModelState.getfield in This converts the field name order to the one defined in options['orderwithrespectto'] automatically, which fails if the model has a field order but isn't using Meta.orderwithrespectto. That is the case for models generated by django-simple-history () for models that are originally using Meta.orderwithrespectto: the resulting historical records model has only order but is not using the Meta option. This shows when running migrate or makemigrations: $ makemigrations --dry-run Waiting for port 'mysql:3306' timeout 1s (attempt 1/60) Port 'mysql:3306' is open Traceback (most recent call last): File "", line 42, in <modulemain() File "", line 36, in main executefromcommandline() File "/usr/local/lib/python3.8/site-", line 425, in executefromcommandline utility.execute() File "/usr/local/lib/python3.8/site-", line 419, in execute self.fetchcommand(subcommand).runfromargv() File "/usr/local/lib/python3.8/site-", line 373, in runfromargv self.execute(args, cmdoptions) File "/usr/local/lib/python3.8/site-", line 417, in execute output = (args, options) File "/usr/local/lib/python3.8/site-", line 90, in wrapped res = handlefunc(args, *kwargs) File "/usr/local/lib/python3.8/site-", line 172, in handle changes = autodetector.changes( File "/usr/local/lib/python3.8/site-", line 43, in changes changes = self.detectchanges(convertapps, graph) File "/usr/local/lib/python3.8/site-", line 189, in detectchanges self.generatealteredfields() File "/usr/local/lib/python3.8/site-", line 928, in generatealteredfields oldfield = [applabel, oldmodelname].getfield(oldfieldname) File "/usr/local/lib/python3.8/site-", line 689, in getfield self.options['orderwithrespectto'] KeyError: 'orderwithrespectto' I believe this could be solved using a bit more defensive code, like: def getfield(self, fieldname): if fieldname == 'order' and 'orderwithrespectto' in self.options: fieldname = self.options['orderwithrespectto'] return [fieldname]

return self.name.lower()      def get_field(self, field_name):        field_name = (            self.options['order_with_respect_to']            if field_name == '_order'            else field_name        )        if field_name == '_order':            field_name = self.options.get('order_with_respect_to', field_name)         return self.fields[field_name]      @classmethod


Question: Messages framework incorrectly serializes/deserializes extratags when it's an empty string Description When a message is serialised and then deserialised with any of the built in storage backends, then extratags=="" is converted to extratags==None. This is because MessageEncoder checks for the truthyness of extratags rather than checking it is not None. To replicate this bug from import settings settings.configure() # Just to allow the following import from import Message from import MessageEncoder, MessageDecoder originalmessage = Message(10, "Here is a message", extratags="") encodedmessage = MessageEncoder().encode(originalmessage) decodedmessage = MessageDecoder().decode(encodedmessage) originalmessage.extratags == "" True decodedmessage.extratags is None True Effect of the bug in application behaviour This error occurred in the wild with a template tag similar to the following: {% if x not in message.extratags %} When the message was displayed as part of a redirect, it had been serialised and deserialized which meant that extratags was None instead of the empty string. This caused an error. It's important to note that this bug affects all of the standard API (, etc. all have a default value of extra_tags equal to "").

# Using 0/1 here instead of False/True to produce more compact json             is_safedata = 1 if isinstance(obj.message, SafeData) else 0             message = [self.message_key, is_safedata, obj.level, obj.message]            if obj.extra_tags:            if obj.extra_tags is not None:                 message.append(obj.extra_tags)             return message         return super().default(obj)


Question: Session ID should be cleansed from error reporting Description the session ID should be cleansed when reporting errors, just like other credentials. A patch is available at  See also and  A quick github search yielded multiple occasions where session IDs ended up in public bug reports:     I am sure you could find many more. This could potentially be exploited by automatically searching for such requests and hijacking the associated accounts.

cleansed_substitute = "********************"     hidden_settings = _lazy_re_compile(        "API|TOKEN|KEY|SECRET|PASS|SIGNATURE", flags=re.I        "API|TOKEN|KEY|SECRET|PASS|SIGNATURE|HTTP_COOKIE", flags=re.I     )      def cleanse_setting(self, key, value):


Question: Session ID should be cleansed from error reporting Description the session ID should be cleansed when reporting errors, just like other credentials. A patch is available at  See also and  A quick github search yielded multiple occasions where session IDs ended up in public bug reports:     I am sure you could find many more. This could potentially be exploited by automatically searching for such requests and hijacking the associated accounts.

Cleanse an individual setting key/value of sensitive content. If the         value is a dictionary, recursively cleanse the keys in that dictionary.         """        try:            is_sensitive = self.hidden_settings.search(key)        except TypeError:            is_sensitive = False        if key == settings.SESSION_COOKIE_NAME:            is_sensitive = True        else:            try:                is_sensitive = self.hidden_settings.search(key)            except TypeError:                is_sensitive = False          if is_sensitive:             cleansed = self.cleansed_substitute


Question: Session ID should be cleansed from error reporting Description the session ID should be cleansed when reporting errors, just like other credentials. A patch is available at  See also and  A quick github search yielded multiple occasions where session IDs ended up in public bug reports:     I am sure you could find many more. This could potentially be exploited by automatically searching for such requests and hijacking the associated accounts.

return {}         return {k: self.cleanse_setting(k, v) for k, v in request.META.items()} <ins>    def get_safe_cookies(self, request):        """        Return a dictionary of request.COOKIES with sensitive values redacted.        """        if not hasattr(request, "COOKIES"):            return {}        return {k: self.cleanse_setting(k, v) for k, v in request.COOKIES.items()}</ins>     def is_active(self, request):         """         This filter is to add safety in production environments (i.e. DEBUG


Question: Session ID should be cleansed from error reporting Description the session ID should be cleansed when reporting errors, just like other credentials. A patch is available at  See also and  A quick github search yielded multiple occasions where session IDs ended up in public bug reports:     I am sure you could find many more. This could potentially be exploited by automatically searching for such requests and hijacking the associated accounts.

"frames": frames,             "request": self.request,             "request_meta": self.filter.get_safe_request_meta(self.request),<ins>            "request_COOKIES_items": self.filter.get_safe_cookies(self.request).items(),</ins>             "user_str": user_str,             "filtered_POST_items": list(                 self.filter.get_post_parameters(self.request).items()


Question: Session ID should be cleansed from error reporting Description the session ID should be cleansed when reporting errors, just like other credentials. A patch is available at  See also and  A quick github search yielded multiple occasions where session IDs ended up in public bug reports:     I am sure you could find many more. This could potentially be exploited by automatically searching for such requests and hijacking the associated accounts.

if self.request is not None:             c["request_GET_items"] = self.request.GET.items()             c["request_FILES_items"] = self.request.FILES.items()<del>            c["request_COOKIES_items"] = self.request.COOKIES.items()</del>             c["request_insecure_uri"] = self._get_raw_insecure_uri()             c["raising_view_name"] = get_caller(self.request)


Question: bulkupdate() does not work with plain F('...') expressions. Description Repro: assign plain F(...) to some model instance field save with bulkupdate Example: Code highlighting: from import SelfRef o = ().first() o.c8 = F('name') # model has char fields 'c8' and 'name' SelfRef.objects.bulkupdate([o], ['c8']) 1 o.refreshfromdb() o.c8 'F(name)' from import connection connection.queries[-2] {'sql': 'UPDATE "exampleappselfref" SET "c8" = CASE WHEN ("exampleappselfref"."id" = ) THEN \'F(name)\' ELSE NULL END WHERE "exampleappselfref"."id" IN ()', 'time': '0.001'} The created SQL contains the string repr of F(), instead of resolving to the column name. Looking at the source code, the culprit seems to be a too narrow type check in  It works, if the type check gets replaced by one of these: Code highlighting: if not hasattr(attr, 'resolve_expression'): ... if not isinstance(attr, (Expression, F)): ...

from django.db.models import AutoField, DateField, DateTimeField, sql from django.db.models.constants import LOOKUP_SEP, OnConflict from django.db.models.deletion import Collectorfrom django.db.models.expressions import Case, Expression, F, Ref, Value, Whenfrom django.db.models.expressions import Case, F, Ref, Value, When from django.db.models.functions import Cast, Trunc from django.db.models.query_utils import FilteredRelation, Q from django.db.models.sql.constants import CURSOR, GET_ITERATOR_CHUNK_SIZE


Question: bulkupdate() does not work with plain F('...') expressions. Description Repro: assign plain F(...) to some model instance field save with bulkupdate Example: Code highlighting: from import SelfRef o = ().first() o.c8 = F('name') # model has char fields 'c8' and 'name' SelfRef.objects.bulkupdate([o], ['c8']) 1 o.refreshfromdb() o.c8 'F(name)' from import connection connection.queries[-2] {'sql': 'UPDATE "exampleappselfref" SET "c8" = CASE WHEN ("exampleappselfref"."id" = ) THEN \'F(name)\' ELSE NULL END WHERE "exampleappselfref"."id" IN ()', 'time': '0.001'} The created SQL contains the string repr of F(), instead of resolving to the column name. Looking at the source code, the culprit seems to be a too narrow type check in  It works, if the type check gets replaced by one of these: Code highlighting: if not hasattr(attr, 'resolve_expression'): ... if not isinstance(attr, (Expression, F)): ...

when_statements = []                 for obj in batch_objs:                     attr = getattr(obj, field.attname)                    if not isinstance(attr, Expression):                    if not hasattr(attr, 'resolve_expression'):                         attr = Value(attr, output_field=field)                     when_statements.append(When(pk=obj.pk, then=attr))                 case_statement = Case(*when_statements, output_field=field)


Question: Introduce empty slots protocol for SafeString & SafeData Description (last modified by Keryn Knight) This is a case-by-case proposal ultimately referencing Because SafeString is used a lot and is otherwise supposed to be treatable as a untainted str we should be able to (AFAIK) update it + it's inheritance chain to use slots = () whilst still allowing custom subclasses of either to add additional attributes. By defining slots as empty on SafeString (and SafeData) we'd avoid creation of a dict on the instance, which mirrors the str() behaviour. According to pympler, currently in Python 3.10 using the following back of the napkins strings: In [4]: s = "test" # this might be interned, as a short string? In [5]: s2 = "test" 100 In [6]: s3 = SafeString("test") In [7]: s4 = SafeString("test" 100) we get: In [8]: asizeof(s) # str Out[8]: 56 In [9]: asizeof(s2) # str Out[9]: 456 In [10]: asizeof(s3) # SafeString Out[10]: 208 In [11]: asizeof(s4) # SafeString Out[11]: 608 But if we swap out the implementation to be slots'd, it looks more like: In [8]: asizeof(s) # str Out[8]: 56 In [9]: asizeof(s2) # str Out[9]: 456 In [10]: asizeof(s3) # SafeString Out[10]: 104 In [11]: asizeof(s4) # SafeString Out[11]: 504 So we're "saving" 104 bytes per SafeString created, by the look of it. I presume it to be some fun implementation detail of something somewhere that it is allegedly accounting for more than 64 bytes, which is the asizeof({}) A quick and dirty check over the test suite suggests that for me locally, running tests in 512.912s accounted for 949.0 MB of SafeStrings, checked by just incrementing a global integer of bytes (using and --parallel=1) and piping that to filesizeformat, so y'know, room for error. After the patch, the same tests accounted for 779.4 MB of SafeString, "saving" 170 MB overall. The only functionality this would preclude -- as far as I know -- is no longer being able to bind arbitrary values to an instance like so: s = SafeString('test') s.test = 1 which would raise AttributeError if slots were , just like trying to assign attributes to str() directly does. I don't believe this will have any marked performance change, as neither SafeString nor SafeData actually have any extra attributes, only methods. I have a branch which implements this, and tests pass for me locally.

class SafeData:<ins>    __slots__ = ()</ins>     def __html__(self):         """         Return the html representation of a string for interoperability.


Question: Introduce empty slots protocol for SafeString & SafeData Description (last modified by Keryn Knight) This is a case-by-case proposal ultimately referencing Because SafeString is used a lot and is otherwise supposed to be treatable as a untainted str we should be able to (AFAIK) update it + it's inheritance chain to use slots = () whilst still allowing custom subclasses of either to add additional attributes. By defining slots as empty on SafeString (and SafeData) we'd avoid creation of a dict on the instance, which mirrors the str() behaviour. According to pympler, currently in Python 3.10 using the following back of the napkins strings: In [4]: s = "test" # this might be interned, as a short string? In [5]: s2 = "test" 100 In [6]: s3 = SafeString("test") In [7]: s4 = SafeString("test" 100) we get: In [8]: asizeof(s) # str Out[8]: 56 In [9]: asizeof(s2) # str Out[9]: 456 In [10]: asizeof(s3) # SafeString Out[10]: 208 In [11]: asizeof(s4) # SafeString Out[11]: 608 But if we swap out the implementation to be slots'd, it looks more like: In [8]: asizeof(s) # str Out[8]: 56 In [9]: asizeof(s2) # str Out[9]: 456 In [10]: asizeof(s3) # SafeString Out[10]: 104 In [11]: asizeof(s4) # SafeString Out[11]: 504 So we're "saving" 104 bytes per SafeString created, by the look of it. I presume it to be some fun implementation detail of something somewhere that it is allegedly accounting for more than 64 bytes, which is the asizeof({}) A quick and dirty check over the test suite suggests that for me locally, running tests in 512.912s accounted for 949.0 MB of SafeStrings, checked by just incrementing a global integer of bytes (using and --parallel=1) and piping that to filesizeformat, so y'know, room for error. After the patch, the same tests accounted for 779.4 MB of SafeString, "saving" 170 MB overall. The only functionality this would preclude -- as far as I know -- is no longer being able to bind arbitrary values to an instance like so: s = SafeString('test') s.test = 1 which would raise AttributeError if slots were , just like trying to assign attributes to str() directly does. I don't believe this will have any marked performance change, as neither SafeString nor SafeData actually have any extra attributes, only methods. I have a branch which implements this, and tests pass for me locally.

A str subclass that has been specifically marked as "safe" for HTML output     purposes.     """<ins>    __slots__ = ()</ins>     def __add__(self, rhs):         """         Concatenating a safe string with another safe bytestring or


Question: aggregate() with 'default' after annotate() crashes. Description I saw this on a PostgreSQL project and reproduced it with SQLite. Django 4.0.1. Annotate (anything) then aggregate works fine: $ shell Python 3.10.2 (main, Jan 21 2022, 19:45:54) [Clang 13.0.0 (clang-1300.0.29.30)] Type 'copyright', 'credits' or 'license' for more information IPython 7.30.1 -- An enhanced Interactive Python. Type '?' for help. In [1]: from import In [2]: from import In [3]: from import * In [4]: () Out[4]: 95 In [5]: Book.objects.annotate(idx=F("id")).aggregate(Sum("id")) Out[5]: {'idsum': 4560} But add the aggregate classes’ default argument (new in 4.0), and it breaks: In [6]: Book.objects.annotate(idx=F("id")).aggregate(Sum("id", default=0)) OperationalError Traceback (most recent call last) ... OperationalError: near "FROM": syntax error The generated SQL: In [7]: %debug 414 return Database.Cursor.execute(self, query) 415 query = self.convertquery(query) --416 return Database.Cursor.execute(self, query, params) 417 418 def executemany(self, query, paramlist): ipdbquery 'SELECT FROM (SELECT "corebook"."id" AS "idx", COALESCE(SUM("corebook"."id"), ?) AS "idsum" FROM "core_book") subquery' ipdbparams (0,) ipdbThe “long form” using Coalesce works: In [8]: Book.objects.annotate(idx=F("id")).aggregate(x=Coalesce(Sum("id"), 0)) Out[8]: {'x': 4560}

if hasattr(default, 'resolve_expression'):             default = default.resolve_expression(query, allow_joins, reuse, summarize)         c.default = None  # Reset the default argument before wrapping.        return Coalesce(c, default, output_field=c._output_field_or_none)        coalesce = Coalesce(c, default, output_field=c._output_field_or_none)        coalesce.is_summary = c.is_summary        return coalesce      @property     def default_alias(self):