Question: Case-When aggregation over aggregated fields doesn't work since 4.2 Description (last modified by Denis Roldán) This same QuerySet was working on 4.1.X and prior versions and stopped working on 4.2.X: annotatedusers = usersqs.annotate( totalthings=Subquery( (userid=OuterRef("pk")).annotate( totalobjs=F("total") ).values("totalobjs") ) ) annotatedusers.aggregate( sumtotalobjs=Sum("totalthings"), avgconversionrate=Case( When( sumtotalobjs=0, then=0, ), default=Round( (Sum("sumtotalconfirmedobjs") / Sum("sumtotalobjs")) 100, 2 ), outputfield=FloatField(), ) ) As you can see sumtotalobjs is an aggregated field that is also used on a second field to calculate the conversion rate. To avoid a zero division problem, we were using a Case-When clause over that field. It works well on any 4.1 and prior versions but stopped working since 4.2, raising a FieldError like: Cannot resolve keyword 'sumtotalobjs' into field The bug is reproducible with an extra test on the django aggregation test suite: def testreferencedgroupbyaggregationoverannotation(self): totalbooksqs = ( (authorspk=OuterRef("pk")) .orderby() .values("pk") .annotate(total=Count("pk")) .values("total") ) annotatedauthors = Author.objects.annotate( totalbooks=Subquery(totalbooksqs.annotate( totalbooks=F("total") ).values("totalbooks")), totalbooksa=Subquery(( nameistartswith="a" ).annotate( totalbooksa=F("total") ).values("totalbooksa")), ).values( "pk", "totalbooks", "totalbooksa", ).orderby("-totalbooks") totals = annotatedauthors.aggregate( sumtotalbooks=Sum("totalbooks"), sumtotalbooksa=Sum("totalbooksa"), aovertotalrate=Case( When( sumtotalbooks=0, then=0, ), default=Round( (Sum("totalbooksa") / Sum("totalbooks")) 100, 2 ), outputfield=FloatField(), ), ) self.assertEqual(totals['sumtotalbooks'], 3) self.assertEqual(totals['sumtotalbooksa'], 0) self.assertEqual(totals['aovertotalrate'], 0) Thanks for the support!

"""         if not aggregate_exprs:             return {}        aggregates = {}        # Store annotation mask prior to temporarily adding aggregations for        # resolving purpose to facilitate their subsequent removal.        refs_subquery = False        replacements = {}        annotation_select_mask = self.annotation_select_mask         for alias, aggregate_expr in aggregate_exprs.items():             self.check_alias(alias)             aggregate = aggregate_expr.resolve_expression(


Question: Case-When aggregation over aggregated fields doesn't work since 4.2 Description (last modified by Denis Roldán) This same QuerySet was working on 4.1.X and prior versions and stopped working on 4.2.X: annotatedusers = usersqs.annotate( totalthings=Subquery( (userid=OuterRef("pk")).annotate( totalobjs=F("total") ).values("totalobjs") ) ) annotatedusers.aggregate( sumtotalobjs=Sum("totalthings"), avgconversionrate=Case( When( sumtotalobjs=0, then=0, ), default=Round( (Sum("sumtotalconfirmedobjs") / Sum("sumtotalobjs")) 100, 2 ), outputfield=FloatField(), ) ) As you can see sumtotalobjs is an aggregated field that is also used on a second field to calculate the conversion rate. To avoid a zero division problem, we were using a Case-When clause over that field. It works well on any 4.1 and prior versions but stopped working since 4.2, raising a FieldError like: Cannot resolve keyword 'sumtotalobjs' into field The bug is reproducible with an extra test on the django aggregation test suite: def testreferencedgroupbyaggregationoverannotation(self): totalbooksqs = ( (authorspk=OuterRef("pk")) .orderby() .values("pk") .annotate(total=Count("pk")) .values("total") ) annotatedauthors = Author.objects.annotate( totalbooks=Subquery(totalbooksqs.annotate( totalbooks=F("total") ).values("totalbooks")), totalbooksa=Subquery(( nameistartswith="a" ).annotate( totalbooksa=F("total") ).values("totalbooksa")), ).values( "pk", "totalbooks", "totalbooksa", ).orderby("-totalbooks") totals = annotatedauthors.aggregate( sumtotalbooks=Sum("totalbooks"), sumtotalbooksa=Sum("totalbooksa"), aovertotalrate=Case( When( sumtotalbooks=0, then=0, ), default=Round( (Sum("totalbooksa") / Sum("totalbooks")) 100, 2 ), outputfield=FloatField(), ), ) self.assertEqual(totals['sumtotalbooks'], 3) self.assertEqual(totals['sumtotalbooksa'], 0) self.assertEqual(totals['aovertotalrate'], 0) Thanks for the support!

)             if not aggregate.contains_aggregate:                 raise TypeError("%s is not an aggregate expression" % alias)            aggregates[alias] = aggregate            # Temporarily add aggregate to annotations to allow remaining            # members of `aggregates` to resolve against each others.            self.append_annotation_mask([alias])            refs_subquery |= any(                getattr(self.annotations[ref], "subquery", False)                for ref in aggregate.get_refs()            )            aggregate = aggregate.replace_expressions(replacements)            self.annotations[alias] = aggregate            replacements[Ref(alias, aggregate)] = aggregate        # Stash resolved aggregates now that they have been allowed to resolve        # against each other.        aggregates = {alias: self.annotations.pop(alias) for alias in aggregate_exprs}        self.set_annotation_mask(annotation_select_mask)         # Existing usage of aggregation can be determined by the presence of         # selected aggregates but also by filters against aliased aggregates.         _, having, qualify = self.where.split_having_qualify()


Question: Case-When aggregation over aggregated fields doesn't work since 4.2 Description (last modified by Denis Roldán) This same QuerySet was working on 4.1.X and prior versions and stopped working on 4.2.X: annotatedusers = usersqs.annotate( totalthings=Subquery( (userid=OuterRef("pk")).annotate( totalobjs=F("total") ).values("totalobjs") ) ) annotatedusers.aggregate( sumtotalobjs=Sum("totalthings"), avgconversionrate=Case( When( sumtotalobjs=0, then=0, ), default=Round( (Sum("sumtotalconfirmedobjs") / Sum("sumtotalobjs")) 100, 2 ), outputfield=FloatField(), ) ) As you can see sumtotalobjs is an aggregated field that is also used on a second field to calculate the conversion rate. To avoid a zero division problem, we were using a Case-When clause over that field. It works well on any 4.1 and prior versions but stopped working since 4.2, raising a FieldError like: Cannot resolve keyword 'sumtotalobjs' into field The bug is reproducible with an extra test on the django aggregation test suite: def testreferencedgroupbyaggregationoverannotation(self): totalbooksqs = ( (authorspk=OuterRef("pk")) .orderby() .values("pk") .annotate(total=Count("pk")) .values("total") ) annotatedauthors = Author.objects.annotate( totalbooks=Subquery(totalbooksqs.annotate( totalbooks=F("total") ).values("totalbooks")), totalbooksa=Subquery(( nameistartswith="a" ).annotate( totalbooksa=F("total") ).values("totalbooksa")), ).values( "pk", "totalbooks", "totalbooksa", ).orderby("-totalbooks") totals = annotatedauthors.aggregate( sumtotalbooks=Sum("totalbooks"), sumtotalbooksa=Sum("totalbooksa"), aovertotalrate=Case( When( sumtotalbooks=0, then=0, ), default=Round( (Sum("totalbooksa") / Sum("totalbooks")) 100, 2 ), outputfield=FloatField(), ), ) self.assertEqual(totals['sumtotalbooks'], 3) self.assertEqual(totals['sumtotalbooksa'], 0) self.assertEqual(totals['aovertotalrate'], 0) Thanks for the support!

isinstance(self.group_by, tuple)             or self.is_sliced             or has_existing_aggregation<ins>            or refs_subquery</ins>             or qualify             or self.distinct             or self.combinator


Question: Allow to override table2model conversion for inspectdb command Description When implementing a custom inspectdb command by inheriting from inspectdb.Command users should be able to override the table name to model conversion logic. Public method for column names already exists:  This would allow overriding the default behaviour of command in a custom inspectdb, for example when table names are already PascalCase (i.e. schema.FooBarBuzz), etc. With default implementation, it will be converted into class SchemaFoobarbuzz(models.Model). Proposed PR:

# 'table_name_filter' is a stealth option         table_name_filter = options.get("table_name_filter") <del>        def table2model(table_name):            return re.sub(r"[^a-zA-Z0-9]", "", table_name.title())</del>         with connection.cursor() as cursor:             yield "# This is an auto-generated Django model module."             yield "# You'll have to do the following manually to clean this up:"


Question: Allow to override table2model conversion for inspectdb command Description When implementing a custom inspectdb command by inheriting from inspectdb.Command users should be able to override the table name to model conversion logic. Public method for column names already exists:  This would allow overriding the default behaviour of command in a custom inspectdb, for example when table names are already PascalCase (i.e. schema.FooBarBuzz), etc. With default implementation, it will be converted into class SchemaFoobarbuzz(models.Model). Proposed PR:

yield "# The error was: %s" % e                     continue                 model_name = table2model(table_name)                model_name = self.normalize_table_name(table_name)                 yield ""                 yield ""                 yield "class %s(models.Model):" % model_name


Question: Allow to override table2model conversion for inspectdb command Description When implementing a custom inspectdb command by inheriting from inspectdb.Command users should be able to override the table name to model conversion logic. Public method for column names already exists:  This would allow overriding the default behaviour of command in a custom inspectdb, for example when table names are already PascalCase (i.e. schema.FooBarBuzz), etc. With default implementation, it will be converted into class SchemaFoobarbuzz(models.Model). Proposed PR:

rel_to = (                             "self"                             if ref_db_table == table_name                            else table2model(ref_db_table)                            else self.normalize_table_name(ref_db_table)                         )                         if rel_to in known_models:                             field_type = "%s(%s" % (rel_type, rel_to)


Question: Allow to override table2model conversion for inspectdb command Description When implementing a custom inspectdb command by inheriting from inspectdb.Command users should be able to override the table name to model conversion logic. Public method for column names already exists:  This would allow overriding the default behaviour of command in a custom inspectdb, for example when table names are already PascalCase (i.e. schema.FooBarBuzz), etc. With default implementation, it will be converted into class SchemaFoobarbuzz(models.Model). Proposed PR:

return new_name, field_params, field_notes <ins>    def normalize_table_name(self, table_name):        """Translate the table name to a Python-compatible model name."""        return re.sub(r"[^a-zA-Z0-9]", "", table_name.title())</ins>     def get_field_type(self, connection, table_name, row):         """         Given the database connection, the table name, and the cursor row


Question: Querying for decimals larger than maxdigits crashes on SQLite Description Introduced in: Refs -- Improved adapting DecimalField values to decimal. Description: I have DecimalField with maxdigits=4. Previously, querying for something larger (e.g. ) would execute the sql and return ObjectNotFound. Now, in 4.2, it throws a decimal.InvalidOperation error, as it tries to quantize the value to have 4 digits. I understand that it doesn't make sense to query for a larger number, but the error that occurs was pretty confusing to me. Also, it is not as easy to check in my application, because I don't have easy access to the maxdigits parameter of the field. In my opinion, the backend should either accept larger values and always return "not found", or the error should be more descriptive, so that it can be caught specifically. Testcase: placed in tests/backends folder and used for git bisect import decimal from import models from import TestCase class DecimalModel(models.Model): decfield = models.DecimalField(decimalplaces=0, maxdigits=4) class InvalidDecimalQuery(TestCase): def testinvaliddecimalquery(self): try: (decfield='') except decimal.InvalidOperation: ("Too large decimal query caused exception.") except DecimalModel.DoesNotExist: pass Stacktrace: Traceback (most recent call last): File "lib/python3.10/site-", line 87, in managermethod return getattr(self.getqueryset(), name)(args, *kwargs) File "lib/python3.10/site-", line 633, in get num = len(clone) File "lib/python3.10/site-", line 380, in len self.fetchall() File "lib/python3.10/site-", line 1881, in fetchall self.resultcache = list(self.iterableclass(self)) File "lib/python3.10/site-", line 91, in iter results = compiler.executesql( File "lib/python3.10/site-", line 1547, in executesql sql, params = () File "lib/python3.10/site-", line 762, in assql self.compile() if is not None else ("", []) File "lib/python3.10/site-", line 544, in compile sql, params = (self, self.connection) File "lib/python3.10/site-", line 145, in assql sql, params = compiler.compile(child) File "lib/python3.10/site-", line 544, in compile sql, params = (self, self.connection) File "lib/python3.10/site-", line 357, in assql return super().assql(compiler, connection) File "lib/python3.10/site-", line 225, in assql rhssql, rhsparams = self.processrhs(compiler, connection) File "lib/python3.10/site-", line 126, in processrhs return self.getdbpreplookup(value, connection) File "lib/python3.10/site-", line 254, in getdbpreplookup else [getdbprepvalue(value, connection, prepared=True)], File "lib/python3.10/site-", line 1761, in getdbprepvalue return ( File "lib/python3.10/site-", line 574, in adaptdecimalfieldvalue return utils.formatnumber(value, maxdigits, decimalplaces) File "lib/python3.10/site-", line 304, in formatnumber value = value.quantize( decimal.InvalidOperation: [<class 'decimal.InvalidOperation']

)         return decimal_value     def get_db_prep_value(self, value, connection, prepared=False):        if not prepared:            value = self.get_prep_value(value)    def get_db_prep_save(self, value, connection):         if hasattr(value, "as_sql"):             return value         return connection.ops.adapt_decimalfield_value(            value, self.max_digits, self.decimal_places            self.to_python(value), self.max_digits, self.decimal_places         )      def get_prep_value(self, value):


Question: ModelAdmin: Error message for readonlyfields's check does not include the field name Description When subclassing a ModelAdmin, the current error message for the readonlyfields would indicate the index of the value at fault but it will not include the field's name (from the test suite): The value of 'readonlyfields[0]' is not a callable, an attribute of 'CityInline', or an attribute of 'adminchecks.City'. Other fields like listeditable, rawidfields, listdisplay, etc. would also include this value: The value of 'listeditable[0]' refers to 'originalrelease', which is not contained in 'listdisplay'. It would be good if we can unify this and include the field name in the readonlyfields checks, it also eases the understanding of the error when using the framework.

except FieldDoesNotExist:                 return [                     checks.Error(                        "The value of '%s' is not a callable, an attribute of "                        "'%s', or an attribute of '%s'."                        "The value of '%s' refers to '%s', which is not a callable, "                        "an attribute of '%s', or an attribute of '%s'."                         % (                             label,                            field_name,                             obj.__class__.__name__,                             obj.model._meta.label,                         ),


Question: On databases lacking XOR, Q(…) ^ Q(…) ^ Q(…) wrongly interpreted as exactly-one rather than parity Description On databases that don’t natively support XOR, such as PostgreSQL, Django generates incorrect fallback SQL for Q(…) ^ Q(…) ^ Q(…) with more than 2 arguments. The correct interpretation, and the interpretation of databases natively supporting XOR (e.g. MySQL), is that a ^ b ^ c is true when an odd number of the arguments are true. But Django’s fallback interpretation is that a ^ b ^ c is true when exactly one argument is true: from import Q from import Client (Q(id=37)).count() 1 (Q(id=37) ^ Q(id=37)).count() 0 (Q(id=37) ^ Q(id=37) ^ Q(id=37)).count() 0 (Q(id=37) ^ Q(id=37) ^ Q(id=37) ^ Q(id=37)).count() 0 (Q(id=37) ^ Q(id=37) ^ Q(id=37) ^ Q(id=37) ^ Q(id=37)).count() 0 (Expected: 1, 0, 1, 0, 1.) This was introduced in .

from django.core.exceptions import EmptyResultSet, FullResultSet from django.db.models.expressions import Case, When<ins>from django.db.models.functions import Mod</ins> from django.db.models.lookups import Exact from django.utils import tree from django.utils.functional import cached_property


Question: On databases lacking XOR, Q(…) ^ Q(…) ^ Q(…) wrongly interpreted as exactly-one rather than parity Description On databases that don’t natively support XOR, such as PostgreSQL, Django generates incorrect fallback SQL for Q(…) ^ Q(…) ^ Q(…) with more than 2 arguments. The correct interpretation, and the interpretation of databases natively supporting XOR (e.g. MySQL), is that a ^ b ^ c is true when an odd number of the arguments are true. But Django’s fallback interpretation is that a ^ b ^ c is true when exactly one argument is true: from import Q from import Client (Q(id=37)).count() 1 (Q(id=37) ^ Q(id=37)).count() 0 (Q(id=37) ^ Q(id=37) ^ Q(id=37)).count() 0 (Q(id=37) ^ Q(id=37) ^ Q(id=37) ^ Q(id=37)).count() 0 (Q(id=37) ^ Q(id=37) ^ Q(id=37) ^ Q(id=37) ^ Q(id=37)).count() 0 (Expected: 1, 0, 1, 0, 1.) This was introduced in .

# Convert if the database doesn't support XOR:             #   a XOR b XOR c XOR ...             # to:            #   (a OR b OR c OR ...) AND (a + b + c + ...) == 1            #   (a OR b OR c OR ...) AND MOD(a + b + c + ..., 2) == 1            # The result of an n-ary XOR is true when an odd number of operands            # are true.             lhs = self.__class__(self.children, OR)             rhs_sum = reduce(                 operator.add,                 (Case(When(c, then=1), default=0) for c in self.children),             )            if len(self.children) > 2:                rhs_sum = Mod(rhs_sum, 2)             rhs = Exact(1, rhs_sum)             return self.__class__([lhs, rhs], AND, self.negated).as_sql(                 compiler, connection


Question: Right() function on Oracle and SQLite returns improper value when the length is zero. Description Hi I have found that the RIGHT database function on Oracle returns the whole string instead of empty string when the given length is 0. You can't explicitly give 0 to the RIGHT function, but it may be computed by the database. Basic example (you can use any model with a CharField): from import Right, Length MyModel.objects.annotate(suffix=Right("foo", Length("foo") - Length("foo"))) On PostgreSQL this will return an empty string under the field suffix, but on Oracle this will return the whole contents of the field foo. This is because Django uses the SUBSTR function on Oracle by multiplying the given length value by -1 and giving it as a position argument. I think it is not intended behavior and it should return the empty string as PostgreSQL does. Or at least be documented as a Note in the Right function documentation.

def get_substr(self):         return Substr(            self.source_expressions[0], self.source_expressions[1] * Value(-1)            self.source_expressions[0],            self.source_expressions[1] * Value(-1),            self.source_expressions[1],         )


Question: Diamond inheritance causes duplicated PK error when creating an object, if the primary key field has a default. Description Hi, I'm not sure if this is a bug or an unsupported feature. But I looked into the source code and now have a pretty good understanding of what is happening. My business code uses a diamond shape inheritance to model different types of user posts: UserPost, ImagePost, VideoPost, and MixedContentPost. The inheritance goes like this: both ImagePost and VideoPost extend from UserPost, and the MixedContentPost inherits from ImagePost and VideoPost. All of them are concrete models I read the doc and expected it to work, similar to the example class Piece(models.Model): pass class Article(Piece): articlepiece = models.OneToOneField(Piece, ondelete=models.CASCADE, parentlink=True) ... class Book(Piece): bookpiece = models.OneToOneField(Piece, ondelete=models.CASCADE, parentlink=True) ... class BookReview(Book, Article): pass However, I found out that the doc's example only works when these models use a primary key field that does not have a default. In my case, we are using a UUIDField as the primary key with a default of uuid4. Trying to create a BookReview in our case, causes a UNIQUE constraint failed error, because django tries to create the Piece object twice, with the same uuid. The same behavior is found if I used a AutoField on Piece, with a custom default function, such as id = 99 def increment(): global id id += 1 return id This default function makes no sense in practice, but I just use it here to illustrate the root cause of the problem: The savetable method in has a like this: # Skip an UPDATE when adding an instance and primary key has a default. if ( not raw and not forceinsert and and and is not NOTPROVIDED ): forceinsert = True When a default is not present, which is the case of the documentation example, Django will first insert the first instance of the Piece object, and then for the second one, since forceinsert is False, savetable tries an update and goes through. Therefore there is not duplicate. However, if a default is present, then the second Piece becomes an insertion as well (because and is not NOTPROVIDED, forceinsert is True). This causes a duplication on the primary key. On the other hand, insertparent does an in-order traversal calling savetable on each node, so even in the no-default pk case, it is calling a redundant update on the root node after the insertion.. So which function is at fault? The source code savetable assumes that if you are calling it with a default pk then you can skip an update. This assumption looks weird to me: why only when there IS a default pk you can skip update? Why not just skip update as long as we know we are inserting? (via ) Is it just to make it special so that AutoField works? If savetable's responsibility is to try updating before inserting, except when the params force it to do an update or insert, then it shouldn't override that behavior by this self-assumeption within it. I think the solution is to simply move the check to savebase. And don't do this check in saveparents. Like this: def savebase( self, raw=False, forceinsert=False, forceupdate=False, using=None, updatefields=None, ): """ Handle the parts of saving which should be done only once per save, yet need to be done in raw saves, too. This includes some sanity checks and signal sending. The 'raw' argument is telling savebase not to save any parent models and not to do any changes to the values before save. This is used by fixture loading. """ using = using or router.dbforwrite(, instance=self) assert not (forceinsert and (forceupdate or updatefields)) assert updatefields is None or updatefields cls = origin = # Skip proxies, but keep the origin as the proxy model. if cls = meta = if not meta.autocreated: ( sender=origin, instance=self, raw=raw, using=using, updatefields=updatefields, ) # A transaction isn't needed if one query is issued. if meta.parents: contextmanager = (using=using, savepoint=False) else: contextmanager = transaction.markforrollbackonerror(using=using) with contextmanager: parentinserted = False if not raw: parentinserted = self.saveparents(cls, using, updatefields) # Skip an UPDATE when adding an instance and primary key has a default. if ( not raw and not forceinsert and and and is not NOTPROVIDED ): forceinsert = True updated = self.savetable( raw, cls, forceinsert or parentinserted, forceupdate, using, updatefields, ) # Store the database on which the object was saved = using # Once saved, this is no longer a to-be- instance. = False # Signal that the save is complete if not meta.autocreated: ( sender=origin, instance=self, created=(not updated), updatefields=updatefields, raw=raw, using=using, ) savebase.alters_data = True I have never contributed to Django before. If you think I'm right on this one I'll look into creating a PR.

save_base.alters_data = True     def _save_parents(self, cls, using, update_fields):    def _save_parents(self, cls, using, update_fields, updated_parents=None):         """Save all the parents of cls using values from self."""         meta = cls._meta         inserted = False        if updated_parents is None:            updated_parents = {}         for parent, field in meta.parents.items():             # Make sure the link fields are synced between parent and self.             if (


Question: Diamond inheritance causes duplicated PK error when creating an object, if the primary key field has a default. Description Hi, I'm not sure if this is a bug or an unsupported feature. But I looked into the source code and now have a pretty good understanding of what is happening. My business code uses a diamond shape inheritance to model different types of user posts: UserPost, ImagePost, VideoPost, and MixedContentPost. The inheritance goes like this: both ImagePost and VideoPost extend from UserPost, and the MixedContentPost inherits from ImagePost and VideoPost. All of them are concrete models I read the doc and expected it to work, similar to the example class Piece(models.Model): pass class Article(Piece): articlepiece = models.OneToOneField(Piece, ondelete=models.CASCADE, parentlink=True) ... class Book(Piece): bookpiece = models.OneToOneField(Piece, ondelete=models.CASCADE, parentlink=True) ... class BookReview(Book, Article): pass However, I found out that the doc's example only works when these models use a primary key field that does not have a default. In my case, we are using a UUIDField as the primary key with a default of uuid4. Trying to create a BookReview in our case, causes a UNIQUE constraint failed error, because django tries to create the Piece object twice, with the same uuid. The same behavior is found if I used a AutoField on Piece, with a custom default function, such as id = 99 def increment(): global id id += 1 return id This default function makes no sense in practice, but I just use it here to illustrate the root cause of the problem: The savetable method in has a like this: # Skip an UPDATE when adding an instance and primary key has a default. if ( not raw and not forceinsert and and and is not NOTPROVIDED ): forceinsert = True When a default is not present, which is the case of the documentation example, Django will first insert the first instance of the Piece object, and then for the second one, since forceinsert is False, savetable tries an update and goes through. Therefore there is not duplicate. However, if a default is present, then the second Piece becomes an insertion as well (because and is not NOTPROVIDED, forceinsert is True). This causes a duplication on the primary key. On the other hand, insertparent does an in-order traversal calling savetable on each node, so even in the no-default pk case, it is calling a redundant update on the root node after the insertion.. So which function is at fault? The source code savetable assumes that if you are calling it with a default pk then you can skip an update. This assumption looks weird to me: why only when there IS a default pk you can skip update? Why not just skip update as long as we know we are inserting? (via ) Is it just to make it special so that AutoField works? If savetable's responsibility is to try updating before inserting, except when the params force it to do an update or insert, then it shouldn't override that behavior by this self-assumeption within it. I think the solution is to simply move the check to savebase. And don't do this check in saveparents. Like this: def savebase( self, raw=False, forceinsert=False, forceupdate=False, using=None, updatefields=None, ): """ Handle the parts of saving which should be done only once per save, yet need to be done in raw saves, too. This includes some sanity checks and signal sending. The 'raw' argument is telling savebase not to save any parent models and not to do any changes to the values before save. This is used by fixture loading. """ using = using or router.dbforwrite(, instance=self) assert not (forceinsert and (forceupdate or updatefields)) assert updatefields is None or updatefields cls = origin = # Skip proxies, but keep the origin as the proxy model. if cls = meta = if not meta.autocreated: ( sender=origin, instance=self, raw=raw, using=using, updatefields=updatefields, ) # A transaction isn't needed if one query is issued. if meta.parents: contextmanager = (using=using, savepoint=False) else: contextmanager = transaction.markforrollbackonerror(using=using) with contextmanager: parentinserted = False if not raw: parentinserted = self.saveparents(cls, using, updatefields) # Skip an UPDATE when adding an instance and primary key has a default. if ( not raw and not forceinsert and and and is not NOTPROVIDED ): forceinsert = True updated = self.savetable( raw, cls, forceinsert or parentinserted, forceupdate, using, updatefields, ) # Store the database on which the object was saved = using # Once saved, this is no longer a to-be- instance. = False # Signal that the save is complete if not meta.autocreated: ( sender=origin, instance=self, created=(not updated), updatefields=updatefields, raw=raw, using=using, ) savebase.alters_data = True I have never contributed to Django before. If you think I'm right on this one I'll look into creating a PR.

and getattr(self, field.attname) is not None             ):                 setattr(self, parent._meta.pk.attname, getattr(self, field.attname))            parent_inserted = self._save_parents(                cls=parent, using=using, update_fields=update_fields            )            updated = self._save_table(                cls=parent,                using=using,                update_fields=update_fields,                force_insert=parent_inserted,            )            if not updated:            if (parent_updated := updated_parents.get(parent)) is None:                parent_inserted = self._save_parents(                    cls=parent,                    using=using,                    update_fields=update_fields,                    updated_parents=updated_parents,                )                updated = self._save_table(                    cls=parent,                    using=using,                    update_fields=update_fields,                    force_insert=parent_inserted,                )                if not updated:                    inserted = True                updated_parents[parent] = updated            elif not parent_updated:                 inserted = True             # Set the parent's PK value to self.             if field:


Question: () doesn't work with selectrelated() on a reverse OneToOneField relation. Description On Django 4.2 calling only() with selectrelated() on a query using the reverse lookup for a OneToOne relation does not generate the correct query. All the fields from the related model are still included in the generated SQL. Sample models: class Main(models.Model): mainfield1 = models.CharField(blank=True, maxlength=45) mainfield2 = models.CharField(blank=True, maxlength=45) mainfield3 = models.CharField(blank=True, maxlength=45) class Secondary(models.Model): main = models.OneToOneField(Main, primarykey=True, relatedname='secondary', ondelete=models.CASCADE) secondaryfield1 = models.CharField(blank=True, maxlength=45) secondaryfield2 = models.CharField(blank=True, maxlength=45) secondaryfield3 = models.CharField(blank=True, maxlength=45) Sample code: Main.objects.selectrelated('secondary').only('mainfield1', 'secondary_secondaryfield1') Generated query on Django 4.2.1: SELECT "bugtestmain"."id", "bugtestmain"."mainfield1", "bugtestsecondary"."mainid", "bugtestsecondary"."secondaryfield1", "bugtestsecondary"."secondaryfield2", "bugtestsecondary"."secondaryfield3" FROM "bugtestmain" LEFT OUTER JOIN "bugtestsecondary" ON ("bugtestmain"."id" = "bugtestsecondary"."mainid") Generated query on Django 4.1.9: SELECT "bugtestmain"."id", "bugtestmain"."mainfield1", "bugtestsecondary"."mainid", "bugtestsecondary"."secondaryfield1" FROM "bugtestmain" LEFT OUTER JOIN "bugtestsecondary" ON ("bugtestmain"."id" = "bugtestsecondary"."main_id")

# Only include fields mentioned in the mask.         for field_name, field_mask in mask.items():             field = opts.get_field(field_name)            field_select_mask = select_mask.setdefault(field, {})            # Retrieve the actual field associated with reverse relationships            # as that's what is expected in the select mask.            if field in opts.related_objects:                field_key = field.field            else:                field_key = field            field_select_mask = select_mask.setdefault(field_key, {})             if field_mask:                 if not field.is_relation:                     raise FieldError(next(iter(field_mask)))


Question: Fields’ help text and errors should be associated with input Description (last modified by Thibaud Colas) With Django’s default field rendering, all field errors are rendered as a list above the field’s label, and help text is rendered after the field’s form element. Example with asp: <ul class="errorlist"<liThis field is required.</li</ul<p<label for="iddurationrequired"Duration required:</label<input type="text" name="durationrequired" required="" id="iddurationrequired"<span class="helptext"Help</span</pOne problem for screen reader users is that the association between the errors and the field, and between the help text and the field, is only communicated visually. This is a failure of either WCAG 2.1 level A SC 1.3.1: Info and Relationships, or SC 3.3.2: Labels or Instructions. More importantly, it just makes it harder than necessary for screen reader users to make use of help text, and to identify error messages. The fix is relatively straightforward – using aria-describedby, as documented in the (non-normative) ARIA1 Using the aria-describedby property to provide a descriptive label for user interface controls technique. Here is another well-known accessibility-oriented UI library that implements this technique: GOV.UK design system – text input with error message. Here is what implementing aria-describedby would look like in the same example as above: <div class="errorlist" id="iddurationrequirederrorlist"<pThis field is required.</p</div<p<label for="iddurationrequired"Duration required:</label<input type="text" name="durationrequired" required="" id="iddurationrequired" aria-describedby="iddurationrequirederrorlist iddurationrequiredhelptext"<span class="helptext" id="iddurationrequired_helptext"Help</span</pWe have additional id attributes, aria-describedby, and errorlist is no longer a <ul. Result in VoiceOver: Unfortunately I tried to have this with the errorlist kept as a ul, but it wasn’t announced by VoiceOver. I haven’t heard of this limitation before so am not sure why that might be the case – I’d appreciate others taking a look if possible.

attrs["required"] = True         if self.field.disabled:             attrs["disabled"] = True<ins>        # If a custom aria-describedby attribute is given and help_text is        # used, the custom aria-described by is preserved so user can set the        # desired order.        if custom_aria_described_by_id := widget.attrs.get("aria-describedby"):            attrs["aria-describedby"] = custom_aria_described_by_id        elif self.field.help_text and self.id_for_label:            attrs["aria-describedby"] = f"{self.id_for_label}_helptext"</ins>         return attrs      @property


Question: Serialization of m2m relation fails with custom manager using selectrelated Description Serialization of many to many relation with custom manager using selectrelated cause FieldError: Field cannot be both deferred and traversed using selectrelated at the same time. Exception is raised because performance optimalization . Workaround is to set simple default manager. However I not sure if this is bug or expected behaviour. class TestTagManager(Manager): def getqueryset(self): qs = super().getqueryset() qs = qs.selectrelated("master") # follow master when retrieving object by default return qs class TestTagMaster(models.Model): name = models.CharField(maxlength=120) class TestTag(models.Model): # default = Manager() # solution is to define custom default manager, which is used by RelatedManager objects = TestTagManager() name = models.CharField(maxlength=120) master = models.ForeignKey(TestTagMaster, ondelete=models.SETNULL, null=True) class Test(models.Model): name = models.CharField(maxlength=120) tags = models.ManyToManyField(TestTag, blank=True) Now when serializing object from import serializers from import TestTag, Test, TestTagMaster tagmaster = (name="master") tag = (name="tag", master=tagmaster) test = (name="test") (tag) () serializers.serialize("json", [test]) Serialize raise exception because is not possible to combine selectrelated and only. File "/opt/venv/lib/python3.11/site-", line 134, in serialize s.serialize(queryset, **options) File "/opt/venv/lib/python3.11/site-", line 167, in serialize self.handlem2mfield(obj, field) File "/opt/venv/lib/python3.11/site-", line 88, in handlem2mfield self.current[] = [m2mvalue(related) for related in m2miter] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-", line 88, in <listcompself.current[] = [m2mvalue(related) for related in m2miter] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-", line 516, in iterator yield from iterable File "/opt/venv/lib/python3.11/site-", line 91, in iter results = compiler.executesql( ^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-", line 1547, in executesql sql, params = () ^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-", line 734, in assql extraselect, orderby, groupby = self.presqlsetup( ^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-", line 84, in presqlsetup self.setupquery(withcolaliases=withcolaliases) File "/opt/venv/lib/python3.11/site-", line 73, in setupquery , self.klassinfo, self.annotationcolmap = self.getselect( ^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-", line 279, in getselect relatedklassinfos = self.getrelatedselections(select, selectmask) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-", line 1209, in getrelatedselections if not selectrelateddescend(f, restricted, requested, selectmask): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-", line 347, in selectrelateddescend raise FieldError( Field cannot be both deferred and traversed using select_related at the same time.

return self._value_from_field(value, value._meta.pk)                  def queryset_iterator(obj, field):                    return getattr(obj, field.name).only("pk").iterator()                    return (                        getattr(obj, field.name).select_related().only("pk").iterator()                    )              m2m_iter = getattr(obj, "_prefetched_objects_cache", {}).get(                 field.name,diff --git a/django/core/serializers/xml_serializer.py b/django/core/serializers/xml_serializer.py-- a/django/core/serializers/xml_serializer.py++ b/django/core/serializers/xml_serializer.py


Question: Serialization of m2m relation fails with custom manager using selectrelated Description Serialization of many to many relation with custom manager using selectrelated cause FieldError: Field cannot be both deferred and traversed using selectrelated at the same time. Exception is raised because performance optimalization . Workaround is to set simple default manager. However I not sure if this is bug or expected behaviour. class TestTagManager(Manager): def getqueryset(self): qs = super().getqueryset() qs = qs.selectrelated("master") # follow master when retrieving object by default return qs class TestTagMaster(models.Model): name = models.CharField(maxlength=120) class TestTag(models.Model): # default = Manager() # solution is to define custom default manager, which is used by RelatedManager objects = TestTagManager() name = models.CharField(maxlength=120) master = models.ForeignKey(TestTagMaster, ondelete=models.SETNULL, null=True) class Test(models.Model): name = models.CharField(maxlength=120) tags = models.ManyToManyField(TestTag, blank=True) Now when serializing object from import serializers from import TestTag, Test, TestTagMaster tagmaster = (name="master") tag = (name="tag", master=tagmaster) test = (name="test") (tag) () serializers.serialize("json", [test]) Serialize raise exception because is not possible to combine selectrelated and only. File "/opt/venv/lib/python3.11/site-", line 134, in serialize s.serialize(queryset, **options) File "/opt/venv/lib/python3.11/site-", line 167, in serialize self.handlem2mfield(obj, field) File "/opt/venv/lib/python3.11/site-", line 88, in handlem2mfield self.current[] = [m2mvalue(related) for related in m2miter] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-", line 88, in <listcompself.current[] = [m2mvalue(related) for related in m2miter] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-", line 516, in iterator yield from iterable File "/opt/venv/lib/python3.11/site-", line 91, in iter results = compiler.executesql( ^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-", line 1547, in executesql sql, params = () ^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-", line 734, in assql extraselect, orderby, groupby = self.presqlsetup( ^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-", line 84, in presqlsetup self.setupquery(withcolaliases=withcolaliases) File "/opt/venv/lib/python3.11/site-", line 73, in setupquery , self.klassinfo, self.annotationcolmap = self.getselect( ^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-", line 279, in getselect relatedklassinfos = self.getrelatedselections(select, selectmask) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-", line 1209, in getrelatedselections if not selectrelateddescend(f, restricted, requested, selectmask): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-", line 347, in selectrelateddescend raise FieldError( Field cannot be both deferred and traversed using select_related at the same time.

self.xml.addQuickElement("object", attrs={"pk": str(value.pk)})                  def queryset_iterator(obj, field):                    return getattr(obj, field.name).only("pk").iterator()                    return (                        getattr(obj, field.name).select_related().only("pk").iterator()                    )              m2m_iter = getattr(obj, "_prefetched_objects_cache", {}).get(                 field.name,


Question: Deprecate formathtml calls without args or kwargs Description (last modified by Michael Howitz) In my experience, a common misuse of formathtml is to format the HTML before calling it: formathtml(f"<i{name}</i") This makes it act like marksafe, allowing data through without escaping. It provides a false sense of security since formathtml is meant to be the "safe way". I propose we deprecate calls to formathtml that don’t pass args or kwargs, and eventually raise a TypeError for such cases. (Following improvement to format_html docs in .)

import html import json import reimport warnings from html.parser import HTMLParser from urllib.parse import parse_qsl, quote, unquote, urlencode, urlsplit, urlunsplit from django.utils.deprecation import RemovedInDjango60Warning from django.utils.encoding import punycode from django.utils.functional import Promise, keep_lazy, keep_lazy_text from django.utils.http import RFC3986_GENDELIMS, RFC3986_SUBDELIMS


Question: Deprecate formathtml calls without args or kwargs Description (last modified by Michael Howitz) In my experience, a common misuse of formathtml is to format the HTML before calling it: formathtml(f"<i{name}</i") This makes it act like marksafe, allowing data through without escaping. It provides a false sense of security since formathtml is meant to be the "safe way". I propose we deprecate calls to formathtml that don’t pass args or kwargs, and eventually raise a TypeError for such cases. (Following improvement to format_html docs in .)

and call mark_safe() on the result. This function should be used instead     of str.format or % interpolation to build up small HTML fragments.     """<ins>    if not (args or kwargs):        # RemovedInDjango60Warning: when the deprecation ends, replace with:        # raise ValueError("args or kwargs must be provided.")        warnings.warn(            "Calling format_html() without passing args or kwargs is deprecated.",            RemovedInDjango60Warning,        )</ins>     args_safe = map(conditional_escape, args)     kwargs_safe = {k: conditional_escape(v) for (k, v) in kwargs.items()}     return mark_safe(format_string.format(*args_safe, **kwargs_safe))


Question: Django Admin with Inlines not using UUIDField default value Description (last modified by Joseph Metzinger) Hello, I am a long time django user, first time bug reporter, so please let me know if I need to do anything else to help get this bug fixed :) I am using Django 3.1.3 and python 3.8.5 and have cerated a toy project to illustrate the bug. I have the following models: class UUIDModel(models.Model): pkid = models.BigAutoField(primarykey=True, editable=False) id = models.UUIDField(default=uuid.uuid4, editable=False, unique=True) class Meta: abstract = True class Thing(UUIDModel): name = models.CharField(maxlength=191) class SubThing(models.Model): name = models.CharField(maxlength=191) thing = models.ForeignKey( 'bugapp.Thing', tofield='id', ondelete = models.CASCADE, relatedname='subthings', ) And the following file: class SubThingInline(admin.StackedInline): model = SubThing class ThingAdmin(admin.ModelAdmin): list_display = ('name',) ordering = ('pkid',) inlines = (SubThingInline,) When logging into the admin, if you delete all of the entries for "subthings", add a name, and save the model, it will work. As soon as you try to add a subthing alongside the main Thing, it fails with the following exception:  It shows that the value of "id" in the Thing model is being set to null. I believe this is a bug in django. Thanks!

to_field = self.instance._meta.get_field(kwargs["to_field"])             else:                 to_field = self.instance._meta.pk            if to_field.has_default():            if to_field.has_default() and (                # Don't ignore a parent's auto-generated key if it's not the                # parent model's pk and form data is provided.                to_field.attname == self.fk.remote_field.model._meta.pk.name                or not form.data            ):                 setattr(self.instance, to_field.attname, None)          form.fields[name] = InlineForeignKeyField(self.instance, **kwargs)


Question: Creating objects with nested MTI crashes. Description (last modified by Mariusz Felisiak) Checking PR I noticed that creating objects with more complicated multi-table inheritance crashes. For example: diff --git index .. a b class ItalianRestaurant(Restaurant): servesgnocchi = models.BooleanField(default=False) 109class ItalianRestaurantManyParents(ItalianRestaurant, Place): 110 pass 111 112 109113class Supplier(Place): customers = models.ManyToManyField(Restaurant, relatedname="provider") diff --git index .. a b from .models import ( 1414 GrandChild, 1515 GrandParent, 1616 ItalianRestaurant, 17 ItalianRestaurantManyParents, 1718 MixinModel, 1819 Parent, 1920 ParkingLot, … … class ModelInheritanceTests(TestCase): # accidentally found). self.assertSequenceEqual((), []) 153 def testcreatediamondmticommonparents(self): 154 with self.assertNumQueries(4): 155 ( 156 name="Ristorante Miron", 157 address="1234 W. Ash", 158 ) 159 def testupdateparentfiltering(self): """ Updating a field of a model subclass doesn't issue an UPDATE crashes with: File "", line 155, in testcreatediamondmticommonparents ( File "", line 87, in managermethod return getattr(self.getqueryset(), name)(args, *kwargs) File "", line 650, in create (forceinsert=True, ) File "", line 814, in save self.savebase( File "", line 876, in savebase parentinserted = self.saveparents(cls, using, updatefields) File "", line 928, in saveparents setattr(self, field.attname, self.getpkval(parent._meta)) AttributeError: 'OneToOneField' object has no attribute 'attname'

if f not in used_fields:                     used_fields[f.name] = f         # Check that parent links in diamond-shaped MTI models don't clash.        for parent_link in cls._meta.parents.values():            if not parent_link:                continue            clash = used_fields.get(parent_link.name) or None            if clash:                errors.append(                    checks.Error(                        f"The field '{parent_link.name}' clashes with the field "                        f"'{clash.name}' from model '{clash.model._meta}'.",                        obj=cls,                        id="models.E006",                    )                )         for f in cls._meta.local_fields:             clash = used_fields.get(f.name) or used_fields.get(f.attname) or None             # Note that we may detect clash between user-defined non-uniquediff --git a/django/db/models/options.py b/django/db/models/options.py-- a/django/db/models/options.py++ b/django/db/models/options.py


Question: Creating objects with nested MTI crashes. Description (last modified by Mariusz Felisiak) Checking PR I noticed that creating objects with more complicated multi-table inheritance crashes. For example: diff --git index .. a b class ItalianRestaurant(Restaurant): servesgnocchi = models.BooleanField(default=False) 109class ItalianRestaurantManyParents(ItalianRestaurant, Place): 110 pass 111 112 109113class Supplier(Place): customers = models.ManyToManyField(Restaurant, relatedname="provider") diff --git index .. a b from .models import ( 1414 GrandChild, 1515 GrandParent, 1616 ItalianRestaurant, 17 ItalianRestaurantManyParents, 1718 MixinModel, 1819 Parent, 1920 ParkingLot, … … class ModelInheritanceTests(TestCase): # accidentally found). self.assertSequenceEqual((), []) 153 def testcreatediamondmticommonparents(self): 154 with self.assertNumQueries(4): 155 ( 156 name="Ristorante Miron", 157 address="1234 W. Ash", 158 ) 159 def testupdateparentfiltering(self): """ Updating a field of a model subclass doesn't issue an UPDATE crashes with: File "", line 155, in testcreatediamondmticommonparents ( File "", line 87, in managermethod return getattr(self.getqueryset(), name)(args, *kwargs) File "", line 650, in create (forceinsert=True, ) File "", line 814, in save self.savebase( File "", line 876, in savebase parentinserted = self.saveparents(cls, using, updatefields) File "", line 928, in saveparents setattr(self, field.attname, self.getpkval(parent._meta)) AttributeError: 'OneToOneField' object has no attribute 'attname'

reverse=True,         include_parents=True,         include_hidden=False,        seen_models=None,        topmost_call=True,     ):         """         Internal helper function to return fields of the model.


Question: Creating objects with nested MTI crashes. Description (last modified by Mariusz Felisiak) Checking PR I noticed that creating objects with more complicated multi-table inheritance crashes. For example: diff --git index .. a b class ItalianRestaurant(Restaurant): servesgnocchi = models.BooleanField(default=False) 109class ItalianRestaurantManyParents(ItalianRestaurant, Place): 110 pass 111 112 109113class Supplier(Place): customers = models.ManyToManyField(Restaurant, relatedname="provider") diff --git index .. a b from .models import ( 1414 GrandChild, 1515 GrandParent, 1616 ItalianRestaurant, 17 ItalianRestaurantManyParents, 1718 MixinModel, 1819 Parent, 1920 ParkingLot, … … class ModelInheritanceTests(TestCase): # accidentally found). self.assertSequenceEqual((), []) 153 def testcreatediamondmticommonparents(self): 154 with self.assertNumQueries(4): 155 ( 156 name="Ristorante Miron", 157 address="1234 W. Ash", 158 ) 159 def testupdateparentfiltering(self): """ Updating a field of a model subclass doesn't issue an UPDATE crashes with: File "", line 155, in testcreatediamondmticommonparents ( File "", line 87, in managermethod return getattr(self.getqueryset(), name)(args, *kwargs) File "", line 650, in create (forceinsert=True, ) File "", line 814, in save self.savebase( File "", line 876, in savebase parentinserted = self.saveparents(cls, using, updatefields) File "", line 928, in saveparents setattr(self, field.attname, self.getpkval(parent._meta)) AttributeError: 'OneToOneField' object has no attribute 'attname'

# implementation and to provide a fast way for Django's internals to         # access specific subsets of fields. <del>        # We must keep track of which models we have already seen. Otherwise we        # could include the same field multiple times from different models.        topmost_call = seen_models is None        if topmost_call:            seen_models = set()        seen_models.add(self.model)</del>         # Creates a cache key composed of all arguments         cache_key = (forward, reverse, include_parents, include_hidden, topmost_call)


Question: Creating objects with nested MTI crashes. Description (last modified by Mariusz Felisiak) Checking PR I noticed that creating objects with more complicated multi-table inheritance crashes. For example: diff --git index .. a b class ItalianRestaurant(Restaurant): servesgnocchi = models.BooleanField(default=False) 109class ItalianRestaurantManyParents(ItalianRestaurant, Place): 110 pass 111 112 109113class Supplier(Place): customers = models.ManyToManyField(Restaurant, relatedname="provider") diff --git index .. a b from .models import ( 1414 GrandChild, 1515 GrandParent, 1616 ItalianRestaurant, 17 ItalianRestaurantManyParents, 1718 MixinModel, 1819 Parent, 1920 ParkingLot, … … class ModelInheritanceTests(TestCase): # accidentally found). self.assertSequenceEqual((), []) 153 def testcreatediamondmticommonparents(self): 154 with self.assertNumQueries(4): 155 ( 156 name="Ristorante Miron", 157 address="1234 W. Ash", 158 ) 159 def testupdateparentfiltering(self): """ Updating a field of a model subclass doesn't issue an UPDATE crashes with: File "", line 155, in testcreatediamondmticommonparents ( File "", line 87, in managermethod return getattr(self.getqueryset(), name)(args, *kwargs) File "", line 650, in create (forceinsert=True, ) File "", line 814, in save self.savebase( File "", line 876, in savebase parentinserted = self.saveparents(cls, using, updatefields) File "", line 928, in saveparents setattr(self, field.attname, self.getpkval(parent._meta)) AttributeError: 'OneToOneField' object has no attribute 'attname'

# Recursively call _get_fields() on each parent, with the same         # options provided in this call.         if include_parents is not False:            # In diamond inheritance it is possible that we see the same model            # from two different routes. In that case, avoid adding fields from            # the same parent again.            parent_fields = set()             for parent in self.parents:                # In diamond inheritance it is possible that we see the same                # model from two different routes. In that case, avoid adding                # fields from the same parent again.                if parent in seen_models:                    continue                 if (                     parent._meta.concrete_model != self.concrete_model                     and include_parents == PROXY_PARENTS


Question: Creating objects with nested MTI crashes. Description (last modified by Mariusz Felisiak) Checking PR I noticed that creating objects with more complicated multi-table inheritance crashes. For example: diff --git index .. a b class ItalianRestaurant(Restaurant): servesgnocchi = models.BooleanField(default=False) 109class ItalianRestaurantManyParents(ItalianRestaurant, Place): 110 pass 111 112 109113class Supplier(Place): customers = models.ManyToManyField(Restaurant, relatedname="provider") diff --git index .. a b from .models import ( 1414 GrandChild, 1515 GrandParent, 1616 ItalianRestaurant, 17 ItalianRestaurantManyParents, 1718 MixinModel, 1819 Parent, 1920 ParkingLot, … … class ModelInheritanceTests(TestCase): # accidentally found). self.assertSequenceEqual((), []) 153 def testcreatediamondmticommonparents(self): 154 with self.assertNumQueries(4): 155 ( 156 name="Ristorante Miron", 157 address="1234 W. Ash", 158 ) 159 def testupdateparentfiltering(self): """ Updating a field of a model subclass doesn't issue an UPDATE crashes with: File "", line 155, in testcreatediamondmticommonparents ( File "", line 87, in managermethod return getattr(self.getqueryset(), name)(args, *kwargs) File "", line 650, in create (forceinsert=True, ) File "", line 814, in save self.savebase( File "", line 876, in savebase parentinserted = self.saveparents(cls, using, updatefields) File "", line 928, in saveparents setattr(self, field.attname, self.getpkval(parent._meta)) AttributeError: 'OneToOneField' object has no attribute 'attname'

reverse=reverse,                     include_parents=include_parents,                     include_hidden=include_hidden,                    seen_models=seen_models,                    topmost_call=False,                 ):                     if (                         not getattr(obj, "parent_link", False)                         or obj.model == self.concrete_model                    ):                    ) and obj not in parent_fields:                         fields.append(obj)                        parent_fields.add(obj)         if reverse and not self.proxy:             # Tree is computed once and cached until the app cache is expired.             # It is composed of a list of fields pointing to the current model


Question: Add system check for filterhorizontal/filtervertical on ManyToManyFields with intermediary models. Description Hi team, I'm a huge fan of Django and have been using it since 0.95 but I stumbled over this one. Neither of  and  call out the requirement to not use ManyToManyField(through="") In the same way:  doesn't call out the consequence that filterhorizontal and filtervertical will stop working if one goes down the pathway of: ManyToManyField(through="") I just wasted half a day chasing this down.

return must_be(                     "a many-to-many field", option=label, obj=obj, id="admin.E020"                 )<ins>            elif not field.remote_field.through._meta.auto_created:                return [                    checks.Error(                        f"The value of '{label}' cannot include the ManyToManyField "                        f"'{field_name}', because that field manually specifies a "                        f"relationship model.",                        obj=obj.__class__,                        id="admin.E013",                    )                ]</ins>             else:                 return []


Question: Apps.clearcache() does not clear getswappablesettingsname cache. Description We use apps.clearcache() in django-stubs to be able to reset the previous state on consequential mypy runs. Code:  But, looks like we don't clear all the object's cache this way, because getswappablesettingsname (which is a functools.lrucachewrapper) is not cleared. I think that this is not correct. .clearcache doc states: Clear all internal caches, for methods that alter the app registry. Looks like that is not the case. I propose to add: self.getswappablesettingsname.cacheclear() line to def clearcache. If others agree, I will make a PR. Original discussion:

This is mostly used in tests.         """<ins>        self.get_swappable_settings_name.cache_clear()</ins>         # Call expire cache on each model. This will purge         # the relation tree and the fields cache.         self.get_models.cache_clear()


Question: Add a system check for unmatched URL paths' < / Description Currently, unmatched angle brackets are silently ignored: path('<int:question_id/vote/', , name='vote'), This leads to a frustrating debugging experience, where a 404 or NoReverseMatch could occur and only the eagle-eyed could spot why. Similar to the ^/$ system check in , I propose a new check in () for unmatched < or . It's rather unlikely those characters are legitimately used in URLs. (Inspired by a new user making this mistake and asking for help on the forum.)

return None      def check(self):        warnings = self._check_pattern_startswith_slash()        warnings = [            *self._check_pattern_startswith_slash(),            *self._check_pattern_unmatched_angle_brackets(),        ]         route = self._route         if "(?P<" in route or route.startswith("^") or route.endswith("$"):             warnings.append(


Question: Add a system check for unmatched URL paths' < / Description Currently, unmatched angle brackets are silently ignored: path('<int:question_id/vote/', , name='vote'), This leads to a frustrating debugging experience, where a 404 or NoReverseMatch could occur and only the eagle-eyed could spot why. Similar to the ^/$ system check in , I propose a new check in () for unmatched < or . It's rather unlikely those characters are legitimately used in URLs. (Inspired by a new user making this mistake and asking for help on the forum.)

)         return warnings <ins>    def _check_pattern_unmatched_angle_brackets(self):        warnings = []        segments = self._route.split("/")        for segment in segments:            open_bracket_counter = 0            unmatched_angle_bracket = None            for char in segment:                if char == "<":                    open_bracket_counter += 1                elif char == ">":                    open_bracket_counter -= 1                    if open_bracket_counter < 0:                        unmatched_angle_bracket = ">"                        break            else:                if open_bracket_counter > 0:                    unmatched_angle_bracket = "<"            if unmatched_angle_bracket is not None:                warnings.append(                    Warning(                        "Your URL pattern %s has an unmatched '%s' bracket."                        % (self.describe(), unmatched_angle_bracket),                        id="urls.W010",                    )                )        return warnings</ins>     def _compile(self, route):         return re.compile(_route_to_regex(route, self._is_endpoint)[0])


Question: Deleting objects after searching related many to many field crashes the admin page Description Minimal reproduction: class Post(models.Model): title = models.String(...) authors = models.ManyToMany("User", ...) class User(models.Model): email = models.String(...) class PostAdmin(admin.ModelAdmin): searchfields = ("title", "authorsemail") then opening the admin site, opening the post page that contains only one post (any title and author assigned) and entering a search term (e.g the first 2 characters of the title), selecting the post and then using the delete action results in an Internal Sever Error 500 with an error/stack-trace: Internal Server Error: /admin/post/post/ Traceback (most recent call last): File "", line 47, in inner response = getresponse(request) File "", line 181, in getresponse response = wrappedcallback(request, callbackargs, callbackkwargs) File "", line 616, in wrapper return self.adminsite.adminview(view)(args, kwargs) File "", line 130, in wrappedview response = viewfunc(request, args, kwargs) File "", line 44, in wrappedviewfunc response = viewfunc(request, args, kwargs) File "", line 241, in inner return view(request, args, kwargs) File "", line 43, in wrapper return boundmethod(args, kwargs) File "", line 130, in wrappedview response = viewfunc(request, args, *kwargs) File "", line 1737, in changelistview response = self.responseaction(request, queryset=cl.getqueryset(request)) File "", line 1406, in responseaction response = func(self, request, queryset) File "", line 45, in deleteselected modeladmin.deletequeryset(request, queryset) File "", line 1107, in deletequeryset () File "", line 728, in delete raise TypeError('Cannot call delete() after .distinct().') TypeError: Cannot call delete() after .distinct(). "POST /admin/post/post/?q=my HTTP/1.1" 500 I can confirm that pip install django==3.1.8 fixes the error, and after having a look at the diff between stable/3.2.x and 3.1.8, I suspect the "regression" comes about from the work done on preserving the filters on delete or something along those lines - I haven't done a thorough investigation yet. Presumably .distinct() is being called because of the search involving the many to many field. I am using a Postgres database.

SuspiciousOperation, ) from django.core.paginator import InvalidPagefrom django.db.models import Exists, F, Field, ManyToOneRel, OrderBy, OuterReffrom django.db.models import F, Field, ManyToOneRel, OrderBy from django.db.models.expressions import Combinable from django.urls import reverse from django.utils.deprecation import RemovedInDjango60Warning


Question: Deleting objects after searching related many to many field crashes the admin page Description Minimal reproduction: class Post(models.Model): title = models.String(...) authors = models.ManyToMany("User", ...) class User(models.Model): email = models.String(...) class PostAdmin(admin.ModelAdmin): searchfields = ("title", "authorsemail") then opening the admin site, opening the post page that contains only one post (any title and author assigned) and entering a search term (e.g the first 2 characters of the title), selecting the post and then using the delete action results in an Internal Sever Error 500 with an error/stack-trace: Internal Server Error: /admin/post/post/ Traceback (most recent call last): File "", line 47, in inner response = getresponse(request) File "", line 181, in getresponse response = wrappedcallback(request, callbackargs, callbackkwargs) File "", line 616, in wrapper return self.adminsite.adminview(view)(args, kwargs) File "", line 130, in wrappedview response = viewfunc(request, args, kwargs) File "", line 44, in wrappedviewfunc response = viewfunc(request, args, kwargs) File "", line 241, in inner return view(request, args, kwargs) File "", line 43, in wrapper return boundmethod(args, kwargs) File "", line 130, in wrappedview response = viewfunc(request, args, *kwargs) File "", line 1737, in changelistview response = self.responseaction(request, queryset=cl.getqueryset(request)) File "", line 1406, in responseaction response = func(self, request, queryset) File "", line 45, in deleteselected modeladmin.deletequeryset(request, queryset) File "", line 1107, in deletequeryset () File "", line 728, in delete raise TypeError('Cannot call delete() after .distinct().') TypeError: Cannot call delete() after .distinct(). "POST /admin/post/post/?q=my HTTP/1.1" 500 I can confirm that pip install django==3.1.8 fixes the error, and after having a look at the diff between stable/3.2.x and 3.1.8, I suspect the "regression" comes about from the work done on preserving the filters on delete or something along those lines - I haven't done a thorough investigation yet. Presumably .distinct() is being called because of the search involving the many to many field. I am using a Postgres database.

# ValueError, ValidationError, or ?.             raise IncorrectLookupParameters(e) <ins>        if not qs.query.select_related:            qs = self.apply_select_related(qs)        # Set ordering.        ordering = self.get_ordering(request, qs)        qs = qs.order_by(*ordering)</ins>         # Apply search results         qs, search_may_have_duplicates = self.model_admin.get_search_results(             request,


Question: Deleting objects after searching related many to many field crashes the admin page Description Minimal reproduction: class Post(models.Model): title = models.String(...) authors = models.ManyToMany("User", ...) class User(models.Model): email = models.String(...) class PostAdmin(admin.ModelAdmin): searchfields = ("title", "authorsemail") then opening the admin site, opening the post page that contains only one post (any title and author assigned) and entering a search term (e.g the first 2 characters of the title), selecting the post and then using the delete action results in an Internal Sever Error 500 with an error/stack-trace: Internal Server Error: /admin/post/post/ Traceback (most recent call last): File "", line 47, in inner response = getresponse(request) File "", line 181, in getresponse response = wrappedcallback(request, callbackargs, callbackkwargs) File "", line 616, in wrapper return self.adminsite.adminview(view)(args, kwargs) File "", line 130, in wrappedview response = viewfunc(request, args, kwargs) File "", line 44, in wrappedviewfunc response = viewfunc(request, args, kwargs) File "", line 241, in inner return view(request, args, kwargs) File "", line 43, in wrapper return boundmethod(args, kwargs) File "", line 130, in wrappedview response = viewfunc(request, args, *kwargs) File "", line 1737, in changelistview response = self.responseaction(request, queryset=cl.getqueryset(request)) File "", line 1406, in responseaction response = func(self, request, queryset) File "", line 45, in deleteselected modeladmin.deletequeryset(request, queryset) File "", line 1107, in deletequeryset () File "", line 728, in delete raise TypeError('Cannot call delete() after .distinct().') TypeError: Cannot call delete() after .distinct(). "POST /admin/post/post/?q=my HTTP/1.1" 500 I can confirm that pip install django==3.1.8 fixes the error, and after having a look at the diff between stable/3.2.x and 3.1.8, I suspect the "regression" comes about from the work done on preserving the filters on delete or something along those lines - I haven't done a thorough investigation yet. Presumably .distinct() is being called because of the search involving the many to many field. I am using a Postgres database.

)         # Remove duplicates from results, if necessary         if filters_may_have_duplicates | search_may_have_duplicates:            qs = qs.filter(pk=OuterRef("pk"))            qs = self.root_queryset.filter(Exists(qs))        # Set ordering.        ordering = self.get_ordering(request, qs)        qs = qs.order_by(*ordering)        if not qs.query.select_related:            qs = self.apply_select_related(qs)        return qs            return qs.distinct()        else:            return qs      def apply_select_related(self, qs):         if self.list_select_related is True:diff --git a/django/db/models/query.py b/django/db/models/query.py-- a/django/db/models/query.py++ b/django/db/models/query.py


Question: Deleting objects after searching related many to many field crashes the admin page Description Minimal reproduction: class Post(models.Model): title = models.String(...) authors = models.ManyToMany("User", ...) class User(models.Model): email = models.String(...) class PostAdmin(admin.ModelAdmin): searchfields = ("title", "authorsemail") then opening the admin site, opening the post page that contains only one post (any title and author assigned) and entering a search term (e.g the first 2 characters of the title), selecting the post and then using the delete action results in an Internal Sever Error 500 with an error/stack-trace: Internal Server Error: /admin/post/post/ Traceback (most recent call last): File "", line 47, in inner response = getresponse(request) File "", line 181, in getresponse response = wrappedcallback(request, callbackargs, callbackkwargs) File "", line 616, in wrapper return self.adminsite.adminview(view)(args, kwargs) File "", line 130, in wrappedview response = viewfunc(request, args, kwargs) File "", line 44, in wrappedviewfunc response = viewfunc(request, args, kwargs) File "", line 241, in inner return view(request, args, kwargs) File "", line 43, in wrapper return boundmethod(args, kwargs) File "", line 130, in wrappedview response = viewfunc(request, args, *kwargs) File "", line 1737, in changelistview response = self.responseaction(request, queryset=cl.getqueryset(request)) File "", line 1406, in responseaction response = func(self, request, queryset) File "", line 45, in deleteselected modeladmin.deletequeryset(request, queryset) File "", line 1107, in deletequeryset () File "", line 728, in delete raise TypeError('Cannot call delete() after .distinct().') TypeError: Cannot call delete() after .distinct(). "POST /admin/post/post/?q=my HTTP/1.1" 500 I can confirm that pip install django==3.1.8 fixes the error, and after having a look at the diff between stable/3.2.x and 3.1.8, I suspect the "regression" comes about from the work done on preserving the filters on delete or something along those lines - I haven't done a thorough investigation yet. Presumably .distinct() is being called because of the search involving the many to many field. I am using a Postgres database.

self._not_support_combined_queries("delete")         if self.query.is_sliced:             raise TypeError("Cannot use 'limit' or 'offset' with delete().")        if self.query.distinct or self.query.distinct_fields:            raise TypeError("Cannot call delete() after .distinct().")        if self.query.distinct_fields:            raise TypeError("Cannot call delete() after .distinct(*fields).")         if self._fields is not None:             raise TypeError("Cannot call delete() after .values() or .values_list()")


Question: Allow returning IDs in QuerySet.bulkcreate() when updating conflicts. Description Currently, when using bulkcreate with a conflict handling flag turned on (e.g. ignoreconflicts or updateconflicts), the primary keys are not set in the returned queryset, as documented in bulkcreate. While I understand using ignoreconflicts can lead to PostgreSQL not returning the IDs when a row is ignored (see this SO thread), I don't understand why we don't return the IDs in the case of updateconflicts. For instance: MyModel.objects.bulkcreate([MyModel(...)], updateconflicts=True, updatefields=[...], uniquefields=[...]) generates a query without a RETURNING part: INSERT INTO "mymodel" (...) VALUES (...) ON CONFLICT(...) DO UPDATE ... If I append the RETURNING clause, the query is indeed valid and the ID is returned (checked with PostgreSQL). I investigated a bit and this in Django source is where the returningfields gets removed. I believe we could discriminate the cases differently so as to keep those returningfields in the case of updateconflicts. This would be highly helpful when using bulkcreate as a bulk upsert feature.

inserted_rows = []         bulk_return = connection.features.can_return_rows_from_bulk_insert         for item in [objs[i : i + batch_size] for i in range(0, len(objs), batch_size)]:            if bulk_return and on_conflict is None:            if bulk_return and (                on_conflict is None or on_conflict == OnConflict.UPDATE            ):                 inserted_rows.extend(                     self._insert(                         item,                         fields=fields,                         using=self.db,                        on_conflict=on_conflict,                        update_fields=update_fields,                        unique_fields=unique_fields,                         returning_fields=self.model._meta.db_returning_fields,                     )                 )


Question: BoundField.aswidget() ignores aria-describedby in attrs argument Description (last modified by Sage Abdullah) BoundField.aswidget() ignores aria-describedby that is passed in the attrs argument. Use case: In Wagtail, we have our own mechanism for rendering form fields (called "Panels") that is built on top of Django's Forms API. We use BoundField.aswidget() in our rendering process to render only the form field's widget, while its help text element is rendered separately via our Panels API with a custom markup (including HTML id) that conforms to our design system. We've been passing additional attributes to BoundField.aswidget() to improve accessibility, including aria-describedby, to establish the relationship between the field widget rendered by Django and our help text element. As of , Django automatically adds aria-describedby to form fields with a helptext. The logic in buildwidgetattrs() (used by aswidget()) checks for an existing aria-describedby in the field's widget's attrs before automatically generating one. However, it does not take into account the possibility of an existing aria-describedby in the attrs argument. A workaround on Wagtail's side could be to modify the widget's attrs before using aswidget(), but this is not ideal as the widget is customisable by the developer and we would have to copy the widget instance to avoid modifying the existing widget instance (which might be shared across form fields). I believe Django should check for aria-describedby in attrs first, before falling back to and the auto-generated value. Test case: class BoundFieldTests(SimpleTestCase): def testaswidgetwithcustomariadescribedby(self): class TestForm(Form): data = CharField(helptext="Some help text") form = TestForm({"data": "some value"}) self.assertHTMLEqual( form["data"].aswidget(attrs={"aria-describedby": "customhelptextid"}), """ <input type="text" name="data" value="some value" aria-describedby="customhelptextid" required id="iddata"""", ) Patch: diff --git index .. --- +++ -290,10 +290,11 class BoundField(RenderableFieldMixin): # If a custom aria-describedby attribute is given and helptext is # used, the custom aria-described by is preserved so user can set the # desired order. if customariadescribedbyid := ("aria-describedby"): attrs["aria-describedby"] = customariadescribedbyid elif and self.idforlabel: attrs["aria-describedby"] = f"{self.idforlabel}helptext" if not ("aria-describedby"): if customariadescribedbyid := ("aria-describedby"): attrs["aria-describedby"] = customariadescribedbyid elif and self.idforlabel: attrs["aria-describedby"] = f"{self.idforlabel}_helptext" return attrs Happy to submit a PR if this is accepted.

attrs["required"] = True         if self.field.disabled:             attrs["disabled"] = True        # If a custom aria-describedby attribute is given and help_text is        # used, the custom aria-described by is preserved so user can set the        # desired order.        if custom_aria_described_by_id := widget.attrs.get("aria-describedby"):            attrs["aria-describedby"] = custom_aria_described_by_id        elif self.field.help_text and self.id_for_label:        # If a custom aria-describedby attribute is given (either via the attrs        # argument or widget.attrs) and help_text is used, the custom        # aria-described by is preserved so user can set the desired order.        if (            not attrs.get("aria-describedby")            and not widget.attrs.get("aria-describedby")            and self.field.help_text            and self.id_for_label        ):             attrs["aria-describedby"] = f"{self.id_for_label}_helptext"         return attrs


Question: Migration serializer for sets results in non-deterministic order. Description We are using quite a lot of complex indextogether / uniquetogether constraints on our models, and the output in the generated migrations is flip-flopping all the time like follows, causing spurious diffs in our checkouts: migrations.AlterUniqueTogether( uniquetogether={("tenant", "dealer"), ("tenant", "order")}, uniquetogether={("tenant", "order"), ("tenant", "dealer")}, This is happening because these constraints are normalized to sets internally in the ModelState, which kind of makes sense, but unfortunately set iteration order (unlike dicts!) is unstable in Python 3 due to hash randomization. However, migrations serializer doesn't have any special facilities for ensuring stable output for sets and this is what causes annoying diffs for us all the time. I suggest to add a trivial serializer specifically for unordered sequences which ensures stable output no matter the iteration order. Stability can be achieved by sorting elements in the set by their string representation. This only affects the writer output, and doesn't interfere with the rest of Django in any way, so this change only improves developer experience, but has no effect on the performance and/or reliability. I hope that even though it's apparently not a major problem for most users you would still accept the fix to ensure stable migration writer output for the rest of us.

return value % (", ".join(strings)), imports  <ins>class BaseUnorderedSequenceSerializer(BaseSequenceSerializer):    def __init__(self, value):        super().__init__(sorted(value, key=repr))</ins> class BaseSimpleSerializer(BaseSerializer):     def serialize(self):         return repr(self.value), set()


Question: Migration serializer for sets results in non-deterministic order. Description We are using quite a lot of complex indextogether / uniquetogether constraints on our models, and the output in the generated migrations is flip-flopping all the time like follows, causing spurious diffs in our checkouts: migrations.AlterUniqueTogether( uniquetogether={("tenant", "dealer"), ("tenant", "order")}, uniquetogether={("tenant", "order"), ("tenant", "dealer")}, This is happening because these constraints are normalized to sets internally in the ModelState, which kind of makes sense, but unfortunately set iteration order (unlike dicts!) is unstable in Python 3 due to hash randomization. However, migrations serializer doesn't have any special facilities for ensuring stable output for sets and this is what causes annoying diffs for us all the time. I suggest to add a trivial serializer specifically for unordered sequences which ensures stable output no matter the iteration order. Stability can be achieved by sorting elements in the set by their string representation. This only affects the writer output, and doesn't interfere with the rest of Django in any way, so this change only improves developer experience, but has no effect on the performance and/or reliability. I hope that even though it's apparently not a major problem for most users you would still accept the fix to ensure stable migration writer output for the rest of us.

return super().serialize()  class FrozensetSerializer(BaseSequenceSerializer):class FrozensetSerializer(BaseUnorderedSequenceSerializer):     def _format(self):         return "frozenset([%s])"


Question: Migration serializer for sets results in non-deterministic order. Description We are using quite a lot of complex indextogether / uniquetogether constraints on our models, and the output in the generated migrations is flip-flopping all the time like follows, causing spurious diffs in our checkouts: migrations.AlterUniqueTogether( uniquetogether={("tenant", "dealer"), ("tenant", "order")}, uniquetogether={("tenant", "order"), ("tenant", "dealer")}, This is happening because these constraints are normalized to sets internally in the ModelState, which kind of makes sense, but unfortunately set iteration order (unlike dicts!) is unstable in Python 3 due to hash randomization. However, migrations serializer doesn't have any special facilities for ensuring stable output for sets and this is what causes annoying diffs for us all the time. I suggest to add a trivial serializer specifically for unordered sequences which ensures stable output no matter the iteration order. Stability can be achieved by sorting elements in the set by their string representation. This only affects the writer output, and doesn't interfere with the rest of Django in any way, so this change only improves developer experience, but has no effect on the performance and/or reliability. I hope that even though it's apparently not a major problem for most users you would still accept the fix to ensure stable migration writer output for the rest of us.

return "[%s]"  class SetSerializer(BaseSequenceSerializer):class SetSerializer(BaseUnorderedSequenceSerializer):     def _format(self):         # Serialize as a set literal except when value is empty because {}         # is an empty dict.diff --git a/django/db/migrations/writer.py b/django/db/migrations/writer.py-- a/django/db/migrations/writer.py++ b/django/db/migrations/writer.py


Question: Migration serializer for sets results in non-deterministic order. Description We are using quite a lot of complex indextogether / uniquetogether constraints on our models, and the output in the generated migrations is flip-flopping all the time like follows, causing spurious diffs in our checkouts: migrations.AlterUniqueTogether( uniquetogether={("tenant", "dealer"), ("tenant", "order")}, uniquetogether={("tenant", "order"), ("tenant", "dealer")}, This is happening because these constraints are normalized to sets internally in the ModelState, which kind of makes sense, but unfortunately set iteration order (unlike dicts!) is unstable in Python 3 due to hash randomization. However, migrations serializer doesn't have any special facilities for ensuring stable output for sets and this is what causes annoying diffs for us all the time. I suggest to add a trivial serializer specifically for unordered sequences which ensures stable output no matter the iteration order. Stability can be achieved by sorting elements in the set by their string representation. This only affects the writer output, and doesn't interfere with the rest of Django in any way, so this change only improves developer experience, but has no effect on the performance and/or reliability. I hope that even though it's apparently not a major problem for most users you would still accept the fix to ensure stable migration writer output for the rest of us.

imports.add("from django.conf import settings")             else:                 dependencies.append("        %s," % self.serialize(dependency)[0])        items["dependencies"] = "\n".join(dependencies) + "\n" if dependencies else ""        items["dependencies"] = (            "\n".join(sorted(dependencies)) + "\n" if dependencies else ""        )          # Format imports nicely, swapping imports of functions from migration files         # for comments


Question: Cannot use aggregate over window functions since 4.2 Description (last modified by younes-chaoui) After upgrading to Django 4.2, I encountered an exception when executing ORM queries that involve aggregates over Window functions. The specific error was aggregate function calls cannot contain window function calls Dependencies : psycopg2 version: 2.9.3 django version: 4.2.3 PostgreSQL version: 13.4 Example Code: queryset = queryset.annotate( cumulDJR=Coalesce(Window(Sum("DJR"), orderby=F("date").asc()), 0.0) ) aggregate = queryset.aggregate( DJRtotal=Sum("DJR"), cumulDJRtotal=Sum("cumulDJR") )

# Store annotation mask prior to temporarily adding aggregations for         # resolving purpose to facilitate their subsequent removal.         refs_subquery = False<ins>        refs_window = False</ins>         replacements = {}         annotation_select_mask = self.annotation_select_mask         for alias, aggregate_expr in aggregate_exprs.items():


Question: Cannot use aggregate over window functions since 4.2 Description (last modified by younes-chaoui) After upgrading to Django 4.2, I encountered an exception when executing ORM queries that involve aggregates over Window functions. The specific error was aggregate function calls cannot contain window function calls Dependencies : psycopg2 version: 2.9.3 django version: 4.2.3 PostgreSQL version: 13.4 Example Code: queryset = queryset.annotate( cumulDJR=Coalesce(Window(Sum("DJR"), orderby=F("date").asc()), 0.0) ) aggregate = queryset.aggregate( DJRtotal=Sum("DJR"), cumulDJRtotal=Sum("cumulDJR") )

getattr(self.annotations[ref], "subquery", False)                 for ref in aggregate.get_refs()             )<ins>            refs_window |= any(                getattr(self.annotations[ref], "contains_over_clause", True)                for ref in aggregate.get_refs()            )</ins>             aggregate = aggregate.replace_expressions(replacements)             self.annotations[alias] = aggregate             replacements[Ref(alias, aggregate)] = aggregate


Question: Cannot use aggregate over window functions since 4.2 Description (last modified by younes-chaoui) After upgrading to Django 4.2, I encountered an exception when executing ORM queries that involve aggregates over Window functions. The specific error was aggregate function calls cannot contain window function calls Dependencies : psycopg2 version: 2.9.3 django version: 4.2.3 PostgreSQL version: 13.4 Example Code: queryset = queryset.annotate( cumulDJR=Coalesce(Window(Sum("DJR"), orderby=F("date").asc()), 0.0) ) aggregate = queryset.aggregate( DJRtotal=Sum("DJR"), cumulDJRtotal=Sum("cumulDJR") )

or self.is_sliced             or has_existing_aggregation             or refs_subquery<ins>            or refs_window</ins>             or qualify             or self.distinct             or self.combinator


Question: Class methods from nested classes cannot be used as Field.default. Description (last modified by Mariusz Felisiak) Given the following model: class Profile(models.Model): class Capability(models.TextChoices): BASIC = ("BASIC", "Basic") PROFESSIONAL = ("PROFESSIONAL", "Professional") def default(cls) -list[str]: return [cls.BASIC] capabilities = ArrayField( models.CharField(choices=Capability.choices, maxlength=30, blank=True), null=True, default=Capability.default ) The resulting migration contained the following: # ... migrations.AddField( modelname='profile', name='capabilities', (basefield=models.CharField(blank=True, choices=[('BASIC', 'Basic'), ('PROFESSIONAL', 'Professional')], maxlength=30), , null=True, size=None), ), # ... As you can see, migrations.AddField is passed as argument "default" a wrong value "", which leads to an error when trying to migrate. The right value should be "".

):             klass = self.value.__self__             module = klass.__module__            return "%s.%s.%s" % (module, klass.__name__, self.value.__name__), {            return "%s.%s.%s" % (module, klass.__qualname__, self.value.__name__), {                 "import %s" % module             }         # Further error checking


Question: Unknown django-admin command raises "settings not configured" when no project exists Description When you try to execute a command that does not exist outside a Django project (e.g. createproject instead of startproject...), Django raise a ImproperlyConfigured that does not fit with the real error: (testdjango):~/testdjango$ django- not-a-real-command test Traceback (most recent call last): File "/root/testdjango/bin/django-", line 5, in <modulemanagement.executefromcommandline() File "/root/testdjango/local/lib/python2.7/site-", line 338, in executefromcommandline utility.execute() File "/root/testdjango/local/lib/python2.7/site-", line 330, in execute self.fetchcommand(subcommand).runfromargv() File "/root/testdjango/local/lib/python2.7/site-", line 182, in fetchcommand settings.INSTALLEDAPPS File "/root/testdjango/local/lib/python2.7/site-", line 48, in getattr (name) File "/root/testdjango/local/lib/python2.7/site-", line 42, in setup % (desc, ENVIRONMENTVARIABLE)) Requested setting INSTALLEDAPPS, but settings are not configured. You must either define the environment variable DJANGOSETTINGS_MODULE or call settings.configure() before accessing settings. The expected output is: (testdjango):~/testdjango$ django- not-a-real-command test Unknown command: 'not-a-real-command' Type 'django- help' for usage.

try:             app_name = commands[subcommand]         except KeyError:            # This might trigger ImproperlyConfigured (masked in get_commands)            settings.INSTALLED_APPS            if os.environ.get('DJANGO_SETTINGS_MODULE'):                # If `subcommand` is missing due to misconfigured settings, the                # following line will retrigger an ImproperlyConfigured exception                # (get_commands() swallows the original one) so the user is                # informed about it.                settings.INSTALLED_APPS            else:                sys.stderr.write("No Django settings specified.\n")             sys.stderr.write("Unknown command: %r\nType '%s help' for usage.\n" %                 (subcommand, self.prog_name))             sys.exit(1)


Question: Set script prefix in () to allow its usage outside of requests Description The script prefix for doesn't get set to anything when being called through , because of course it doesn't know what that value should be. This is a problem if you're rendering views (or otherwise reversing urls) from a command (as one of my sites does to send emails). This is solvable by calling setscriptprefix from , but that feels kind of dirty since it's then about to be rewritten in the WSGI handler. I don't know what a good solution to this would be. Perhaps it would be nice to be able to set a global default script path somewhere that would then get incorporated into the default values of things like LOGIN_URL. Maybe just a note in the documentation would be good. It took me a while to figure out, because I haven't been able to find anything else about this online. (I guess that non-/ script paths are uncommon and reversing urls from is also uncommon, so both together are very uncommon.)

<ins>from __future__ import unicode_literals</ins> from django.utils.version import get_version  VERSION = (1, 10, 0, 'alpha', 0)


Question: Set script prefix in () to allow its usage outside of requests Description The script prefix for doesn't get set to anything when being called through , because of course it doesn't know what that value should be. This is a problem if you're rendering views (or otherwise reversing urls) from a command (as one of my sites does to send emails). This is solvable by calling setscriptprefix from , but that feels kind of dirty since it's then about to be rewritten in the WSGI handler. I don't know what a good solution to this would be. Perhaps it would be nice to be able to set a global default script path somewhere that would then get incorporated into the default values of things like LOGIN_URL. Maybe just a note in the documentation would be good. It took me a while to figure out, because I haven't been able to find anything else about this online. (I guess that non-/ script paths are uncommon and reversing urls from is also uncommon, so both together are very uncommon.)

__version__ = get_version(VERSION)  def setup():def setup(set_prefix=True):     """     Configure the settings (this happens as a side effect of accessing the     first setting), configure logging and populate the app registry.    Set the thread-local urlresolvers script prefix if `set_prefix` is True.     """     from django.apps import apps     from django.conf import settings    from django.core.urlresolvers import set_script_prefix    from django.utils.encoding import force_text     from django.utils.log import configure_logging      configure_logging(settings.LOGGING_CONFIG, settings.LOGGING)    if set_prefix:        set_script_prefix(            '/' if settings.FORCE_SCRIPT_NAME is None else force_text(settings.FORCE_SCRIPT_NAME)        )     apps.populate(settings.INSTALLED_APPS)diff --git a/django/core/wsgi.py b/django/core/wsgi.py-- a/django/core/wsgi.py++ b/django/core/wsgi.py


Question: Set script prefix in () to allow its usage outside of requests Description The script prefix for doesn't get set to anything when being called through , because of course it doesn't know what that value should be. This is a problem if you're rendering views (or otherwise reversing urls) from a command (as one of my sites does to send emails). This is solvable by calling setscriptprefix from , but that feels kind of dirty since it's then about to be rewritten in the WSGI handler. I don't know what a good solution to this would be. Perhaps it would be nice to be able to set a global default script path somewhere that would then get incorporated into the default values of things like LOGIN_URL. Maybe just a note in the documentation would be good. It took me a while to figure out, because I haven't been able to find anything else about this online. (I guess that non-/ script paths are uncommon and reversing urls from is also uncommon, so both together are very uncommon.)

Allows us to avoid making django.core.handlers.WSGIHandler public API, in     case the internal WSGI implementation changes or moves in the future.     """    django.setup()    django.setup(set_prefix=False)     return WSGIHandler()


Question: Allow auth machinery to be used without installing auth app Description Django projects should be able to use the auth machinery (like and ) without having in INSTALLEDAPPS See !topic/django-developers/2DzLBbk8w-w This ticket is for resolving the current issue in 1.9. I also want to write a test or two to prevent this issue from happening in the future.

from .management import create_permissions  class AuthConfig(AppConfig):class BaseAuthConfig(AppConfig):    """    AppConfig which assumes that the auth models don't exist.    """     name = 'django.contrib.auth'     verbose_name = _("Authentication and Authorization")      def ready(self):        checks.register(check_user_model, checks.Tags.models)class AuthConfig(BaseAuthConfig):    """    The default AppConfig for auth.    """    def ready(self):        super(AuthConfig, self).ready()         post_migrate.connect(             create_permissions,             dispatch_uid="django.contrib.auth.management.create_permissions"         )        checks.register(check_user_model, checks.Tags.models)         checks.register(check_models_permissions, checks.Tags.models)