Question: Unnecessary joins in admin changelist query Description Django 1.2.5 Models: class Client(models.Model): name = models.CharField(('name'), maxlength=256) name2 = models.CharField(('unofficial or obsolete name'), maxlength=256, blank=True, null=True) contactperson = models.CharField(('contact person'), maxlength=256, blank=True, null=True) ... class ClientOffice(models.Model): name = models.CharField(('name'), maxlength=256) name2 = models.CharField(('unofficial or obsolete name'), maxlength=256, blank=True, null=True) ... client = models.ForeignKey(Client, verbosename=('client')) ... and admin options like these: class ClientAdmin(admin.ModelAdmin): searchfields = ('name', 'name2', 'contact_person', 'clientofficename', 'clientofficename2') ... Numbers: () () Now, if we try searching for clients in admin by a search query containig several words (3), got django/admin stalled. The problem is going to be that each word in the search query leads to additional JOIN in final SQL query beacause of qs = (...) pattern. The attached patch is for Django 1.2.5, but adopting for the current SVN trunk is trivial.

if search_fields and search_term:             orm_lookups = [construct_search(str(search_field))                            for search_field in search_fields]<ins>            term_queries = []</ins>             for bit in smart_split(search_term):                 if bit.startswith(('"', "'")) and bit[0] == bit[-1]:                     bit = unescape_string_literal(bit)


Question: Unnecessary joins in admin changelist query Description Django 1.2.5 Models: class Client(models.Model): name = models.CharField(('name'), maxlength=256) name2 = models.CharField(('unofficial or obsolete name'), maxlength=256, blank=True, null=True) contactperson = models.CharField(('contact person'), maxlength=256, blank=True, null=True) ... class ClientOffice(models.Model): name = models.CharField(('name'), maxlength=256) name2 = models.CharField(('unofficial or obsolete name'), maxlength=256, blank=True, null=True) ... client = models.ForeignKey(Client, verbosename=('client')) ... and admin options like these: class ClientAdmin(admin.ModelAdmin): searchfields = ('name', 'name2', 'contact_person', 'clientofficename', 'clientofficename2') ... Numbers: () () Now, if we try searching for clients in admin by a search query containig several words (3), got django/admin stalled. The problem is going to be that each word in the search query leads to additional JOIN in final SQL query beacause of qs = (...) pattern. The attached patch is for Django 1.2.5, but adopting for the current SVN trunk is trivial.

*((orm_lookup, bit) for orm_lookup in orm_lookups),                     _connector=models.Q.OR,                 )                queryset = queryset.filter(or_queries)                term_queries.append(or_queries)            queryset = queryset.filter(models.Q(*term_queries))             may_have_duplicates |= any(                 lookup_spawns_duplicates(self.opts, search_spec)                 for search_spec in orm_lookups


Question: Django admin allows filtering using the field lookups such as "in", but it is impossible to include a value that contains a comma Description The admin site allows you to filter the queryset in the changelist in a plenty of different ways. Notably, it allows you to filter the records by multiple values (if the field's value is one of the specified value options, then such record is considered matching). For example, you can test it with a query string like this: /admin/auth/user/?username_in=johnny,viola,gordon Unfortunately, there is a big limitation at the moment: you can't include a value option that contains a comma (or a few). The function that splits the string is preparelookup_value, found in

class FieldListFilter(ListFilter):     _field_list_filters = []     _take_priority_index = 0<ins>    list_separator = ','</ins>      def __init__(self, field, request, params, model, model_admin, field_path):         self.field = field


Question: Django admin allows filtering using the field lookups such as "in", but it is impossible to include a value that contains a comma Description The admin site allows you to filter the queryset in the changelist in a plenty of different ways. Notably, it allows you to filter the records by multiple values (if the field's value is one of the specified value options, then such record is considered matching). For example, you can test it with a query string like this: /admin/auth/user/?username_in=johnny,viola,gordon Unfortunately, there is a big limitation at the moment: you can't include a value option that contains a comma (or a few). The function that splits the string is preparelookup_value, found in

for p in self.expected_parameters():             if p in params:                 value = params.pop(p)                self.used_parameters[p] = prepare_lookup_value(p, value)                self.used_parameters[p] = prepare_lookup_value(p, value, self.list_separator)      def has_output(self):         return Truediff --git a/django/contrib/admin/utils.py b/django/contrib/admin/utils.py-- a/django/contrib/admin/utils.py++ b/django/contrib/admin/utils.py


Question: Django admin allows filtering using the field lookups such as "in", but it is impossible to include a value that contains a comma Description The admin site allows you to filter the queryset in the changelist in a plenty of different ways. Notably, it allows you to filter the records by multiple values (if the field's value is one of the specified value options, then such record is considered matching). For example, you can test it with a query string like this: /admin/auth/user/?username_in=johnny,viola,gordon Unfortunately, there is a big limitation at the moment: you can't include a value option that contains a comma (or a few). The function that splits the string is preparelookup_value, found in

return False  def prepare_lookup_value(key, value):def prepare_lookup_value(key, value, separator=','):     """     Return a lookup value prepared to be used in queryset filtering.     """     # if key ends with __in, split parameter into separate values     if key.endswith('__in'):        value = value.split(',')        value = value.split(separator)     # if key ends with __isnull, special case '' and the string literals 'false' and '0'     elif key.endswith('__isnull'):         value = value.lower() not in ('', 'false', '0')


Question: Foreign key to a specific field is not handled in inspectdb Description (last modified by Tim Graham) if you have a DB like that CREATE TABLE foo ( id serial primary key, otherid int UNIQUE); CREATE TABLE bar ( id serial primary key, otherid int, constraint myconst FOREIGN KEY(otherid) references foo(otherid) ); the generated model for the bar table will have the otherid be a FK to foo and not foo(otherid). I'm attaching a potential fix for this. Sorry I had no time for the UTs.

extra_params['unique'] = True                      if is_relation:                        ref_db_column, ref_db_table = relations[column_name]                         if extra_params.pop('unique', False) or extra_params.get('primary_key'):                             rel_type = 'OneToOneField'                         else:                             rel_type = 'ForeignKey'                            ref_pk_column = connection.introspection.get_primary_key_column(cursor, ref_db_table)                            if ref_pk_column and ref_pk_column != ref_db_column:                                extra_params['to_field'] = ref_db_column                         rel_to = (                            "self" if relations[column_name][1] == table_name                            else table2model(relations[column_name][1])                            'self' if ref_db_table == table_name                            else table2model(ref_db_table)                         )                         if rel_to in known_models:                             field_type = '%s(%s' % (rel_type, rel_to)


Question: --squashed-name can overwrite existing migration files Description squashmigrations --squashed-name initial myapp 0002 will overwrite If that migration has already been applied anywhere, the code and the database will be in an inconsistent state. Furthermore, the 0002... migration likely depends on 0001initial, so the new replaces = …0002… line to introduces a CircularDependencyError. squashmigrations should exit with an error instead of overwriting existing migration files. --squashed-name can overwrite existing migration files Description squashmigrations --squashed-name initial myapp 0002 will overwrite If that migration has already been applied anywhere, the code and the database will be in an inconsistent state. Furthermore, the 0002... migration likely depends on 0001initial, so the new replaces = …0002… line to introduces a CircularDependencyError. squashmigrations should exit with an error instead of overwriting existing migration files.

<ins>import os</ins> from django.apps import apps from django.conf import settings from django.core.management.base import BaseCommand, CommandError


Question: --squashed-name can overwrite existing migration files Description squashmigrations --squashed-name initial myapp 0002 will overwrite If that migration has already been applied anywhere, the code and the database will be in an inconsistent state. Furthermore, the 0002... migration likely depends on 0001initial, so the new replaces = …0002… line to introduces a CircularDependencyError. squashmigrations should exit with an error instead of overwriting existing migration files. --squashed-name can overwrite existing migration files Description squashmigrations --squashed-name initial myapp 0002 will overwrite If that migration has already been applied anywhere, the code and the database will be in an inconsistent state. Furthermore, the 0002... migration likely depends on 0001initial, so the new replaces = …0002… line to introduces a CircularDependencyError. squashmigrations should exit with an error instead of overwriting existing migration files.

# Write out the new migration file         writer = MigrationWriter(new_migration, include_header)<ins>        if os.path.exists(writer.path):            raise CommandError(                f'Migration {new_migration.name} already exists. Use a different name.'            )</ins>         with open(writer.path, "w", encoding='utf-8') as fh:             fh.write(writer.as_string())


Question: CacheMiddleware and FetchFromCacheMiddleware are not thread safe. Description CacheMiddleware persist = caches[cachealias] on startup and it is not thread safe.  I found that after some production errors with pylibmc and uwsgi threaded. Created a small project to reproduce it. Nothing fancy, just pylibmc cache and a cached view. It fails even with development server, with concurrent requests. Traceback (most recent call last): File "versions/pylibmcbug/lib/python3.9/site-", line 47, in inner response = getresponse(request) File "versions/pylibmcbug/lib/python3.9/site-", line 181, in getresponse response = wrappedcallback(request, callbackargs, *callbackkwargs) File "versions/pylibmcbug/lib/python3.9/site-", line 122, in wrappedview result = middleware.processrequest(request) File "versions/pylibmcbug/lib/python3.9/site-", line 145, in processrequest cachekey = getcachekey(request, self.keyprefix, 'GET', ) File "versions/pylibmcbug/lib/python3.9/site-", line 362, in getcachekey headerlist = (cachekey) File "versions/pylibmcbug/lib/python3.9/site-", line 77, in get return (key, default) pylibmc.ConnectionError: error 3 from memcachedget(): (0x7f290400bd60) FAILURE, poll() returned a value that was not dealt with, host: localhost: - Looking for git history, it is this way since 2010.

self.page_timeout = None         self.key_prefix = settings.CACHE_MIDDLEWARE_KEY_PREFIX         self.cache_alias = settings.CACHE_MIDDLEWARE_ALIAS        self.cache = caches[self.cache_alias]    @property    def cache(self):        return caches[self.cache_alias]      def _should_update_cache(self, request, response):         return hasattr(request, '_cache_update_cache') and request._cache_update_cache


Question: CacheMiddleware and FetchFromCacheMiddleware are not thread safe. Description CacheMiddleware persist = caches[cachealias] on startup and it is not thread safe.  I found that after some production errors with pylibmc and uwsgi threaded. Created a small project to reproduce it. Nothing fancy, just pylibmc cache and a cached view. It fails even with development server, with concurrent requests. Traceback (most recent call last): File "versions/pylibmcbug/lib/python3.9/site-", line 47, in inner response = getresponse(request) File "versions/pylibmcbug/lib/python3.9/site-", line 181, in getresponse response = wrappedcallback(request, callbackargs, *callbackkwargs) File "versions/pylibmcbug/lib/python3.9/site-", line 122, in wrappedview result = middleware.processrequest(request) File "versions/pylibmcbug/lib/python3.9/site-", line 145, in processrequest cachekey = getcachekey(request, self.keyprefix, 'GET', ) File "versions/pylibmcbug/lib/python3.9/site-", line 362, in getcachekey headerlist = (cachekey) File "versions/pylibmcbug/lib/python3.9/site-", line 77, in get return (key, default) pylibmc.ConnectionError: error 3 from memcachedget(): (0x7f290400bd60) FAILURE, poll() returned a value that was not dealt with, host: localhost: - Looking for git history, it is this way since 2010.

super().__init__(get_response)         self.key_prefix = settings.CACHE_MIDDLEWARE_KEY_PREFIX         self.cache_alias = settings.CACHE_MIDDLEWARE_ALIAS        self.cache = caches[self.cache_alias]    @property    def cache(self):        return caches[self.cache_alias]      def process_request(self, request):         """


Question: CacheMiddleware and FetchFromCacheMiddleware are not thread safe. Description CacheMiddleware persist = caches[cachealias] on startup and it is not thread safe.  I found that after some production errors with pylibmc and uwsgi threaded. Created a small project to reproduce it. Nothing fancy, just pylibmc cache and a cached view. It fails even with development server, with concurrent requests. Traceback (most recent call last): File "versions/pylibmcbug/lib/python3.9/site-", line 47, in inner response = getresponse(request) File "versions/pylibmcbug/lib/python3.9/site-", line 181, in getresponse response = wrappedcallback(request, callbackargs, *callbackkwargs) File "versions/pylibmcbug/lib/python3.9/site-", line 122, in wrappedview result = middleware.processrequest(request) File "versions/pylibmcbug/lib/python3.9/site-", line 145, in processrequest cachekey = getcachekey(request, self.keyprefix, 'GET', ) File "versions/pylibmcbug/lib/python3.9/site-", line 362, in getcachekey headerlist = (cachekey) File "versions/pylibmcbug/lib/python3.9/site-", line 77, in get return (key, default) pylibmc.ConnectionError: error 3 from memcachedget(): (0x7f290400bd60) FAILURE, poll() returned a value that was not dealt with, host: localhost: - Looking for git history, it is this way since 2010.

if cache_alias is None:                 cache_alias = DEFAULT_CACHE_ALIAS             self.cache_alias = cache_alias<del>            self.cache = caches[self.cache_alias]</del>         except KeyError:             pass


Question: Aggregate filtered by an Exists subquery crashes Description For example: ("publisher").aggregate( maxrating=Max( "rating", filter=Exists( (book=OuterRef("pk")), ), ) Will crash with the following traceback: Traceback (most recent call last): File "", line 146, in testfilteredaggregatewithexists aggregate = ('publisher').aggregate( File "", line 405, in aggregate return query.getaggregation(, kwargs) File "", line 501, in getaggregation result = compiler.executesql(SINGLE) File "", line 1189, in executesql sql, params = () File "", line 531, in assql extraselect, orderby, groupby = self.presqlsetup() File "", line 59, in presqlsetup self.setupquery() File "", line 50, in setupquery , self.klassinfo, self.annotationcolmap = self.getselect() File "", line 267, in getselect sql, params = self.compile(col) File "", line 463, in compile sql, params = (self, self.connection) File "", line 90, in assql return sql, params + filterparams TypeError: can only concatenate list (not "tuple") to list The following patch should fix the issue: diff --git index .. --- +++ -87,7 +87,7 class Aggregate(Func): compiler, connection, template=template, filter=filtersql, extracontext ) return sql, params + filterparams return sql, (params, filterparams) else: copy = () = None

sql, sql_params = super().as_sql(compiler, connection, ordering=(                 'ORDER BY ' + ', '.join(ordering_expr_sql)             ))            return sql, sql_params + ordering_params            return sql, (*sql_params, *ordering_params)         return super().as_sql(compiler, connection, ordering='')      def set_source_expressions(self, exprs):diff --git a/django/db/models/aggregates.py b/django/db/models/aggregates.py-- a/django/db/models/aggregates.py++ b/django/db/models/aggregates.py


Question: Aggregate filtered by an Exists subquery crashes Description For example: ("publisher").aggregate( maxrating=Max( "rating", filter=Exists( (book=OuterRef("pk")), ), ) Will crash with the following traceback: Traceback (most recent call last): File "", line 146, in testfilteredaggregatewithexists aggregate = ('publisher').aggregate( File "", line 405, in aggregate return query.getaggregation(, kwargs) File "", line 501, in getaggregation result = compiler.executesql(SINGLE) File "", line 1189, in executesql sql, params = () File "", line 531, in assql extraselect, orderby, groupby = self.presqlsetup() File "", line 59, in presqlsetup self.setupquery() File "", line 50, in setupquery , self.klassinfo, self.annotationcolmap = self.getselect() File "", line 267, in getselect sql, params = self.compile(col) File "", line 463, in compile sql, params = (self, self.connection) File "", line 90, in assql return sql, params + filterparams TypeError: can only concatenate list (not "tuple") to list The following patch should fix the issue: diff --git index .. --- +++ -87,7 +87,7 class Aggregate(Func): compiler, connection, template=template, filter=filtersql, extracontext ) return sql, params + filterparams return sql, (params, filterparams) else: copy = () = None

compiler, connection, template=template, filter=filter_sql,                     **extra_context                 )                return sql, params + filter_params                return sql, (*params, *filter_params)             else:                 copy = self.copy()                 copy.filter = None


Question: Remove "for = ..." from MultiWidget's <label. Description The instance from Raw MultiWidget class generate idforlabel like f'{id}0' It has not sense. For example ChoiceWidget has self.addidindex and I can decide it myself, how I will see labelid - with or without index. I think, it is better to remove completely idforlabel method from MultiWidget Class.

return context      def id_for_label(self, id_):        if id_:            id_ += '_0'        return id_        return ''      def value_from_datadict(self, data, files, name):         return [


Question: Case() and ExpressionWrapper() doesn't work with DecimalField on SQLite. Description I noticed that, on sqlite, some comparisons against DecimalField annotations behave unexpectedly, in particular when wrapping a DecimalField value in a Case/When or ExpressionWrapper. I suspect that there might be some inconsistencies in the type conversions here somehow. I've created a few testcase to illustrate the problem on current git main:  and see below. def test00comparefield(self): """Comparing a Case annotation wrapping a field to a literal works.""" (a='', d=1) try: (dgt=0).get() finally: from import connection print(connection.queries[-1]['sql']) def test01compareannotationvalueliteral(self): """Comparing a literal annotation using Value to a literal works.""" # Fields are not actually used here (a='', d=0) try: Foo.objects.annotate( x=models.Value(1, (maxdigits=1, decimalplaces=0)), ).filter(xgt=0).get() finally: from import connection print(connection.queries[-1]['sql']) def test02compareannotationexpressionwrapperliteral(self): """Comparing a literal annotation using ExpressionWraper and Value to a literal works.""" # Fields are not actually used here (a='', d=0) try: Foo.objects.annotate( x=models.ExpressionWrapper( models.Value(1), (maxdigits=1, decimalplaces=0), ), ).filter(xgt=0).get() finally: from import connection print(connection.queries[-1]['sql']) def test03comparecaseannotation(self): """Comparing a Case annotation wrapping a field to a literal works.""" (a='', d=1) try: Foo.objects.annotate( x=models.Case(models.When(a='', then=models.F('d'))), ).filter(xgt=0).get() finally: from import connection print(connection.queries[-1]['sql']) test00comparefield compares a field directly with a literal, which works. test01compareannotationvalueliteral adds a literal annotation using just Value and then compares it, which also works. test02compareannotationexpressionwrapperliteral adds a literal annotation using Value wrapped in ExpressionWrapper, which does not work becomes a literal int, rather than a string like the compared value. test03comparecaseannotation wraps the field in a case/when and then compares it, which also does not work (maybe the CASE changes the type?) Running these testcases against sqlite gives: SELECT "modelfieldsfoo"."id", "modelfieldsfoo"."a", "modelfieldsfoo"."d" FROM "modelfieldsfoo" WHERE "modelfieldsfoo"."d" '0' LIMIT 21 .SELECT "modelfieldsfoo"."id", "modelfieldsfoo"."a", "modelfieldsfoo"."d", CAST('1' AS NUMERIC) AS "x" FROM "modelfieldsfoo" WHERE CAST('1' AS NUMERIC) '0' LIMIT 21 .SELECT "modelfieldsfoo"."id", "modelfieldsfoo"."a", "modelfieldsfoo"."d", 1 AS "x" FROM "modelfieldsfoo" WHERE 1 '0' LIMIT 21 ESELECT "modelfieldsfoo"."id", "modelfieldsfoo"."a", "modelfieldsfoo"."d", CASE WHEN "modelfieldsfoo"."a" = '' THEN "modelfieldsfoo"."d" ELSE NULL END AS "x" FROM "modelfieldsfoo" WHERE CASE WHEN ("modelfieldsfoo"."a" = '') THEN "modelfieldsfoo"."d" ELSE NULL END '0' LIMIT 21 E.s........... ERROR: test02compareannotationexpressionwrapperliteral (modelfields.testdecimalfield.DecimalFieldTests) Comparing a literal annotation using ExpressionWraper and Value to a literal works. Traceback (most recent call last): File "", line 437, in inner return func(args, kwargs) File "", line 142, in test02compareannotationexpressionwrapperliteral Foo.objects.annotate( File "", line 441, in get raise ( Foo matching query does not exist. ERROR: test03comparecaseannotation (modelfields.testdecimalfield.DecimalFieldTests) Comparing a Case annotation wrapping a field to a literal works. Traceback (most recent call last): File "", line 437, in inner return func(args, kwargs) File "", line 154, in test03comparecaseannotation Foo.objects.annotate( File "", line 441, in get raise ( Foo matching query does not exist. Note in the printed queries that the 0 value that is compared with is a string in the query ('0'), while the ExpressionWrappered value is just a plain number (1). The Value without ExpressionWrapper has a cast to NUMERIC that I suspect makes that case work? Also note that the DEBUG stuff and try-catch is only to capture the query that is being generated, it is not needed to trigger the problem. I tried printing the attribute first, but that seems to hide the quotes around the literal 0 in the query, so took me a while to realize what was happening. The numbers in the testcase names are to ensure the SQL queries are printed in a recognizable order. All four testcases work on MySQL, there the 0 value is written in the query without the quotes, and the cast to NUMERIC is also missing. I suspect this issue is highly similar to , and can a fix can probably build on the fix for that issue.

return self.as_sql(compiler, connection, **extra_context)  class ExpressionWrapper(Expression):class ExpressionWrapper(SQLiteNumericMixin, Expression):     """     An expression that can wrap another expression so that it can provide     extra context to the inner expression, such as the output_field.


Question: Case() and ExpressionWrapper() doesn't work with DecimalField on SQLite. Description I noticed that, on sqlite, some comparisons against DecimalField annotations behave unexpectedly, in particular when wrapping a DecimalField value in a Case/When or ExpressionWrapper. I suspect that there might be some inconsistencies in the type conversions here somehow. I've created a few testcase to illustrate the problem on current git main:  and see below. def test00comparefield(self): """Comparing a Case annotation wrapping a field to a literal works.""" (a='', d=1) try: (dgt=0).get() finally: from import connection print(connection.queries[-1]['sql']) def test01compareannotationvalueliteral(self): """Comparing a literal annotation using Value to a literal works.""" # Fields are not actually used here (a='', d=0) try: Foo.objects.annotate( x=models.Value(1, (maxdigits=1, decimalplaces=0)), ).filter(xgt=0).get() finally: from import connection print(connection.queries[-1]['sql']) def test02compareannotationexpressionwrapperliteral(self): """Comparing a literal annotation using ExpressionWraper and Value to a literal works.""" # Fields are not actually used here (a='', d=0) try: Foo.objects.annotate( x=models.ExpressionWrapper( models.Value(1), (maxdigits=1, decimalplaces=0), ), ).filter(xgt=0).get() finally: from import connection print(connection.queries[-1]['sql']) def test03comparecaseannotation(self): """Comparing a Case annotation wrapping a field to a literal works.""" (a='', d=1) try: Foo.objects.annotate( x=models.Case(models.When(a='', then=models.F('d'))), ).filter(xgt=0).get() finally: from import connection print(connection.queries[-1]['sql']) test00comparefield compares a field directly with a literal, which works. test01compareannotationvalueliteral adds a literal annotation using just Value and then compares it, which also works. test02compareannotationexpressionwrapperliteral adds a literal annotation using Value wrapped in ExpressionWrapper, which does not work becomes a literal int, rather than a string like the compared value. test03comparecaseannotation wraps the field in a case/when and then compares it, which also does not work (maybe the CASE changes the type?) Running these testcases against sqlite gives: SELECT "modelfieldsfoo"."id", "modelfieldsfoo"."a", "modelfieldsfoo"."d" FROM "modelfieldsfoo" WHERE "modelfieldsfoo"."d" '0' LIMIT 21 .SELECT "modelfieldsfoo"."id", "modelfieldsfoo"."a", "modelfieldsfoo"."d", CAST('1' AS NUMERIC) AS "x" FROM "modelfieldsfoo" WHERE CAST('1' AS NUMERIC) '0' LIMIT 21 .SELECT "modelfieldsfoo"."id", "modelfieldsfoo"."a", "modelfieldsfoo"."d", 1 AS "x" FROM "modelfieldsfoo" WHERE 1 '0' LIMIT 21 ESELECT "modelfieldsfoo"."id", "modelfieldsfoo"."a", "modelfieldsfoo"."d", CASE WHEN "modelfieldsfoo"."a" = '' THEN "modelfieldsfoo"."d" ELSE NULL END AS "x" FROM "modelfieldsfoo" WHERE CASE WHEN ("modelfieldsfoo"."a" = '') THEN "modelfieldsfoo"."d" ELSE NULL END '0' LIMIT 21 E.s........... ERROR: test02compareannotationexpressionwrapperliteral (modelfields.testdecimalfield.DecimalFieldTests) Comparing a literal annotation using ExpressionWraper and Value to a literal works. Traceback (most recent call last): File "", line 437, in inner return func(args, kwargs) File "", line 142, in test02compareannotationexpressionwrapperliteral Foo.objects.annotate( File "", line 441, in get raise ( Foo matching query does not exist. ERROR: test03comparecaseannotation (modelfields.testdecimalfield.DecimalFieldTests) Comparing a Case annotation wrapping a field to a literal works. Traceback (most recent call last): File "", line 437, in inner return func(args, kwargs) File "", line 154, in test03comparecaseannotation Foo.objects.annotate( File "", line 441, in get raise ( Foo matching query does not exist. Note in the printed queries that the 0 value that is compared with is a string in the query ('0'), while the ExpressionWrappered value is just a plain number (1). The Value without ExpressionWrapper has a cast to NUMERIC that I suspect makes that case work? Also note that the DEBUG stuff and try-catch is only to capture the query that is being generated, it is not needed to trigger the problem. I tried printing the attribute first, but that seems to hide the quotes around the literal 0 in the query, so took me a while to realize what was happening. The numbers in the testcase names are to ensure the SQL queries are printed in a recognizable order. All four testcases work on MySQL, there the 0 value is written in the query without the quotes, and the cast to NUMERIC is also missing. I suspect this issue is highly similar to , and can a fix can probably build on the fix for that issue.

return cols  class Case(Expression):class Case(SQLiteNumericMixin, Expression):     """     An SQL searched CASE expression:


Question: SQLite schema introspection should use PRAGMA instead of parsing to retrieve foreign keys Description The SQLite backend's DatabaseIntrospection.getrelations uses a complex combination of regexes to extract foreign key constraints from DDL while PRAGMA foreignkeylist works just fine as getforeignkey_constraints.

import re from collections import namedtuple  import sqlparse from django.db import DatabaseError from django.db.backends.base.introspection import (     BaseDatabaseIntrospection, FieldInfo as BaseFieldInfo, TableInfo, )


Question: SQLite schema introspection should use PRAGMA instead of parsing to retrieve foreign keys Description The SQLite backend's DatabaseIntrospection.getrelations uses a complex combination of regexes to extract foreign key constraints from DDL while PRAGMA foreignkeylist works just fine as getforeignkey_constraints.

"""         cursor.execute('PRAGMA table_info(%s)' % self.connection.ops.quote_name(table_name))         table_info = cursor.fetchall()<ins>        if not table_info:            raise DatabaseError(f'Table {table_name} does not exist (empty pragma).')</ins>         collations = self._get_column_collations(cursor, table_name)         json_columns = set()         if self.connection.features.can_introspect_json_field:


Question: SQLite schema introspection should use PRAGMA instead of parsing to retrieve foreign keys Description The SQLite backend's DatabaseIntrospection.getrelations uses a complex combination of regexes to extract foreign key constraints from DDL while PRAGMA foreignkeylist works just fine as getforeignkey_constraints.

def get_relations(self, cursor, table_name):         """        Return a dictionary of {field_name: (field_name_other_table, other_table)}        Return a dictionary of {column_name: (ref_column_name, ref_table_name)}         representing all foreign keys in the given table.         """        # Dictionary of relations to return        relations = {}        # Schema for this table         cursor.execute(            "SELECT sql, type FROM sqlite_master "            "WHERE tbl_name = %s AND type IN ('table', 'view')",            [table_name]            'PRAGMA foreign_key_list(%s)' % self.connection.ops.quote_name(table_name)         )        create_sql, table_type = cursor.fetchone()        if table_type == 'view':            # It might be a view, then no results will be returned            return relations        results = create_sql[create_sql.index('(') + 1:create_sql.rindex(')')]        # Walk through and look for references to other tables. SQLite doesn't        # really have enforced references, but since it echoes out the SQL used        # to create the table we can look for REFERENCES statements used there.        for field_desc in results.split(','):            field_desc = field_desc.strip()            if field_desc.startswith("UNIQUE"):                continue            m = re.search(r'references (\S*) ?\(["|]?(.*)["|]?\)', field_desc, re.I)            if not m:                continue            table, column = [s.strip('"') for s in m.groups()]            if field_desc.startswith("FOREIGN KEY"):                # Find name of the target FK field                m = re.match(r'FOREIGN KEY\s*\(([^\)]*)\).*', field_desc, re.I)                field_name = m[1].strip('"')            else:                field_name = field_desc.split()[0].strip('"')            cursor.execute("SELECT sql FROM sqlite_master WHERE tbl_name = %s", [table])            result = cursor.fetchall()[0]            other_table_results = result[0].strip()            li, ri = other_table_results.index('('), other_table_results.rindex(')')            other_table_results = other_table_results[li + 1:ri]            for other_desc in other_table_results.split(','):                other_desc = other_desc.strip()                if other_desc.startswith('UNIQUE'):                    continue                other_name = other_desc.split(' ', 1)[0].strip('"')                if other_name == column:                    relations[field_name] = (other_name, table)                    break        return relations        return {            column_name: (ref_column_name, ref_table_name)            for _, _, ref_table_name, column_name, ref_column_name, *_ in cursor.fetchall()        }      def get_primary_key_column(self, cursor, table_name):         """Return the column name of the primary key for the given table."""


Question: SQLite schema introspection should use PRAGMA instead of parsing to retrieve foreign keys Description The SQLite backend's DatabaseIntrospection.getrelations uses a complex combination of regexes to extract foreign key constraints from DDL while PRAGMA foreignkeylist works just fine as getforeignkey_constraints.

return name         return None <del>    def _get_foreign_key_constraints(self, cursor, table_name):        constraints = {}        cursor.execute('PRAGMA foreign_key_list(%s)' % self.connection.ops.quote_name(table_name))        for row in cursor.fetchall():            # Remaining on_update/on_delete/match values are of no interest.            id_, _, table, from_, to = row[:5]            constraints['fk_%d' % id_] = {                'columns': [from_],                'primary_key': False,                'unique': False,                'foreign_key': (table, to),                'check': False,                'index': False,            }        return constraints</del>     def _parse_column_or_constraint_definition(self, tokens, columns):         token = None         is_constraint_definition = None


Question: SQLite schema introspection should use PRAGMA instead of parsing to retrieve foreign keys Description The SQLite backend's DatabaseIntrospection.getrelations uses a complex combination of regexes to extract foreign key constraints from DDL while PRAGMA foreignkeylist works just fine as getforeignkey_constraints.

"check": False,                 "index": False,             }        constraints.update(self._get_foreign_key_constraints(cursor, table_name))        relations = enumerate(self.get_relations(cursor, table_name).items())        constraints.update({            f'fk_{index}': {                'columns': [column_name],                'primary_key': False,                'unique': False,                'foreign_key': (ref_table_name, ref_column_name),                'check': False,                'index': False,            }            for index, (column_name, (ref_column_name, ref_table_name)) in relations        })         return constraints      def _get_index_columns_orders(self, sql):diff --git a/django/db/backends/sqlite3/schema.py b/django/db/backends/sqlite3/schema.py-- a/django/db/backends/sqlite3/schema.py++ b/django/db/backends/sqlite3/schema.py


Question: SQLite schema introspection should use PRAGMA instead of parsing to retrieve foreign keys Description The SQLite backend's DatabaseIntrospection.getrelations uses a complex combination of regexes to extract foreign key constraints from DDL while PRAGMA foreignkeylist works just fine as getforeignkey_constraints.

for other_table in self.connection.introspection.get_table_list(cursor):                 if ignore_self and other_table.name == table_name:                     continue                constraints = self.connection.introspection._get_foreign_key_constraints(cursor, other_table.name)                for constraint in constraints.values():                    constraint_table, constraint_column = constraint['foreign_key']                relations = self.connection.introspection.get_relations(cursor, other_table.name)                for constraint_column, constraint_table in relations.values():                     if (constraint_table == table_name and                             (column_name is None or constraint_column == column_name)):                         return True


Question: Internationalisation didn't support language locale containing both script and region. Description The i18npatterns didn't work with locale contains both script and region, like en-latn-us. Given LANGUAGECODE = 'en-us' LANGUAGES = [ ('en-us', "English"), ('en-latn-us', "Latin English"), ('en-Latn-US', "BCP 47 case format"), ] from import i18npatterns from import HttpResponse def bangiah(request): return HttpResponse('U!') urlpatterns += i18npatterns( path('', bangiah), ) The response of http://localhost:8000/en-us/ is 200 U!. The response of http://localhost:8000/en-lat-us/ is 404 not found. The response of http://localhost:8000/en-Latn-US/ is 404 not found. Steps to Reproduce Start a new project with django-admin startproject tshi and cd tshi/ Append to as follows LANGUAGES = [ ('en-us', "English"), ('en-latn-us', "Latin English"), ('en-Latn-US', "BCP 47 case format"), ] MIDDLEWARE += [ '', ] Edit by appending follows from import i18npatterns from import HttpResponse def bangiah(request): return HttpResponse('U!') urlpatterns += i18npatterns( path('', bangiah), ) python migrate python runserver The results The response of http://localhost:8000/en-us/ is 200 U!. The response of http://localhost:8000/en-lat-us/ is 404 not found. The response of http://localhost:8000/en-Latn-US/ is 404 not found. Expect to happen instead The response of http://localhost:8000/en-latn-us/ and http://localhost:8000/en-Latn-US/ should be 200 U!. The en-Latn-US tag follows format defined in RFC 5646. It's documented that the language part is always in lowercase, following Accept-Language. Accept-Language is following Content-Language Header, which is following RFC 5646. The RFC 5646 defined langtag as follow: langtag = language ["-" script] ["-" region] ("-" variant) ("-" extension) ["-" privateuse] language = 23ALPHA ; shortest ISO 639 code ["-" extlang] ; sometimes followed by ; extended language subtags / 4ALPHA ; or reserved for future use / 58ALPHA ; or registered language subtag extlang = 3ALPHA ; selected ISO 639 codes *2("-" 3ALPHA) ; permanently reserved script = 4ALPHA ; ISO code region = 2ALPHA ; ISO 3166-1 code / 3DIGIT ; UN M.49 code I have confirmed that this issue can be reproduced as described on a fresh Django project Python version: 3.7.5 Django version: 3.2.7

re.IGNORECASE ) language_code_prefix_re = _lazy_re_compile(r'^/(\w+([@-]\w+)?)(/|$)')language_code_prefix_re = _lazy_re_compile(r'^/(\w+([@-]\w+){0,2})(/|$)')   @receiver(setting_changed)


Question: startproject should honor umask Description Ticket fixed the problem of not being made executable by copying all the permission bits (not just the executable flags). This means that the user's umask doesn't work, e.g.: $ umask 077 $ touch foo $ ls -l foo -rw------- 1 talex talex 0 2007-05-12 13:27 foo $ PYTHONPATH=trunk ./trunk/django/bin/django- startproject mysite $ ls -l -rw-r--r-- 1 talex talex 2804 2007-05-12 13:28 I discovered this whilst trying to make a Zero Install package for Django. Everything in the Zero Install cache is read-only, so startproject fails with: File "", line 799, in startproject fp = open(mainsettingsfile, 'w') IOError: [Errno 13] Permission denied: '' Thanks,

if self.verbosity >= 2:                     self.stdout.write('Creating %s' % new_path)                 try:                    shutil.copymode(old_path, new_path)                    self.apply_umask(old_path, new_path)                     self.make_writeable(new_path)                 except OSError:                     self.stderr.write(


Question: startproject should honor umask Description Ticket fixed the problem of not being made executable by copying all the permission bits (not just the executable flags). This means that the user's umask doesn't work, e.g.: $ umask 077 $ touch foo $ ls -l foo -rw------- 1 talex talex 0 2007-05-12 13:27 foo $ PYTHONPATH=trunk ./trunk/django/bin/django- startproject mysite $ ls -l -rw-r--r-- 1 talex talex 2804 2007-05-12 13:28 I discovered this whilst trying to make a Zero Install package for Django. Everything in the Zero Install cache is read-only, so startproject fails with: File "", line 799, in startproject fp = open(mainsettingsfile, 'w') IOError: [Errno 13] Permission denied: '' Thanks,

scheme = template.split(':', 1)[0].lower()         return scheme in self.url_schemes <ins>    def apply_umask(self, old_path, new_path):        current_umask = os.umask(0)        os.umask(current_umask)        current_mode = stat.S_IMODE(os.stat(old_path).st_mode)        os.chmod(new_path, current_mode & ~current_umask)</ins>     def make_writeable(self, filename):         """         Make sure that the file is writeable.


Question: Make the elementid argument of jsonscript optional Description I recently had a use-case where I wanted to use json_script but I didn't need any id for it (I was including the <scriptinside a <templateso I didn't need an id to refer to it). I can't see any reason (security or otherwise) for the id to be required and making it optional doesn't seem to break any tests.

@register.filter(is_safe=True)def json_script(value, element_id):def json_script(value, element_id=None):     """     Output value JSON-encoded, wrapped in a


Question: Make the elementid argument of jsonscript optional Description I recently had a use-case where I wanted to use json_script but I didn't need any id for it (I was including the <scriptinside a <templateso I didn't need an id to refer to it). I can't see any reason (security or otherwise) for the id to be required and making it optional doesn't seem to break any tests.

}  def json_script(value, element_id):def json_script(value, element_id=None):     """     Escape all the HTML/XML special characters with their unicode escapes, so     value is safe to be output anywhere except for inside a tag attribute. Wrap


Question: Make the elementid argument of jsonscript optional Description I recently had a use-case where I wanted to use json_script but I didn't need any id for it (I was including the <scriptinside a <templateso I didn't need an id to refer to it). I can't see any reason (security or otherwise) for the id to be required and making it optional doesn't seem to break any tests.

"""     from django.core.serializers.json import DjangoJSONEncoder     json_str = json.dumps(value, cls=DjangoJSONEncoder).translate(_json_script_escapes)    return format_html(        '',        element_id, mark_safe(json_str)    )    if element_id:        template = ''        args = (element_id, mark_safe(json_str))    else:        template = ''        args = (mark_safe(json_str),)    return format_html(template, *args)   def conditional_escape(text):


Question: KeyError with migration autodetector and FK field with hardcoded reference Description Hi, I encountered this issue on an old Django project (probably 10 years old) with tons of models and probably a lot of questionable design decisions. The symptom is that running our test suite in verbose mode doesn't work: $ python test -v 2 Creating test database for alias 'default' ('testproject')... Operations to perform: Synchronize unmigrated apps: [... about 40 apps] Apply all migrations: (none) Synchronizing apps without migrations: Creating tables... Creating table authpermission Creating table authgroup Creating table authuser Creating table djangocontenttype Creating table djangosession Creating table djangoadminlog [... 100 or so more tables] Running deferred SQL... Running migrations: No migrations to apply. Traceback (most recent call last): File "", line 17, in <moduleexecutefromcommandline() File "", line 401, in executefromcommandline utility.execute() File "", line 395, in execute self.fetchcommand(subcommand).runfromargv() File "", line 23, in runfromargv super().runfromargv(argv) File "", line 330, in runfromargv self.execute(args, cmdoptions) File "", line 371, in execute output = (args, options) File "", line 53, in handle failures = testrunner.runtests(testlabels) File "", line 697, in runtests oldconfig = self.setupdatabases(aliases=databases) File "", line 618, in setupdatabases self.parallel, kwargs File "", line 174, in setupdatabases serialize=connection.settingsdict['TEST'].get('SERIALIZE', True), File "", line 77, in createtestdb runsyncdb=True, File "", line 168, in callcommand return command.execute(args, defaults) File "", line 371, in execute output = (args, options) File "", line 85, in wrapped res = handlefunc(args, kwargs) File "", line 227, in handle changes = autodetector.changes() File "", line 43, in changes changes = self.detectchanges(convertapps, graph) File "", line 160, in detectchanges self.generaterenamedmodels() File "", line 476, in generaterenamedmodels modelfieldsdef = self.onlyrelationagnosticfields() File "", line 99, in onlyrelationagnosticfields del deconstruction[2]['to'] KeyError: 'to' I finally did some digging and found that the culprit is a custom ForeignKey field that hardcodes its to argument (and thus also removes it from its deconstructed kwargs). It seems that the autodetector doesn't like that. Here's a self-contained reproduction test to replicate the issue: from import models from import MigrationAutodetector from import ModelState, ProjectState from import TestCase class CustomFKField(models.ForeignKey): def init(self, args, kwargs): kwargs['to'] = 'testapp.HardcodedModel' super().init(*args, kwargs) def deconstruct(self): name, path, args, kwargs = super().deconstruct() del kwargs["to"] return name, path, args, kwargs class ReproTestCase(TestCase): def testreprodution(self): before = ProjectState() before.addmodel(ModelState('testapp', 'HardcodedModel', [])) after = ProjectState() after.addmodel(ModelState('testapp', 'HardcodedModel', [])) after.addmodel(ModelState('testapp', 'TestModel', [('custom', CustomFKField(ondelete=models.CASCADE))])) changes = MigrationAutodetector(before, after).detectchanges() self.assertEqual(len(changes['testapp']), 1) While I'll happily admit that my custom field's design might be questionable, I don't think it's incorrect and I think the autodetector is at fault here. Changing del deconstruction[2]['to'] to deconstruction[2].pop('to', None) on the line indicated by the traceback makes my test suite run again, in all its glorious verbosity. Seems like an innocent enough fix to me.

for name, field in sorted(fields.items()):             deconstruction = self.deep_deconstruct(field)             if field.remote_field and field.remote_field.model:                del deconstruction[2]['to']                deconstruction[2].pop('to', None)             fields_def.append(deconstruction)         return fields_def


Question: Window(orderby) should allow usage of descending string syntax to be used Description (last modified by Simon Charette) The QuerySet.orderby and some aggregates ordering kwarg allows for the leading dash syntax to be used but Window.orderby doesn't as it solely wraps the provided orderby in ExpressionList(expressions=orderby). This makes for an inconsistent API so I suggest we reuse the logic in in As a related note it seems most of the logic of OrderableAggMixin could be simplified by using ExpressionList. It's a shame that we used ordering and not orderby as a kwarg for OrderableAggMixin as it's now inconsistent. Also not sure how much of a public API the OrderBy expression is but I wish it was initially named Sort (or Ordering?) so that we could define class OrderBy(ExpressionList): template = 'ORDER BY %(expressions)s' def init(self, expressions, extra): expressions = [ (Sort(F(expr[1:]), descending=True) if isinstance(expr, str) and expr[0] == '-' else expr) for expr in expressions ] super().init(expressions, *extra) And then simply use this abstraction in Window and Postgres orderable aggregates. Assigning to myself as I plan to have a look at this in next few days.

from django.db.models import F, OrderByfrom django.db.models.expressions import OrderByList   class OrderableAggMixin:      def __init__(self, *expressions, ordering=(), **extra):        if not isinstance(ordering, (list, tuple)):            ordering = [ordering]        ordering = ordering or []        # Transform minus sign prefixed strings into an OrderBy() expression.        ordering = (            (OrderBy(F(o[1:]), descending=True) if isinstance(o, str) and o[0] == '-' else o)            for o in ordering        )        if isinstance(ordering, (list, tuple)):            self.order_by = OrderByList(*ordering)        else:            self.order_by = OrderByList(ordering)         super().__init__(*expressions, **extra)        self.ordering = self._parse_expressions(*ordering)      def resolve_expression(self, *args, **kwargs):        self.ordering = [expr.resolve_expression(*args, **kwargs) for expr in self.ordering]        self.order_by = self.order_by.resolve_expression(*args, **kwargs)         return super().resolve_expression(*args, **kwargs)     def as_sql(self, compiler, connection):        if self.ordering:            ordering_params = []            ordering_expr_sql = []            for expr in self.ordering:                expr_sql, expr_params = compiler.compile(expr)                ordering_expr_sql.append(expr_sql)                ordering_params.extend(expr_params)            sql, sql_params = super().as_sql(compiler, connection, ordering=(                'ORDER BY ' + ', '.join(ordering_expr_sql)            ))            return sql, (*sql_params, *ordering_params)        return super().as_sql(compiler, connection, ordering='')    def get_source_expressions(self):        return super().get_source_expressions() + [self.order_by]      def set_source_expressions(self, exprs):        # Extract the ordering expressions because ORDER BY clause is handled        # in a custom way.        self.ordering = exprs[self._get_ordering_expressions_index():]        return super().set_source_expressions(exprs[:self._get_ordering_expressions_index()])        *exprs, self.order_by = exprs        return super().set_source_expressions(exprs)     def get_source_expressions(self):        return super().get_source_expressions() + self.ordering    def _get_ordering_expressions_index(self):        """Return the index at which the ordering expressions start."""        source_expressions = self.get_source_expressions()        return len(source_expressions) - len(self.ordering)    def as_sql(self, compiler, connection):        order_by_sql, order_by_params = compiler.compile(self.order_by)        sql, sql_params = super().as_sql(compiler, connection, ordering=order_by_sql)        return sql, (*sql_params, *order_by_params)diff --git a/django/db/models/expressions.py b/django/db/models/expressions.py-- a/django/db/models/expressions.py++ b/django/db/models/expressions.py


Question: Window(orderby) should allow usage of descending string syntax to be used Description (last modified by Simon Charette) The QuerySet.orderby and some aggregates ordering kwarg allows for the leading dash syntax to be used but Window.orderby doesn't as it solely wraps the provided orderby in ExpressionList(expressions=orderby). This makes for an inconsistent API so I suggest we reuse the logic in in As a related note it seems most of the logic of OrderableAggMixin could be simplified by using ExpressionList. It's a shame that we used ordering and not orderby as a kwarg for OrderableAggMixin as it's now inconsistent. Also not sure how much of a public API the OrderBy expression is but I wish it was initially named Sort (or Ordering?) so that we could define class OrderBy(ExpressionList): template = 'ORDER BY %(expressions)s' def init(self, expressions, extra): expressions = [ (Sort(F(expr[1:]), descending=True) if isinstance(expr, str) and expr[0] == '-' else expr) for expr in expressions ] super().init(expressions, *extra) And then simply use this abstraction in Window and Postgres orderable aggregates. Assigning to myself as I plan to have a look at this in next few days.

class ExpressionList(Func):     """     An expression containing multiple expressions. Can be used to provide a    list of expressions as an argument to another expression, like an    ordering clause.    list of expressions as an argument to another expression, like a partition    clause.     """     template = '%(expressions)s'


Question: Window(orderby) should allow usage of descending string syntax to be used Description (last modified by Simon Charette) The QuerySet.orderby and some aggregates ordering kwarg allows for the leading dash syntax to be used but Window.orderby doesn't as it solely wraps the provided orderby in ExpressionList(expressions=orderby). This makes for an inconsistent API so I suggest we reuse the logic in in As a related note it seems most of the logic of OrderableAggMixin could be simplified by using ExpressionList. It's a shame that we used ordering and not orderby as a kwarg for OrderableAggMixin as it's now inconsistent. Also not sure how much of a public API the OrderBy expression is but I wish it was initially named Sort (or Ordering?) so that we could define class OrderBy(ExpressionList): template = 'ORDER BY %(expressions)s' def init(self, expressions, extra): expressions = [ (Sort(F(expr[1:]), descending=True) if isinstance(expr, str) and expr[0] == '-' else expr) for expr in expressions ] super().init(expressions, *extra) And then simply use this abstraction in Window and Postgres orderable aggregates. Assigning to myself as I plan to have a look at this in next few days.

return self.as_sql(compiler, connection, **extra_context)  <ins>class OrderByList(Func):    template = 'ORDER BY %(expressions)s'    def __init__(self, *expressions, **extra):        expressions = (            (                OrderBy(F(expr[1:]), descending=True)                if isinstance(expr, str) and expr[0] == '-'                else expr            )            for expr in expressions        )        super().__init__(*expressions, **extra)    def as_sql(self, *args, **kwargs):        if not self.source_expressions:            return '', ()        return super().as_sql(*args, **kwargs)</ins> class ExpressionWrapper(SQLiteNumericMixin, Expression):     """     An expression that can wrap another expression so that it can provide


Question: Window(orderby) should allow usage of descending string syntax to be used Description (last modified by Simon Charette) The QuerySet.orderby and some aggregates ordering kwarg allows for the leading dash syntax to be used but Window.orderby doesn't as it solely wraps the provided orderby in ExpressionList(expressions=orderby). This makes for an inconsistent API so I suggest we reuse the logic in in As a related note it seems most of the logic of OrderableAggMixin could be simplified by using ExpressionList. It's a shame that we used ordering and not orderby as a kwarg for OrderableAggMixin as it's now inconsistent. Also not sure how much of a public API the OrderBy expression is but I wish it was initially named Sort (or Ordering?) so that we could define class OrderBy(ExpressionList): template = 'ORDER BY %(expressions)s' def init(self, expressions, extra): expressions = [ (Sort(F(expr[1:]), descending=True) if isinstance(expr, str) and expr[0] == '-' else expr) for expr in expressions ] super().init(expressions, *extra) And then simply use this abstraction in Window and Postgres orderable aggregates. Assigning to myself as I plan to have a look at this in next few days.

if self.order_by is not None:             if isinstance(self.order_by, (list, tuple)):                self.order_by = ExpressionList(*self.order_by)            elif not isinstance(self.order_by, BaseExpression):                self.order_by = OrderByList(*self.order_by)            elif isinstance(self.order_by, (BaseExpression, str)):                self.order_by = OrderByList(self.order_by)            else:                 raise ValueError(                    'order_by must be either an Expression or a sequence of '                    'expressions.'                    'Window.order_by must be either a string reference to a '                    'field, an expression, or a list or tuple of them.'                 )         super().__init__(output_field=output_field)         self.source_expression = self._parse_expressions(expression)[0]


Question: Window(orderby) should allow usage of descending string syntax to be used Description (last modified by Simon Charette) The QuerySet.orderby and some aggregates ordering kwarg allows for the leading dash syntax to be used but Window.orderby doesn't as it solely wraps the provided orderby in ExpressionList(expressions=orderby). This makes for an inconsistent API so I suggest we reuse the logic in in As a related note it seems most of the logic of OrderableAggMixin could be simplified by using ExpressionList. It's a shame that we used ordering and not orderby as a kwarg for OrderableAggMixin as it's now inconsistent. Also not sure how much of a public API the OrderBy expression is but I wish it was initially named Sort (or Ordering?) so that we could define class OrderBy(ExpressionList): template = 'ORDER BY %(expressions)s' def init(self, expressions, extra): expressions = [ (Sort(F(expr[1:]), descending=True) if isinstance(expr, str) and expr[0] == '-' else expr) for expr in expressions ] super().init(expressions, *extra) And then simply use this abstraction in Window and Postgres orderable aggregates. Assigning to myself as I plan to have a look at this in next few days.

compiler=compiler, connection=connection,                 template='PARTITION BY %(expressions)s',             )            window_sql.extend(sql_expr)            window_sql.append(sql_expr)             window_params.extend(sql_params)          if self.order_by is not None:            window_sql.append(' ORDER BY ')             order_sql, order_params = compiler.compile(self.order_by)            window_sql.extend(order_sql)            window_sql.append(order_sql)             window_params.extend(order_params)          if self.frame:             frame_sql, frame_params = compiler.compile(self.frame)            window_sql.append(' ' + frame_sql)            window_sql.append(frame_sql)             window_params.extend(frame_params)          params.extend(window_params)


Question: Window(orderby) should allow usage of descending string syntax to be used Description (last modified by Simon Charette) The QuerySet.orderby and some aggregates ordering kwarg allows for the leading dash syntax to be used but Window.orderby doesn't as it solely wraps the provided orderby in ExpressionList(expressions=orderby). This makes for an inconsistent API so I suggest we reuse the logic in in As a related note it seems most of the logic of OrderableAggMixin could be simplified by using ExpressionList. It's a shame that we used ordering and not orderby as a kwarg for OrderableAggMixin as it's now inconsistent. Also not sure how much of a public API the OrderBy expression is but I wish it was initially named Sort (or Ordering?) so that we could define class OrderBy(ExpressionList): template = 'ORDER BY %(expressions)s' def init(self, expressions, extra): expressions = [ (Sort(F(expr[1:]), descending=True) if isinstance(expr, str) and expr[0] == '-' else expr) for expr in expressions ] super().init(expressions, *extra) And then simply use this abstraction in Window and Postgres orderable aggregates. Assigning to myself as I plan to have a look at this in next few days.

return template % {             'expression': expr_sql,            'window': ''.join(window_sql).strip()            'window': ' '.join(window_sql).strip()         }, params      def as_sqlite(self, compiler, connection):


Question: Window(orderby) should allow usage of descending string syntax to be used Description (last modified by Simon Charette) The QuerySet.orderby and some aggregates ordering kwarg allows for the leading dash syntax to be used but Window.orderby doesn't as it solely wraps the provided orderby in ExpressionList(expressions=orderby). This makes for an inconsistent API so I suggest we reuse the logic in in As a related note it seems most of the logic of OrderableAggMixin could be simplified by using ExpressionList. It's a shame that we used ordering and not orderby as a kwarg for OrderableAggMixin as it's now inconsistent. Also not sure how much of a public API the OrderBy expression is but I wish it was initially named Sort (or Ordering?) so that we could define class OrderBy(ExpressionList): template = 'ORDER BY %(expressions)s' def init(self, expressions, extra): expressions = [ (Sort(F(expr[1:]), descending=True) if isinstance(expr, str) and expr[0] == '-' else expr) for expr in expressions ] super().init(expressions, *extra) And then simply use this abstraction in Window and Postgres orderable aggregates. Assigning to myself as I plan to have a look at this in next few days.

return '{} OVER ({}{}{})'.format(             str(self.source_expression),             'PARTITION BY ' + str(self.partition_by) if self.partition_by else '',            'ORDER BY ' + str(self.order_by) if self.order_by else '',            str(self.order_by or ''),             str(self.frame or ''),         )


Question: django-admin start[project|app] doesn't send proper user agent header when HTTP(S) url is sent. Description (last modified by rsp2k) By default, when fetching a remote template, django-admin start[app|project] uses the default urllib User Agent which causes some sites to block requests (namely gitlab, since they use cloudflare See ). This patch sets the 'User Agent' header to Django/DJANGO_VERSION.

import stat import tempfile from importlib import import_modulefrom urllib.request import urlretrievefrom urllib.request import build_opener  import django from django.conf import settings


Question: django-admin start[project|app] doesn't send proper user agent header when HTTP(S) url is sent. Description (last modified by rsp2k) By default, when fetching a remote template, django-admin start[app|project] uses the default urllib User Agent which causes some sites to block requests (namely gitlab, since they use cloudflare See ). This patch sets the 'User Agent' header to Django/DJANGO_VERSION.

if self.verbosity >= 2:             self.stdout.write('Downloading %s' % display_url)        the_path = os.path.join(tempdir, filename)        opener = build_opener()        opener.addheaders = [('User-Agent', f'Django/{django.__version__}')]         try:            the_path, info = urlretrieve(url, os.path.join(tempdir, filename))            with opener.open(url) as source, open(the_path, 'wb') as target:                headers = source.info()                target.write(source.read())         except OSError as e:             raise CommandError("couldn't download URL %s to %s: %s" %                                (url, filename, e))


Question: django-admin start[project|app] doesn't send proper user agent header when HTTP(S) url is sent. Description (last modified by rsp2k) By default, when fetching a remote template, django-admin start[app|project] uses the default urllib User Agent which causes some sites to block requests (namely gitlab, since they use cloudflare See ). This patch sets the 'User Agent' header to Django/DJANGO_VERSION.

used_name = the_path.split('/')[-1]          # Trying to get better name from response headers        content_disposition = info.get('content-disposition')        content_disposition = headers['content-disposition']         if content_disposition:             _, params = cgi.parse_header(content_disposition)             guessed_filename = params.get('filename') or used_name


Question: django-admin start[project|app] doesn't send proper user agent header when HTTP(S) url is sent. Description (last modified by rsp2k) By default, when fetching a remote template, django-admin start[app|project] uses the default urllib User Agent which causes some sites to block requests (namely gitlab, since they use cloudflare See ). This patch sets the 'User Agent' header to Django/DJANGO_VERSION.

# Falling back to content type guessing         ext = self.splitext(guessed_filename)[1]        content_type = info.get('content-type')        content_type = headers['content-type']         if not ext and content_type:             ext = mimetypes.guess_extension(content_type)             if ext:


Question: LEVELTAGS not updated when using Description When reading messages inside tests, new message tags created using is not updated. That causes the property results to be an empty string and not know the new tags.

from django.apps import AppConfigfrom django.contrib.messages.storage import basefrom django.contrib.messages.utils import get_level_tagsfrom django.test.signals import setting_changed from django.utils.translation import gettext_lazy as _  def update_level_tags(setting, **kwargs):    if setting == 'MESSAGE_TAGS':        base.LEVEL_TAGS = get_level_tags() class MessagesConfig(AppConfig):     name = 'django.contrib.messages'     verbose_name = _("Messages")    def ready(self):        setting_changed.connect(update_level_tags)


Question: Query.changealiases raises an AssertionError Description Python Version: 3.9.2 Django Version: 2.2.24, 3.2.9 (reproduced using two different versions) Code to Reproduce from import models class Foo(models.Model): qux = models.ForeignKey("app.Qux", ondelete=models.CASCADE, relatedname="foos") class Bar(models.Model): foo = models.ForeignKey("app.Foo", ondelete=models.CASCADE, relatedname="bars") anotherfoo = models.ForeignKey("app.Foo", ondelete=models.CASCADE, relatedname="otherbars") baz = models.ForeignKey("app.Baz", ondelete=models.CASCADE, relatedname="bars") class Baz(models.Model): pass class Qux(models.Model): bazes = models.ManyToManyField("app.Baz", relatedname="quxes") from import Q from import Foo, Qux qux = () qs1 = () qs2 = ( Q(()) | Q(()) ) qs2 | qs1 qs1 | qs2 Description I have encountered this bug during working on a project, recreated the code to reproduce as simple as I can. I have also examined the reason behind this bug, as far as I understand the reason is that during an or operation of two QuerySets, in Query.combine method of the variable combined, if rhs's Query currently have sequential aliases (e.g. T4 and T5) and related tablenames also exist in lhs.tablemap, calling Query.tablealias in will result in creation of aliases T5 for T4 and T6 for T5, thus changemap's keys intersect with changemap's values, so the AssertionError above is raised. Expectation Could you please fix this bug? Maybe aliasmap of rhs can be provided to and Query.tablealias, and suffix (number) of the new alias might be incremented until it is not in rhs.aliasmap, to prevent intersection between changemap's keys and values. Assertion in the first line of QuerySet.changealiases is not documented via a comment. As far as I understand, it is there because if keys and values intersects it means that an alias might be changed twice (e.g. first T4 -T5, and then T5 -T6) according to their order in the changemap. IMHO there can be a comment about what it assures, or an explanation can be to the AssertionError (like the assertions in the Query.combine method). It seems like QuerySet's OR operation is not commutative (they can create different queries, even though the results are the same), IMHO this can be explicitly declared on the documentation.

if self.distinct_fields != rhs.distinct_fields:             raise TypeError('Cannot combine queries with different distinct fields.') <ins>        # If lhs and rhs shares the same alias prefix, it is possible to have        # conflicting alias changes like T4 -> T5, T5 -> T6, which might end up        # as T4 -> T6 while combining two querysets. To prevent this, change an        # alias prefix of the rhs and update current aliases accordingly,        # except if the alias is the base table since it must be present in the        # query on both sides.        initial_alias = self.get_initial_alias()        rhs.bump_prefix(self, exclude={initial_alias})</ins>         # Work out how to relabel the rhs aliases, if necessary.         change_map = {}         conjunction = (connector == AND)


Question: Query.changealiases raises an AssertionError Description Python Version: 3.9.2 Django Version: 2.2.24, 3.2.9 (reproduced using two different versions) Code to Reproduce from import models class Foo(models.Model): qux = models.ForeignKey("app.Qux", ondelete=models.CASCADE, relatedname="foos") class Bar(models.Model): foo = models.ForeignKey("app.Foo", ondelete=models.CASCADE, relatedname="bars") anotherfoo = models.ForeignKey("app.Foo", ondelete=models.CASCADE, relatedname="otherbars") baz = models.ForeignKey("app.Baz", ondelete=models.CASCADE, relatedname="bars") class Baz(models.Model): pass class Qux(models.Model): bazes = models.ManyToManyField("app.Baz", relatedname="quxes") from import Q from import Foo, Qux qux = () qs1 = () qs2 = ( Q(()) | Q(()) ) qs2 | qs1 qs1 | qs2 Description I have encountered this bug during working on a project, recreated the code to reproduce as simple as I can. I have also examined the reason behind this bug, as far as I understand the reason is that during an or operation of two QuerySets, in Query.combine method of the variable combined, if rhs's Query currently have sequential aliases (e.g. T4 and T5) and related tablenames also exist in lhs.tablemap, calling Query.tablealias in will result in creation of aliases T5 for T4 and T6 for T5, thus changemap's keys intersect with changemap's values, so the AssertionError above is raised. Expectation Could you please fix this bug? Maybe aliasmap of rhs can be provided to and Query.tablealias, and suffix (number) of the new alias might be incremented until it is not in rhs.aliasmap, to prevent intersection between changemap's keys and values. Assertion in the first line of QuerySet.changealiases is not documented via a comment. As far as I understand, it is there because if keys and values intersects it means that an alias might be changed twice (e.g. first T4 -T5, and then T5 -T6) according to their order in the changemap. IMHO there can be a comment about what it assures, or an explanation can be to the AssertionError (like the assertions in the Query.combine method). It seems like QuerySet's OR operation is not commutative (they can create different queries, even though the results are the same), IMHO this can be explicitly declared on the documentation.

# the AND case. The results will be correct but this creates too many         # joins. This is something that could be fixed later on.         reuse = set() if conjunction else set(self.alias_map)<del>        # Base table must be present in the query - this is the same        # table on both sides.        self.get_initial_alias()</del>         joinpromoter = JoinPromoter(connector, 2, False)         joinpromoter.add_votes(             j for j in self.alias_map if self.alias_map[j].join_type == INNER)


Question: Query.changealiases raises an AssertionError Description Python Version: 3.9.2 Django Version: 2.2.24, 3.2.9 (reproduced using two different versions) Code to Reproduce from import models class Foo(models.Model): qux = models.ForeignKey("app.Qux", ondelete=models.CASCADE, relatedname="foos") class Bar(models.Model): foo = models.ForeignKey("app.Foo", ondelete=models.CASCADE, relatedname="bars") anotherfoo = models.ForeignKey("app.Foo", ondelete=models.CASCADE, relatedname="otherbars") baz = models.ForeignKey("app.Baz", ondelete=models.CASCADE, relatedname="bars") class Baz(models.Model): pass class Qux(models.Model): bazes = models.ManyToManyField("app.Baz", relatedname="quxes") from import Q from import Foo, Qux qux = () qs1 = () qs2 = ( Q(()) | Q(()) ) qs2 | qs1 qs1 | qs2 Description I have encountered this bug during working on a project, recreated the code to reproduce as simple as I can. I have also examined the reason behind this bug, as far as I understand the reason is that during an or operation of two QuerySets, in Query.combine method of the variable combined, if rhs's Query currently have sequential aliases (e.g. T4 and T5) and related tablenames also exist in lhs.tablemap, calling Query.tablealias in will result in creation of aliases T5 for T4 and T6 for T5, thus changemap's keys intersect with changemap's values, so the AssertionError above is raised. Expectation Could you please fix this bug? Maybe aliasmap of rhs can be provided to and Query.tablealias, and suffix (number) of the new alias might be incremented until it is not in rhs.aliasmap, to prevent intersection between changemap's keys and values. Assertion in the first line of QuerySet.changealiases is not documented via a comment. As far as I understand, it is there because if keys and values intersects it means that an alias might be changed twice (e.g. first T4 -T5, and then T5 -T6) according to their order in the changemap. IMHO there can be a comment about what it assures, or an explanation can be to the AssertionError (like the assertions in the Query.combine method). It seems like QuerySet's OR operation is not commutative (they can create different queries, even though the results are the same), IMHO this can be explicitly declared on the documentation.

relabelling any references to them in select columns and the where         clause.         """<ins>        # If keys and values of change_map were to intersect, an alias might be        # updated twice (e.g. T4 -> T5, T5 -> T6, so also T4 -> T6) depending        # on their order in change_map.</ins>         assert set(change_map).isdisjoint(change_map.values())          # 1. Update references in "select" (normal columns plus aliases),


Question: Query.changealiases raises an AssertionError Description Python Version: 3.9.2 Django Version: 2.2.24, 3.2.9 (reproduced using two different versions) Code to Reproduce from import models class Foo(models.Model): qux = models.ForeignKey("app.Qux", ondelete=models.CASCADE, relatedname="foos") class Bar(models.Model): foo = models.ForeignKey("app.Foo", ondelete=models.CASCADE, relatedname="bars") anotherfoo = models.ForeignKey("app.Foo", ondelete=models.CASCADE, relatedname="otherbars") baz = models.ForeignKey("app.Baz", ondelete=models.CASCADE, relatedname="bars") class Baz(models.Model): pass class Qux(models.Model): bazes = models.ManyToManyField("app.Baz", relatedname="quxes") from import Q from import Foo, Qux qux = () qs1 = () qs2 = ( Q(()) | Q(()) ) qs2 | qs1 qs1 | qs2 Description I have encountered this bug during working on a project, recreated the code to reproduce as simple as I can. I have also examined the reason behind this bug, as far as I understand the reason is that during an or operation of two QuerySets, in Query.combine method of the variable combined, if rhs's Query currently have sequential aliases (e.g. T4 and T5) and related tablenames also exist in lhs.tablemap, calling Query.tablealias in will result in creation of aliases T5 for T4 and T6 for T5, thus changemap's keys intersect with changemap's values, so the AssertionError above is raised. Expectation Could you please fix this bug? Maybe aliasmap of rhs can be provided to and Query.tablealias, and suffix (number) of the new alias might be incremented until it is not in rhs.aliasmap, to prevent intersection between changemap's keys and values. Assertion in the first line of QuerySet.changealiases is not documented via a comment. As far as I understand, it is there because if keys and values intersects it means that an alias might be changed twice (e.g. first T4 -T5, and then T5 -T6) according to their order in the changemap. IMHO there can be a comment about what it assures, or an explanation can be to the AssertionError (like the assertions in the Query.combine method). It seems like QuerySet's OR operation is not commutative (they can create different queries, even though the results are the same), IMHO this can be explicitly declared on the documentation.

for alias, aliased in self.external_aliases.items()         }     def bump_prefix(self, outer_query):    def bump_prefix(self, other_query, exclude=None):         """         Change the alias prefix to the next letter in the alphabet in a way        that the outer query's aliases and this query's aliases will not        that the other query's aliases and this query's aliases will not         conflict. Even tables that previously had no alias will get an alias        after this call.        after this call. To prevent changing aliases use the exclude parameter.         """         def prefix_gen():             """


Question: Query.changealiases raises an AssertionError Description Python Version: 3.9.2 Django Version: 2.2.24, 3.2.9 (reproduced using two different versions) Code to Reproduce from import models class Foo(models.Model): qux = models.ForeignKey("app.Qux", ondelete=models.CASCADE, relatedname="foos") class Bar(models.Model): foo = models.ForeignKey("app.Foo", ondelete=models.CASCADE, relatedname="bars") anotherfoo = models.ForeignKey("app.Foo", ondelete=models.CASCADE, relatedname="otherbars") baz = models.ForeignKey("app.Baz", ondelete=models.CASCADE, relatedname="bars") class Baz(models.Model): pass class Qux(models.Model): bazes = models.ManyToManyField("app.Baz", relatedname="quxes") from import Q from import Foo, Qux qux = () qs1 = () qs2 = ( Q(()) | Q(()) ) qs2 | qs1 qs1 | qs2 Description I have encountered this bug during working on a project, recreated the code to reproduce as simple as I can. I have also examined the reason behind this bug, as far as I understand the reason is that during an or operation of two QuerySets, in Query.combine method of the variable combined, if rhs's Query currently have sequential aliases (e.g. T4 and T5) and related tablenames also exist in lhs.tablemap, calling Query.tablealias in will result in creation of aliases T5 for T4 and T6 for T5, thus changemap's keys intersect with changemap's values, so the AssertionError above is raised. Expectation Could you please fix this bug? Maybe aliasmap of rhs can be provided to and Query.tablealias, and suffix (number) of the new alias might be incremented until it is not in rhs.aliasmap, to prevent intersection between changemap's keys and values. Assertion in the first line of QuerySet.changealiases is not documented via a comment. As far as I understand, it is there because if keys and values intersects it means that an alias might be changed twice (e.g. first T4 -T5, and then T5 -T6) according to their order in the changemap. IMHO there can be a comment about what it assures, or an explanation can be to the AssertionError (like the assertions in the Query.combine method). It seems like QuerySet's OR operation is not commutative (they can create different queries, even though the results are the same), IMHO this can be explicitly declared on the documentation.

yield ''.join(s)                 prefix = None         if self.alias_prefix != outer_query.alias_prefix:        if self.alias_prefix != other_query.alias_prefix:             # No clashes between self and outer query should be possible.             return


Question: Query.changealiases raises an AssertionError Description Python Version: 3.9.2 Django Version: 2.2.24, 3.2.9 (reproduced using two different versions) Code to Reproduce from import models class Foo(models.Model): qux = models.ForeignKey("app.Qux", ondelete=models.CASCADE, relatedname="foos") class Bar(models.Model): foo = models.ForeignKey("app.Foo", ondelete=models.CASCADE, relatedname="bars") anotherfoo = models.ForeignKey("app.Foo", ondelete=models.CASCADE, relatedname="otherbars") baz = models.ForeignKey("app.Baz", ondelete=models.CASCADE, relatedname="bars") class Baz(models.Model): pass class Qux(models.Model): bazes = models.ManyToManyField("app.Baz", relatedname="quxes") from import Q from import Foo, Qux qux = () qs1 = () qs2 = ( Q(()) | Q(()) ) qs2 | qs1 qs1 | qs2 Description I have encountered this bug during working on a project, recreated the code to reproduce as simple as I can. I have also examined the reason behind this bug, as far as I understand the reason is that during an or operation of two QuerySets, in Query.combine method of the variable combined, if rhs's Query currently have sequential aliases (e.g. T4 and T5) and related tablenames also exist in lhs.tablemap, calling Query.tablealias in will result in creation of aliases T5 for T4 and T6 for T5, thus changemap's keys intersect with changemap's values, so the AssertionError above is raised. Expectation Could you please fix this bug? Maybe aliasmap of rhs can be provided to and Query.tablealias, and suffix (number) of the new alias might be incremented until it is not in rhs.aliasmap, to prevent intersection between changemap's keys and values. Assertion in the first line of QuerySet.changealiases is not documented via a comment. As far as I understand, it is there because if keys and values intersects it means that an alias might be changed twice (e.g. first T4 -T5, and then T5 -T6) according to their order in the changemap. IMHO there can be a comment about what it assures, or an explanation can be to the AssertionError (like the assertions in the Query.combine method). It seems like QuerySet's OR operation is not commutative (they can create different queries, even though the results are the same), IMHO this can be explicitly declared on the documentation.

'Maximum recursion depth exceeded: too many subqueries.'                 )         self.subq_aliases = self.subq_aliases.union([self.alias_prefix])        outer_query.subq_aliases = outer_query.subq_aliases.union(self.subq_aliases)        other_query.subq_aliases = other_query.subq_aliases.union(self.subq_aliases)        if exclude is None:            exclude = {}         self.change_aliases({             alias: '%s%d' % (self.alias_prefix, pos)             for pos, alias in enumerate(self.alias_map)            if alias not in exclude         })      def get_initial_alias(self):


Question: Saving parent object after setting on child leads to unexpected data loss in bulkupdate(). Description Consider following example: class Child(models.Model): pass class Parent(models.Model): child = models.ForeignKey(Child, ondelete=models.CASCADE, null=True) parent = (child=None) = Child() () Parent.objects.bulkupdate([parent], fields=["child"]) Expected behavior: parent model instance was updated with the ID to the child in the database. Actual behavior: Parent model is still referencing Null. There should probably be some check for ForeignKeys in the bulkupdate logic, and if one is updated, the ID of the child model should be re-copied to the child_id field that is actually being written to the database.

using=using, raw=raw,         )     def _prepare_related_fields_for_save(self, operation_name):    def _prepare_related_fields_for_save(self, operation_name, fields=None):         # Ensure that a model instance without a PK hasn't been assigned to         # a ForeignKey or OneToOneField on this model. If the field is         # nullable, allowing the save would result in silent data loss.         for field in self._meta.concrete_fields:            if fields and field not in fields:                continue             # If the related field isn't cached, then an instance hasn't been             # assigned and there's no need to worry about this check.             if field.is_relation and field.is_cached(self):diff --git a/django/db/models/query.py b/django/db/models/query.py-- a/django/db/models/query.py++ b/django/db/models/query.py


Question: Saving parent object after setting on child leads to unexpected data loss in bulkupdate(). Description Consider following example: class Child(models.Model): pass class Parent(models.Model): child = models.ForeignKey(Child, ondelete=models.CASCADE, null=True) parent = (child=None) = Child() () Parent.objects.bulkupdate([parent], fields=["child"]) Expected behavior: parent model instance was updated with the ID to the child in the database. Actual behavior: Parent model is still referencing Null. There should probably be some check for ForeignKeys in the bulkupdate logic, and if one is updated, the ID of the child model should be re-copied to the child_id field that is actually being written to the database.

raise ValueError('bulk_update() cannot be used with primary key fields.')         if not objs:             return 0<ins>        for obj in objs:            obj._prepare_related_fields_for_save(operation_name='bulk_update', fields=fields)</ins>         # PK is used twice in the resulting update query, once in the filter         # and once in the WHEN. Each field will also have one CAST.         connection = connections[self.db]


Question: Deprecate PickleSerializer and move it out of core Description Pickle serializer has long been known to be dangerous. This is mitigated by requiring MAC on pickle in cookies, but nevertheless, RCEs continue to happen:  To further discourage it's use, we should consider deprecating PickleSerializer and moving it into a third party package.

if not expiry:   # Checks both None and 0 cases             return self.get_session_cookie_age()        if not isinstance(expiry, datetime):        if not isinstance(expiry, (datetime, str)):             return expiry        if isinstance(expiry, str):            expiry = datetime.fromisoformat(expiry)         delta = expiry - modification         return delta.days * 86400 + delta.seconds


Question: Deprecate PickleSerializer and move it out of core Description Pickle serializer has long been known to be dangerous. This is mitigated by requiring MAC on pickle in cookies, but nevertheless, RCEs continue to happen:  To further discourage it's use, we should consider deprecating PickleSerializer and moving it into a third party package.

if isinstance(expiry, datetime):             return expiry<ins>        elif isinstance(expiry, str):            return datetime.fromisoformat(expiry)</ins>         expiry = expiry or self.get_session_cookie_age()         return modification + timedelta(seconds=expiry)


Question: Deprecate PickleSerializer and move it out of core Description Pickle serializer has long been known to be dangerous. This is mitigated by requiring MAC on pickle in cookies, but nevertheless, RCEs continue to happen:  To further discourage it's use, we should consider deprecating PickleSerializer and moving it into a third party package.

return         if isinstance(value, timedelta):             value = timezone.now() + value<ins>        if isinstance(value, datetime):            value = value.isoformat()</ins>         self['_session_expiry'] = value      def get_expire_at_browser_close(self):


Question: Deprecate PickleSerializer and move it out of core Description Pickle serializer has long been known to be dangerous. This is mitigated by requiring MAC on pickle in cookies, but nevertheless, RCEs continue to happen:  To further discourage it's use, we should consider deprecating PickleSerializer and moving it into a third party package.

``get_expiry_date()`` or ``get_expiry_age()`` to find the actual expiry         date/age, if there is one.         """        if self.get('_session_expiry') is None:        if (expiry := self.get('_session_expiry')) is None:             return settings.SESSION_EXPIRE_AT_BROWSER_CLOSE        return self.get('_session_expiry') == 0        return expiry == 0      def flush(self):         """diff --git a/django/contrib/sessions/serializers.py b/django/contrib/sessions/serializers.py-- a/django/contrib/sessions/serializers.py++ b/django/contrib/sessions/serializers.py


Question: Deprecate PickleSerializer and move it out of core Description Pickle serializer has long been known to be dangerous. This is mitigated by requiring MAC on pickle in cookies, but nevertheless, RCEs continue to happen:  To further discourage it's use, we should consider deprecating PickleSerializer and moving it into a third party package.

# RemovedInDjango50Warning. from django.core.serializers.base import (     PickleSerializer as BasePickleSerializer, )diff --git a/django/core/cache/backends/redis.py b/django/core/cache/backends/redis.py-- a/django/core/cache/backends/redis.py++ b/django/core/cache/backends/redis.py


Question: Deprecate PickleSerializer and move it out of core Description Pickle serializer has long been known to be dangerous. This is mitigated by requiring MAC on pickle in cookies, but nevertheless, RCEs continue to happen:  To further discourage it's use, we should consider deprecating PickleSerializer and moving it into a third party package.

"""Redis cache backend.""" import pickle import random import re  from django.core.cache.backends.base import DEFAULT_TIMEOUT, BaseCachefrom django.core.serializers.base import PickleSerializer from django.utils.functional import cached_property from django.utils.module_loading import import_string  class RedisSerializer(PickleSerializer):    """    Similar to PickSerializer, except integers are serialized as native Redis    integers for better incr() and decr() atomicity.    """class RedisSerializer:    def __init__(self, protocol=None):        self.protocol = pickle.HIGHEST_PROTOCOL if protocol is None else protocol     def dumps(self, obj):         # Only skip pickling for integers, a int subclasses as bool should be         # pickled.         if type(obj) is int:             return obj        return super().dumps(obj)        return pickle.dumps(obj, self.protocol)      def loads(self, data):         try:             return int(data)         except ValueError:            return super().loads(data)            return pickle.loads(data)   class RedisCacheClient:diff --git a/django/core/serializers/base.py b/django/core/serializers/base.py-- a/django/core/serializers/base.py++ b/django/core/serializers/base.py


Question: Deprecate PickleSerializer and move it out of core Description Pickle serializer has long been known to be dangerous. This is mitigated by requiring MAC on pickle in cookies, but nevertheless, RCEs continue to happen:  To further discourage it's use, we should consider deprecating PickleSerializer and moving it into a third party package.

Module for abstract serializer/unserializer base classes. """ import pickleimport warnings from io import StringIO  from django.core.exceptions import ObjectDoesNotExist from django.db import modelsfrom django.utils.deprecation import RemovedInDjango50Warning  DEFER_FIELD = object()


Question: Deprecate PickleSerializer and move it out of core Description Pickle serializer has long been known to be dangerous. This is mitigated by requiring MAC on pickle in cookies, but nevertheless, RCEs continue to happen:  To further discourage it's use, we should consider deprecating PickleSerializer and moving it into a third party package.

cache backends.     """     def __init__(self, protocol=None):<ins>        warnings.warn(            'PickleSerializer is deprecated due to its security risk. Use '            'JSONSerializer instead.',            RemovedInDjango50Warning,        )</ins>         self.protocol = pickle.HIGHEST_PROTOCOL if protocol is None else protocol      def dumps(self, obj):


Question: Detecting uniqueness doesn't work for models with functional unique constraints. Description When creating a new object from Django Administration site, I'm getting an error "Tag with this already exists.". (Tag is my model name) I'm using on this model the new functional unique constraints introducted in Django 4.0 (). I suppose this is related to the contraints put on the model. However, creating a Tag object with code - e.g. (...) - works fine. It only fails in Django admin. I'm not sure if it's a bug or misunderstanding/misconfiguration on my side? This is my model: class Tag(models.Model): """Basic tag model.""" # When using getorcreate, we need to pass tag name as a default value, like this: # Tag.objects.getorcreate(defaults={'name': tag}, nameiexact=tag, ) user = models.ForeignKey(settings.AUTHUSERMODEL, ondelete=models.CASCADE, verbosename=('user')) name = models.CharField(maxlength=50, dbindex=False, verbosename=('name'), blank=False, # It's the default behavior, but let's be explicit validators=[RegexValidator(regex=r'[A-z0-9À-ž\s]+', message='This field accepts only letters, digits and space.')]) slug = models.SlugField(maxlength=50, verbosename=('slug')) class Meta: ordering = ['name'] # Make sure tag name: # - are unique per user # - are case insensitive (prevent adding "test" if "Test" already exists) # - aren't empty constraints = [models.UniqueConstraint(fields=['user', 'name'], name='%(applabel)s%(class)snameuniqueperuser'), models.UniqueConstraint(Lower('name'), name='%(applabel)s%(class)snamecaseinsensitive'), models.CheckConstraint(check=models.Q(namelengthgt=0), name='%(applabel)s%(class)snamenotempty')] (snip) This is the admin configuration: class TagAdmin(admin.ModelAdmin): listdisplay = ('pk', 'user', 'name', 'slug') listdisplaylinks = ['pk'] fields = ('user', 'name',) listfilter = ('useremail',) search_fields = ('name',)

return [             constraint             for constraint in self.constraints            if isinstance(constraint, UniqueConstraint) and constraint.condition is None            if (                isinstance(constraint, UniqueConstraint) and                constraint.condition is None and                not constraint.contains_expressions            )         ]      @cached_property


Question: Use simplified paths for deconstruct of expressions Description Previously F() deconstructed to: (). But since it can also be imported from , PR changed it to deconstruct to (). This simplifies generated migration code where it will be referenced only as from import models / models.F(). As Mariusz pointed out on the PR, the same technique can be applied to other expressions, further simplifying generated migrations.

return self  <ins>@deconstructible(path='django.db.models.Func')</ins> class Func(SQLiteNumericMixin, Expression):     """An SQL function call."""     function = None


Question: Use simplified paths for deconstruct of expressions Description Previously F() deconstructed to: (). But since it can also be imported from , PR changed it to deconstruct to (). This simplifies generated migration code where it will be referenced only as from import models / models.F(). As Mariusz pointed out on the PR, the same technique can be applied to other expressions, further simplifying generated migrations.

return copy  <ins>@deconstructible(path='django.db.models.Value')</ins> class Value(SQLiteNumericMixin, Expression):     """Represent a wrapped value as a node within an expression."""     # Provide a default value for `for_save` in order to allow unresolved


Question: Use simplified paths for deconstruct of expressions Description Previously F() deconstructed to: (). But since it can also be imported from , PR changed it to deconstruct to (). This simplifies generated migration code where it will be referenced only as from import models / models.F(). As Mariusz pointed out on the PR, the same technique can be applied to other expressions, further simplifying generated migrations.

return super().as_sql(*args, **kwargs)  <ins>@deconstructible(path='django.db.models.ExpressionWrapper')</ins> class ExpressionWrapper(SQLiteNumericMixin, Expression):     """     An expression that can wrap another expression so that it can provide


Question: Use simplified paths for deconstruct of expressions Description Previously F() deconstructed to: (). But since it can also be imported from , PR changed it to deconstruct to (). This simplifies generated migration code where it will be referenced only as from import models / models.F(). As Mariusz pointed out on the PR, the same technique can be applied to other expressions, further simplifying generated migrations.

return "{}({})".format(self.__class__.__name__, self.expression)  <ins>@deconstructible(path='django.db.models.When')</ins> class When(Expression):     template = 'WHEN %(condition)s THEN %(result)s'     # This isn't a complete conditional expression, must be used in Case().


Question: Use simplified paths for deconstruct of expressions Description Previously F() deconstructed to: (). But since it can also be imported from , PR changed it to deconstruct to (). This simplifies generated migration code where it will be referenced only as from import models / models.F(). As Mariusz pointed out on the PR, the same technique can be applied to other expressions, further simplifying generated migrations.

return cols  <ins>@deconstructible(path='django.db.models.Case')</ins> class Case(SQLiteNumericMixin, Expression):     """     An SQL searched CASE expression:


Question: Use simplified paths for deconstruct of expressions Description Previously F() deconstructed to: (). But since it can also be imported from , PR changed it to deconstruct to (). This simplifies generated migration code where it will be referenced only as from import models / models.F(). As Mariusz pointed out on the PR, the same technique can be applied to other expressions, further simplifying generated migrations.

return sql, params  <ins>@deconstructible(path='django.db.models.OrderBy')</ins> class OrderBy(Expression):     template = '%(expression)s %(ordering)s'     conditional = False


Question: DatabaseCache backend doesn't quote all fields in queries Description Snowflake requires all fields to be quoted, otherwise they're treated as uppercase. The attached patch works with stable/3.2.x, but I'll have to review it once django-snowflake development is caught up to Django's main branch.

with connection.cursor() as cursor:             cursor.execute(                'SELECT %s FROM %s WHERE %s = %%s and expires > %%s' % (                'SELECT %s FROM %s WHERE %s = %%s and %s > %%s' % (                     quote_name('cache_key'),                     quote_name(self._table),                     quote_name('cache_key'),                    quote_name('expires'),                 ),                 [key, connection.ops.adapt_datetimefield_value(now)]             )