Question: def parse_number(cls, name):         """         Given a migration name, try to extract a number from the beginning of        it. If no number is found, return None.        it. For a squashed migration such as '0001_squashed_0004…', return the        second number. If no number is found, return None.         """        if squashed_match := re.search(r'.*_squashed_(\d+)', name):            return int(squashed_match[1])         match = re.match(r'^\d+', name)         if match:             return int(match[0])

makemigrations generates "wrong" numbered migration file if squashed migrations are in place Description When an app has migrations 0001initial and 0002auto201412021234 that are squashed to 0001squashed0002auto201412021234, a new call to makemigrations will generate a migration file called 0002auto201412022345 instead of 0003auto201412022345 which is quite irritating as long as 0002auto201412021234 is still around. It does make sense though when only 0001squashed0002auto20141202_1234 is left. Although the latter case eventually hits every project, I'd prefer the former.


Question: old_field_dec[0:2] == field_dec[0:2] and                             dict(old_field_dec[2], db_column=old_db_column) == field_dec[2])):                         if self.questioner.ask_rename(model_name, rem_field_name, field_name, field):<ins>                            # A db_column mismatch requires a prior noop                            # AlterField for the subsequent RenameField to be a                            # noop on attempts at preserving the old name.                            if old_field.db_column != field.db_column:                                altered_field = field.clone()                                altered_field.name = rem_field_name                                self.add_operation(                                    app_label,                                    operations.AlterField(                                        model_name=model_name,                                        name=rem_field_name,                                        field=altered_field,                                    ),                                )</ins>                             self.add_operation(                                 app_label,                                 operations.RenameField(

Renaming field and providing prior field name to dbcolumn should be an SQL noop Description (last modified by Jacob Walls) Renaming a field and setting the prior implicit field name as the dbcolumn to avoid db operations creates a migration emitting unnecessary SQL. Similar to , which handled a very similar scenario but where there is no field rename, I would expect a SQL noop. I tested with SQLite and MySQL 5.7.31. class Apple(models.Model): core = models.BooleanField() class Apple(models.Model): corerenamed = models.BooleanField(dbcolumn='core') Was renamed to apple.corerenamed (a BooleanField)? [y/N] y Migrations for 'renamez': Rename field core on apple to corerenamed Alter field corerenamed on apple python sqlmigrate renamez 0002 showing unnecessary SQL: BEGIN; -- Rename field core on apple to corerenamed ALTER TABLE "renamezapple" RENAME COLUMN "core" TO "corerenamed"; -- Alter field corerenamed on apple ALTER TABLE "renamezapple" RENAME COLUMN "corerenamed" TO "core"; COMMIT; Without renaming the field, follow the same flow and get an AlterField migration without SQL, which is what intended: BEGIN; -- Alter field core on apple COMMIT;


Question: new_field.remote_field.through = old_field.remote_field.through             old_field_dec = self.deep_deconstruct(old_field)             new_field_dec = self.deep_deconstruct(new_field)            if old_field_dec != new_field_dec:            # If the field was confirmed to be renamed it means that only            # db_column was allowed to change which generate_renamed_fields()            # already accounts for by adding an AlterField operation.            if old_field_dec != new_field_dec and old_field_name == field_name:                 both_m2m = old_field.many_to_many and new_field.many_to_many                 neither_m2m = not old_field.many_to_many and not new_field.many_to_many                 if both_m2m or neither_m2m:diff --git a/django/db/migrations/operations/fields.py b/django/db/migrations/operations/fields.py-- a/django/db/migrations/operations/fields.py++ b/django/db/migrations/operations/fields.py


Question: def reduce(self, operation, app_label):         if isinstance(operation, RemoveField) and self.is_same_field_operation(operation):             return [operation]        elif isinstance(operation, RenameField) and self.is_same_field_operation(operation):        elif (            isinstance(operation, RenameField) and            self.is_same_field_operation(operation) and            self.field.db_column is None        ):             return [                 operation,                 AlterField(


Question: def rename_table_references(self, old_table, new_table):         if self.table != old_table:             return        expressions = deepcopy(self.expressions)        self.columns = []        for col in self.compiler.query._gen_cols([expressions]):            col.alias = new_table        self.expressions = expressions        self.expressions = self.expressions.relabeled_clone({old_table: new_table})         super().rename_table_references(old_table, new_table)      def rename_column_references(self, table, old_column, new_column):

Remaking table with unique constraint crashes on SQLite. Description In Django 4.0a1, this model: class Tag(models.Model): name = models.SlugField(helptext="The tag key.") value = models.CharField(maxlength=150, helptext="The tag value.") class Meta: ordering = ["name", "value"] constraints = [ models.UniqueConstraint( "name", "value", name="uniquenamevalue", ) ] def str(self): return f"{}={}" with these migrations, using sqlite: class Migration(migrations.Migration): initial = True dependencies = [ ] operations = [ migrations.CreateModel( name='Tag', fields=[ ('id', models.BigAutoField(autocreated=True, primarykey=True, serialize=False, verbosename='ID')), ('name', models.SlugField(helptext='The tag key.')), ('value', models.CharField(helptext='The tag value.', maxlength=200)), ], options={ 'ordering': ['name', 'value'], }, ), migrations.AddConstraint( modelname='tag', constraint=models.UniqueConstraint(('name'), ('value'), name='uniquenamevalue'), ), ] class Migration(migrations.Migration): dependencies = [ ('myapp', '0001initial'), ] operations = [ migrations.AlterField( modelname='tag', name='value', field=models.CharField(helptext='The tag value.', maxlength=150), ), ] raises this error: migrate Operations to perform: Apply all migrations: admin, auth, contenttypes, myapp, sessions Running migrations: Applying Traceback (most recent call last): File "D:\Projects\Development\sqliteerror.venv\lib\site-packages\django\db\backends\", line 84, in execute return (sql, params) File "D:\Projects\Development\sqliteerror.venv\lib\site-packages\django\db\backends\sqlite3\", line 416, in execute return Database.Cursor.execute(self, query, params) sqlite3.OperationalError: the "." operator prohibited in index expressions The above exception was the direct cause of the following exception: Traceback (most recent call last): File "D:\Projects\Development\sqliteerror.venv\lib\site-packages\django\core\management\", line 373, in runfromargv self.execute(args, cmdoptions) File "D:\Projects\Development\sqliteerror.


Question: def rename_table_references(self, old_table, new_table):         if self.table != old_table:             return        expressions = deepcopy(self.expressions)        self.columns = []        for col in self.compiler.query._gen_cols([expressions]):            col.alias = new_table        self.expressions = expressions        self.expressions = self.expressions.relabeled_clone({old_table: new_table})         super().rename_table_references(old_table, new_table)      def rename_column_references(self, table, old_column, new_column):

venv\lib\site-packages\django\core\management\", line 417, in execute output = (args, options) File "D:\Projects\Development\sqliteerror.venv\lib\site-packages\django\core\management\", line 90, in wrapped res = handlefunc(args, *kwargs) File "D:\Projects\Development\sqliteerror.venv\lib\site-packages\django\core\management\commands\", line 253, in handle postmigratestate = executor.migrate( File "D:\Projects\Development\sqliteerror.venv\lib\site-packages\django\db\migrations\", line 126, in migrate state = self.migrateallforwards(state, plan, fullplan, fake=fake, fakeinitial=fakeinitial) File "D:\Projects\Development\sqliteerror.venv\lib\site-packages\django\db\migrations\", line 156, in migrateallforwards state = self.applymigration(state, migration, fake=fake, fakeinitial=fakeinitial) File "D:\Projects\Development\sqliteerror.venv\lib\site-packages\django\db\migrations\", line 236, in applymigration state = (state, schemaeditor) File "D:\Projects\Development\sqliteerror.venv\lib\site-packages\django\db\migrations\", line 125, in apply operation.databaseforwards(self.applabel, schemaeditor, oldstate, projectstate) File "D:\Projects\Development\sqliteerror.venv\lib\site-packages\django\db\migrations\operations\", line 225, in databaseforwards schemaeditor.alterfield(frommodel, fromfield, tofield) File "D:\Projects\Development\sqliteerror.venv\lib\site-packages\django\db\backends\sqlite3\", line 140, in alterfield super().alterfield(model, oldfield, newfield, strict=strict) File "D:\Projects\Development\sqliteerror.venv\lib\site-packages\django\db\backends\base\", line 618, in alterfield self.alterfield(model, oldfield, newfield, oldtype, newtype, File "D:\Projects\Development\sqliteerror.venv\lib\site-packages\django\db\backends\sqlite3\", line 362, in alterfield self.remaketable(model,


Question: def rename_table_references(self, old_table, new_table):         if self.table != old_table:             return        expressions = deepcopy(self.expressions)        self.columns = []        for col in self.compiler.query._gen_cols([expressions]):            col.alias = new_table        self.expressions = expressions        self.expressions = self.expressions.relabeled_clone({old_table: new_table})         super().rename_table_references(old_table, new_table)      def rename_column_references(self, table, old_column, new_column):

alterfield=(oldfield, newfield)) File "D:\Projects\Development\sqliteerror.venv\lib\site-packages\django\db\backends\sqlite3\", line 303, in remaketable self.execute(sql) File "D:\Projects\Development\sqliteerror.venv\lib\site-packages\django\db\backends\base\", line 151, in execute cursor.execute(sql, params) File "D:\Projects\Development\sqliteerror.venv\lib\site-packages\django\db\backends\", line 98, in execute return super().execute(sql, params) File "D:\Projects\Development\sqliteerror.venv\lib\site-packages\django\db\backends\", line 66, in execute return self.executewithwrappers(sql, params, many=False, executor=self.execute) File "D:\Projects\Development\sqliteerror.venv\lib\site-packages\django\db\backends\", line 75, in executewithwrappers return executor(sql, params, many, context) File "D:\Projects\Development\sqliteerror.venv\lib\site-packages\django\db\backends\", line 84, in execute return (sql, params) File "D:\Projects\Development\sqliteerror.venv\lib\site-packages\django\db\", line 90, in exit raise djexcvalue.withtraceback(traceback) from excvalue File "D:\Projects\Development\sqliteerror.venv\lib\site-packages\django\db\backends\", line 84, in execute return (sql, params) File "D:\Projects\Development\sqliteerror.venv\lib\site-packages\django\db\backends\sqlite3\", line 416, in execute return Database.Cursor.execute(self, query, params) the "." operator prohibited in index expressions


Question: new_model = to_state.apps.get_model(app_label, self.new_name)         if self.allow_migrate_model(schema_editor.connection.alias, new_model):             old_model = from_state.apps.get_model(app_label, self.old_name)            old_db_table = old_model._meta.db_table            new_db_table = new_model._meta.db_table            # Don't alter when a table name is not changed.            if old_db_table == new_db_table:                return             # Move the main table            schema_editor.alter_db_table(                new_model,                old_model._meta.db_table,                new_model._meta.db_table,            )            schema_editor.alter_db_table(new_model, old_db_table, new_db_table)             # Alter the fields pointing to us             for related_object in old_model._meta.related_objects:                 if related_object.related_model == old_model:

RenameModel with dbtable should be a noop. Description A RenameModel operation that already has dbtable defined must be a noop. In Postgres, it drops and recreates foreign key constraints. In sqlite it recreates the table (as expected for a table renaming).


Question: opt.dest in options and             (opt.required or opt in mutually_exclusive_required_options)         ):<ins>            opt_dest_count = sum(v == opt.dest for v in opt_mapping.values())            if opt_dest_count > 1:                raise TypeError(                    f'Cannot pass the dest {opt.dest!r} that matches multiple '                    f'arguments via **options.'                )</ins>             parse_args.append(min(opt.option_strings))             if isinstance(opt, (_AppendConstAction, _CountAction, _StoreConstAction)):                 continue

callcommand() fails when required mutually exclusive arguments use the same . Description I have a command which accepts two different ways to specify a time -- either as a timestamp or as a duration in the future: pause (--for duration | --until time) class Command(BaseCommand): def addarguments(self, parser) -None: group = parser.addmutuallyexclusivegroup(required=True) group.addargument('--for', dest='until', action='store', type=parsedurationtotime) group.addargument('--until', action='store', type=parsetime) def handle(self, until: datetime, **): pass This works fine on the command line, however there doesn't seem to be a way to make this work through callcommand. Specifically there are two sides to the failure: while I can provide an until value (as a string, which is processed by parsetime) there is no mechanism to pass a for value if that's how I want to spell the input the for value is always required and attempts to parse the (string) until value passed, which then errors since the input formats are very different


Question: if search_fields and search_term:             orm_lookups = [construct_search(str(search_field))                            for search_field in search_fields]<ins>            term_queries = []</ins>             for bit in smart_split(search_term):                 if bit.startswith(('"', "'")) and bit[0] == bit[-1]:                     bit = unescape_string_literal(bit)

Unnecessary joins in admin changelist query Description Django 1.2.5 Models: class Client(models.Model): name = models.CharField(('name'), maxlength=256) name2 = models.CharField(('unofficial or obsolete name'), maxlength=256, blank=True, null=True) contactperson = models.CharField(('contact person'), maxlength=256, blank=True, null=True) ... class ClientOffice(models.Model): name = models.CharField(('name'), maxlength=256) name2 = models.CharField(('unofficial or obsolete name'), maxlength=256, blank=True, null=True) ... client = models.ForeignKey(Client, verbosename=('client')) ... and admin options like these: class ClientAdmin(admin.ModelAdmin): searchfields = ('name', 'name2', 'contact_person', 'clientofficename', 'clientofficename2') ... Numbers: () () Now, if we try searching for clients in admin by a search query containig several words (3), got django/admin stalled. The problem is going to be that each word in the search query leads to additional JOIN in final SQL query beacause of qs = (...) pattern. The attached patch is for Django 1.2.5, but adopting for the current SVN trunk is trivial.


Question: *((orm_lookup, bit) for orm_lookup in orm_lookups),                     _connector=models.Q.OR,                 )                queryset = queryset.filter(or_queries)                term_queries.append(or_queries)            queryset = queryset.filter(models.Q(*term_queries))             may_have_duplicates |= any(                 lookup_spawns_duplicates(self.opts, search_spec)                 for search_spec in orm_lookups


Question: class FieldListFilter(ListFilter):     _field_list_filters = []     _take_priority_index = 0<ins>    list_separator = ','</ins>      def __init__(self, field, request, params, model, model_admin, field_path):         self.field = field

Django admin allows filtering using the field lookups such as "in", but it is impossible to include a value that contains a comma Description The admin site allows you to filter the queryset in the changelist in a plenty of different ways. Notably, it allows you to filter the records by multiple values (if the field's value is one of the specified value options, then such record is considered matching). For example, you can test it with a query string like this: /admin/auth/user/?username_in=johnny,viola,gordon Unfortunately, there is a big limitation at the moment: you can't include a value option that contains a comma (or a few). The function that splits the string is preparelookup_value, found in


Question: for p in self.expected_parameters():             if p in params:                 value = params.pop(p)                self.used_parameters[p] = prepare_lookup_value(p, value)                self.used_parameters[p] = prepare_lookup_value(p, value, self.list_separator)      def has_output(self):         return Truediff --git a/django/contrib/admin/utils.py b/django/contrib/admin/utils.py-- a/django/contrib/admin/utils.py++ b/django/contrib/admin/utils.py


Question: return False  def prepare_lookup_value(key, value):def prepare_lookup_value(key, value, separator=','):     """     Return a lookup value prepared to be used in queryset filtering.     """     # if key ends with __in, split parameter into separate values     if key.endswith('__in'):        value = value.split(',')        value = value.split(separator)     # if key ends with __isnull, special case '' and the string literals 'false' and '0'     elif key.endswith('__isnull'):         value = value.lower() not in ('', 'false', '0')


Question: extra_params['unique'] = True                      if is_relation:                        ref_db_column, ref_db_table = relations[column_name]                         if extra_params.pop('unique', False) or extra_params.get('primary_key'):                             rel_type = 'OneToOneField'                         else:                             rel_type = 'ForeignKey'                            ref_pk_column = connection.introspection.get_primary_key_column(cursor, ref_db_table)                            if ref_pk_column and ref_pk_column != ref_db_column:                                extra_params['to_field'] = ref_db_column                         rel_to = (                            "self" if relations[column_name][1] == table_name                            else table2model(relations[column_name][1])                            'self' if ref_db_table == table_name                            else table2model(ref_db_table)                         )                         if rel_to in known_models:                             field_type = '%s(%s' % (rel_type, rel_to)

Foreign key to a specific field is not handled in inspectdb Description (last modified by Tim Graham) if you have a DB like that CREATE TABLE foo ( id serial primary key, otherid int UNIQUE); CREATE TABLE bar ( id serial primary key, otherid int, constraint myconst FOREIGN KEY(otherid) references foo(otherid) ); the generated model for the bar table will have the otherid be a FK to foo and not foo(otherid). I'm attaching a potential fix for this. Sorry I had no time for the UTs.


Question: <ins>import os</ins> from django.apps import apps from django.conf import settings from django.core.management.base import BaseCommand, CommandError

--squashed-name can overwrite existing migration files Description squashmigrations --squashed-name initial myapp 0002 will overwrite If that migration has already been applied anywhere, the code and the database will be in an inconsistent state. Furthermore, the 0002... migration likely depends on 0001initial, so the new replaces = …0002… line to introduces a CircularDependencyError. squashmigrations should exit with an error instead of overwriting existing migration files. --squashed-name can overwrite existing migration files Description squashmigrations --squashed-name initial myapp 0002 will overwrite If that migration has already been applied anywhere, the code and the database will be in an inconsistent state. Furthermore, the 0002... migration likely depends on 0001initial, so the new replaces = …0002… line to introduces a CircularDependencyError. squashmigrations should exit with an error instead of overwriting existing migration files.


Question: # Write out the new migration file         writer = MigrationWriter(new_migration, include_header)<ins>        if os.path.exists(writer.path):            raise CommandError(                f'Migration {new_migration.name} already exists. Use a different name.'            )</ins>         with open(writer.path, "w", encoding='utf-8') as fh:             fh.write(writer.as_string())


Question: self.page_timeout = None         self.key_prefix = settings.CACHE_MIDDLEWARE_KEY_PREFIX         self.cache_alias = settings.CACHE_MIDDLEWARE_ALIAS        self.cache = caches[self.cache_alias]    @property    def cache(self):        return caches[self.cache_alias]      def _should_update_cache(self, request, response):         return hasattr(request, '_cache_update_cache') and request._cache_update_cache

CacheMiddleware and FetchFromCacheMiddleware are not thread safe. Description CacheMiddleware persist = caches[cachealias] on startup and it is not thread safe.  I found that after some production errors with pylibmc and uwsgi threaded. Created a small project to reproduce it. Nothing fancy, just pylibmc cache and a cached view. It fails even with development server, with concurrent requests. Traceback (most recent call last): File "versions/pylibmcbug/lib/python3.9/site-", line 47, in inner response = getresponse(request) File "versions/pylibmcbug/lib/python3.9/site-", line 181, in getresponse response = wrappedcallback(request, callbackargs, *callbackkwargs) File "versions/pylibmcbug/lib/python3.9/site-", line 122, in wrappedview result = middleware.processrequest(request) File "versions/pylibmcbug/lib/python3.9/site-", line 145, in processrequest cachekey = getcachekey(request, self.keyprefix, 'GET', ) File "versions/pylibmcbug/lib/python3.9/site-", line 362, in getcachekey headerlist = (cachekey) File "versions/pylibmcbug/lib/python3.9/site-", line 77, in get return (key, default) pylibmc.ConnectionError: error 3 from memcachedget(): (0x7f290400bd60) FAILURE, poll() returned a value that was not dealt with, host: localhost: - Looking for git history, it is this way since 2010.


Question: super().__init__(get_response)         self.key_prefix = settings.CACHE_MIDDLEWARE_KEY_PREFIX         self.cache_alias = settings.CACHE_MIDDLEWARE_ALIAS        self.cache = caches[self.cache_alias]    @property    def cache(self):        return caches[self.cache_alias]      def process_request(self, request):         """


Question: if cache_alias is None:                 cache_alias = DEFAULT_CACHE_ALIAS             self.cache_alias = cache_alias<del>            self.cache = caches[self.cache_alias]</del>         except KeyError:             pass


Question: sql, sql_params = super().as_sql(compiler, connection, ordering=(                 'ORDER BY ' + ', '.join(ordering_expr_sql)             ))            return sql, sql_params + ordering_params            return sql, (*sql_params, *ordering_params)         return super().as_sql(compiler, connection, ordering='')      def set_source_expressions(self, exprs):diff --git a/django/db/models/aggregates.py b/django/db/models/aggregates.py-- a/django/db/models/aggregates.py++ b/django/db/models/aggregates.py

Aggregate filtered by an Exists subquery crashes Description For example: ("publisher").aggregate( maxrating=Max( "rating", filter=Exists( (book=OuterRef("pk")), ), ) Will crash with the following traceback: Traceback (most recent call last): File "", line 146, in testfilteredaggregatewithexists aggregate = ('publisher').aggregate( File "", line 405, in aggregate return query.getaggregation(, kwargs) File "", line 501, in getaggregation result = compiler.executesql(SINGLE) File "", line 1189, in executesql sql, params = () File "", line 531, in assql extraselect, orderby, groupby = self.presqlsetup() File "", line 59, in presqlsetup self.setupquery() File "", line 50, in setupquery , self.klassinfo, self.annotationcolmap = self.getselect() File "", line 267, in getselect sql, params = self.compile(col) File "", line 463, in compile sql, params = (self, self.connection) File "", line 90, in assql return sql, params + filterparams TypeError: can only concatenate list (not "tuple") to list The following patch should fix the issue: diff --git index .. --- +++ -87,7 +87,7 class Aggregate(Func): compiler, connection, template=template, filter=filtersql, extracontext ) return sql, params + filterparams return sql, (params, filterparams) else: copy = () = None


Question: compiler, connection, template=template, filter=filter_sql,                     **extra_context                 )                return sql, params + filter_params                return sql, (*params, *filter_params)             else:                 copy = self.copy()                 copy.filter = None


Question: return context      def id_for_label(self, id_):        if id_:            id_ += '_0'        return id_        return ''      def value_from_datadict(self, data, files, name):         return [

Remove "for = ..." from MultiWidget's <label. Description The instance from Raw MultiWidget class generate idforlabel like f'{id}0' It has not sense. For example ChoiceWidget has self.addidindex and I can decide it myself, how I will see labelid - with or without index. I think, it is better to remove completely idforlabel method from MultiWidget Class.


Question: return self.as_sql(compiler, connection, **extra_context)  class ExpressionWrapper(Expression):class ExpressionWrapper(SQLiteNumericMixin, Expression):     """     An expression that can wrap another expression so that it can provide     extra context to the inner expression, such as the output_field.

Case() and ExpressionWrapper() doesn't work with DecimalField on SQLite. Description I noticed that, on sqlite, some comparisons against DecimalField annotations behave unexpectedly, in particular when wrapping a DecimalField value in a Case/When or ExpressionWrapper. I suspect that there might be some inconsistencies in the type conversions here somehow. I've created a few testcase to illustrate the problem on current git main:  and see below. def test00comparefield(self): """Comparing a Case annotation wrapping a field to a literal works.""" (a='', d=1) try: (dgt=0).get() finally: from import connection print(connection.queries[-1]['sql']) def test01compareannotationvalueliteral(self): """Comparing a literal annotation using Value to a literal works.""" # Fields are not actually used here (a='', d=0) try: Foo.objects.annotate( x=models.Value(1, (maxdigits=1, decimalplaces=0)), ).filter(xgt=0).get() finally: from import connection print(connection.queries[-1]['sql']) def test02compareannotationexpressionwrapperliteral(self): """Comparing a literal annotation using ExpressionWraper and Value to a literal works.""" # Fields are not actually used here (a='', d=0) try: Foo.objects.annotate( x=models.ExpressionWrapper( models.Value(1), (maxdigits=1, decimalplaces=0), ), ).filter(xgt=0).get() finally: from import connection print(connection.queries[-1]['sql']) def test03comparecaseannotation(self): """Comparing a Case annotation wrapping a field to a literal works.""" (a='', d=1) try: Foo.objects.annotate( x=models.Case(models.When(a='', then=models.F('d'))), ).filter(xgt=0).get() finally: from import connection print(connection.queries[-1]['sql']) test00comparefield compares a field directly with a literal, which works. test01compareannotationvalueliteral adds a literal annotation using just Value and then compares it, which also works. test02compareannotationexpressionwrapperliteral adds a literal annotation using Value wrapped in ExpressionWrapper, which does not work becomes a literal int, rather than a string like the compared value. test03comparecaseannotation wraps the field in a case/when and then compares it, which also does not work (maybe the CASE changes the type?)


Question: return self.as_sql(compiler, connection, **extra_context)  class ExpressionWrapper(Expression):class ExpressionWrapper(SQLiteNumericMixin, Expression):     """     An expression that can wrap another expression so that it can provide     extra context to the inner expression, such as the output_field.

Running these testcases against sqlite gives: SELECT "modelfieldsfoo"."id", "modelfieldsfoo"."a", "modelfieldsfoo"."d" FROM "modelfieldsfoo" WHERE "modelfieldsfoo"."d" '0' LIMIT 21 .SELECT "modelfieldsfoo"."id", "modelfieldsfoo"."a", "modelfieldsfoo"."d", CAST('1' AS NUMERIC) AS "x" FROM "modelfieldsfoo" WHERE CAST('1' AS NUMERIC) '0' LIMIT 21 .SELECT "modelfieldsfoo"."id", "modelfieldsfoo"."a", "modelfieldsfoo"."d", 1 AS "x" FROM "modelfieldsfoo" WHERE 1 '0' LIMIT 21 ESELECT "modelfieldsfoo"."id", "modelfieldsfoo"."a", "modelfieldsfoo"."d", CASE WHEN "modelfieldsfoo"."a" = '' THEN "modelfieldsfoo"."d" ELSE NULL END AS "x" FROM "modelfieldsfoo" WHERE CASE WHEN ("modelfieldsfoo"."a" = '') THEN "modelfieldsfoo"."d" ELSE NULL END '0' LIMIT 21 E.s........... ERROR: test02compareannotationexpressionwrapperliteral (modelfields.testdecimalfield.DecimalFieldTests) Comparing a literal annotation using ExpressionWraper and Value to a literal works. Traceback (most recent call last): File "", line 437, in inner return func(args, kwargs) File "", line 142, in test02compareannotationexpressionwrapperliteral Foo.objects.annotate( File "", line 441, in get raise ( Foo matching query does not exist. ERROR: test03comparecaseannotation (modelfields.testdecimalfield.DecimalFieldTests) Comparing a Case annotation wrapping a field to a literal works. Traceback (most recent call last): File "", line 437, in inner return func(args, kwargs) File "", line 154, in test03comparecaseannotation Foo.objects.annotate( File "", line 441, in get raise ( Foo matching query does not exist. Note in the printed queries that the 0 value that is compared with is a string in the query ('0'), while the ExpressionWrappered value is just a plain number (1). The Value without ExpressionWrapper has a cast to NUMERIC that I suspect makes that case work?


Question: return self.as_sql(compiler, connection, **extra_context)  class ExpressionWrapper(Expression):class ExpressionWrapper(SQLiteNumericMixin, Expression):     """     An expression that can wrap another expression so that it can provide     extra context to the inner expression, such as the output_field.

Also note that the DEBUG stuff and try-catch is only to capture the query that is being generated, it is not needed to trigger the problem. I tried printing the attribute first, but that seems to hide the quotes around the literal 0 in the query, so took me a while to realize what was happening. The numbers in the testcase names are to ensure the SQL queries are printed in a recognizable order. All four testcases work on MySQL, there the 0 value is written in the query without the quotes, and the cast to NUMERIC is also missing. I suspect this issue is highly similar to , and can a fix can probably build on the fix for that issue.


Question: return cols  class Case(Expression):class Case(SQLiteNumericMixin, Expression):     """     An SQL searched CASE expression:


Question: return cols  class Case(Expression):class Case(SQLiteNumericMixin, Expression):     """     An SQL searched CASE expression:


Question: return cols  class Case(Expression):class Case(SQLiteNumericMixin, Expression):     """     An SQL searched CASE expression:


Question: import re from collections import namedtuple  import sqlparse from django.db import DatabaseError from django.db.backends.base.introspection import (     BaseDatabaseIntrospection, FieldInfo as BaseFieldInfo, TableInfo, )

SQLite schema introspection should use PRAGMA instead of parsing to retrieve foreign keys Description The SQLite backend's DatabaseIntrospection.getrelations uses a complex combination of regexes to extract foreign key constraints from DDL while PRAGMA foreignkeylist works just fine as getforeignkey_constraints.


Question: """         cursor.execute('PRAGMA table_info(%s)' % self.connection.ops.quote_name(table_name))         table_info = cursor.fetchall()<ins>        if not table_info:            raise DatabaseError(f'Table {table_name} does not exist (empty pragma).')</ins>         collations = self._get_column_collations(cursor, table_name)         json_columns = set()         if self.connection.features.can_introspect_json_field:


Question: def get_relations(self, cursor, table_name):         """        Return a dictionary of {field_name: (field_name_other_table, other_table)}        Return a dictionary of {column_name: (ref_column_name, ref_table_name)}         representing all foreign keys in the given table.         """        # Dictionary of relations to return        relations = {}        # Schema for this table         cursor.execute(            "SELECT sql, type FROM sqlite_master "            "WHERE tbl_name = %s AND type IN ('table', 'view')",            [table_name]            'PRAGMA foreign_key_list(%s)' % self.connection.ops.quote_name(table_name)         )        create_sql, table_type = cursor.fetchone()        if table_type == 'view':            # It might be a view, then no results will be returned            return relations        results = create_sql[create_sql.index('(') + 1:create_sql.rindex(')')]        # Walk through and look for references to other tables. SQLite doesn't        # really have enforced references, but since it echoes out the SQL used        # to create the table we can look for REFERENCES statements used there.        for field_desc in results.split(','):            field_desc = field_desc.strip()            if field_desc.startswith("UNIQUE"):                continue            m = re.search(r'references (\S*) ?\(["|]?(.*)["|]?\)', field_desc, re.I)            if not m:                continue            table, column = [s.strip('"') for s in m.groups()]            if field_desc.startswith("FOREIGN KEY"):                # Find name of the target FK field                m = re.match(r'FOREIGN KEY\s*\(([^\)]*)\).*', field_desc, re.I)                field_name = m[1].strip('"')            else:                field_name = field_desc.split()[0].strip('"')            cursor.execute("SELECT sql FROM sqlite_master WHERE tbl_name = %s", [table])            result = cursor.fetchall()[0]            other_table_results = result[0].strip()            li, ri = other_table_results.index('('), other_table_results.rindex(')')            other_table_results = other_table_results[li + 1:ri]            for other_desc in other_table_results.split(','):                other_desc = other_desc.strip()                if other_desc.startswith('UNIQUE'):                    continue                other_name = other_desc.split(' ', 1)[0].strip('"')                if other_name == column:                    relations[field_name] = (other_name, table)                    break        return relations        return {            column_name: (ref_column_name, ref_table_name)            for _, _, ref_table_name, column_name, ref_column_name, *_ in cursor.fetchall()        }      def get_primary_key_column(self, cursor, table_name):         """Return the column name of the primary key for the given table."""


Question: return name         return None <del>    def _get_foreign_key_constraints(self, cursor, table_name):        constraints = {}        cursor.execute('PRAGMA foreign_key_list(%s)' % self.connection.ops.quote_name(table_name))        for row in cursor.fetchall():            # Remaining on_update/on_delete/match values are of no interest.            id_, _, table, from_, to = row[:5]            constraints['fk_%d' % id_] = {                'columns': [from_],                'primary_key': False,                'unique': False,                'foreign_key': (table, to),                'check': False,                'index': False,            }        return constraints</del>     def _parse_column_or_constraint_definition(self, tokens, columns):         token = None         is_constraint_definition = None


Question: "check": False,                 "index": False,             }        constraints.update(self._get_foreign_key_constraints(cursor, table_name))        relations = enumerate(self.get_relations(cursor, table_name).items())        constraints.update({            f'fk_{index}': {                'columns': [column_name],                'primary_key': False,                'unique': False,                'foreign_key': (ref_table_name, ref_column_name),                'check': False,                'index': False,            }            for index, (column_name, (ref_column_name, ref_table_name)) in relations        })         return constraints      def _get_index_columns_orders(self, sql):diff --git a/django/db/backends/sqlite3/schema.py b/django/db/backends/sqlite3/schema.py-- a/django/db/backends/sqlite3/schema.py++ b/django/db/backends/sqlite3/schema.py


Question: for other_table in self.connection.introspection.get_table_list(cursor):                 if ignore_self and other_table.name == table_name:                     continue                constraints = self.connection.introspection._get_foreign_key_constraints(cursor, other_table.name)                for constraint in constraints.values():                    constraint_table, constraint_column = constraint['foreign_key']                relations = self.connection.introspection.get_relations(cursor, other_table.name)                for constraint_column, constraint_table in relations.values():                     if (constraint_table == table_name and                             (column_name is None or constraint_column == column_name)):                         return True


Question: re.IGNORECASE ) language_code_prefix_re = _lazy_re_compile(r'^/(\w+([@-]\w+)?)(/|$)')language_code_prefix_re = _lazy_re_compile(r'^/(\w+([@-]\w+){0,2})(/|$)')   @receiver(setting_changed)

Internationalisation didn't support language locale containing both script and region. Description The i18npatterns didn't work with locale contains both script and region, like en-latn-us. Given LANGUAGECODE = 'en-us' LANGUAGES = [ ('en-us', "English"), ('en-latn-us', "Latin English"), ('en-Latn-US', "BCP 47 case format"), ] from import i18npatterns from import HttpResponse def bangiah(request): return HttpResponse('U!') urlpatterns += i18npatterns( path('', bangiah), ) The response of http://localhost:8000/en-us/ is 200 U!. The response of http://localhost:8000/en-lat-us/ is 404 not found. The response of http://localhost:8000/en-Latn-US/ is 404 not found. Steps to Reproduce Start a new project with django-admin startproject tshi and cd tshi/ Append to as follows LANGUAGES = [ ('en-us', "English"), ('en-latn-us', "Latin English"), ('en-Latn-US', "BCP 47 case format"), ] MIDDLEWARE += [ '', ] Edit by appending follows from import i18npatterns from import HttpResponse def bangiah(request): return HttpResponse('U!') urlpatterns += i18npatterns( path('', bangiah), ) python migrate python runserver The results The response of http://localhost:8000/en-us/ is 200 U!. The response of http://localhost:8000/en-lat-us/ is 404 not found. The response of http://localhost:8000/en-Latn-US/ is 404 not found. Expect to happen instead The response of http://localhost:8000/en-latn-us/ and http://localhost:8000/en-Latn-US/ should be 200 U!. The en-Latn-US tag follows format defined in RFC 5646. It's documented that the language part is always in lowercase, following Accept-Language. Accept-Language is following Content-Language Header, which is following RFC 5646.


Question: re.IGNORECASE ) language_code_prefix_re = _lazy_re_compile(r'^/(\w+([@-]\w+)?)(/|$)')language_code_prefix_re = _lazy_re_compile(r'^/(\w+([@-]\w+){0,2})(/|$)')   @receiver(setting_changed)

The RFC 5646 defined langtag as follow: langtag = language ["-" script] ["-" region] ("-" variant) ("-" extension) ["-" privateuse] language = 23ALPHA ; shortest ISO 639 code ["-" extlang] ; sometimes followed by ; extended language subtags / 4ALPHA ; or reserved for future use / 58ALPHA ; or registered language subtag extlang = 3ALPHA ; selected ISO 639 codes *2("-" 3ALPHA) ; permanently reserved script = 4ALPHA ; ISO code region = 2ALPHA ; ISO 3166-1 code / 3DIGIT ; UN M.49 code I have confirmed that this issue can be reproduced as described on a fresh Django project Python version: 3.7.5 Django version: 3.2.7


Question: if self.verbosity >= 2:                     self.stdout.write('Creating %s' % new_path)                 try:                    shutil.copymode(old_path, new_path)                    self.apply_umask(old_path, new_path)                     self.make_writeable(new_path)                 except OSError:                     self.stderr.write(

startproject should honor umask Description Ticket fixed the problem of not being made executable by copying all the permission bits (not just the executable flags). This means that the user's umask doesn't work, e.g.: $ umask 077 $ touch foo $ ls -l foo -rw------- 1 talex talex 0 2007-05-12 13:27 foo $ PYTHONPATH=trunk ./trunk/django/bin/django- startproject mysite $ ls -l -rw-r--r-- 1 talex talex 2804 2007-05-12 13:28 I discovered this whilst trying to make a Zero Install package for Django. Everything in the Zero Install cache is read-only, so startproject fails with: File "", line 799, in startproject fp = open(mainsettingsfile, 'w') IOError: [Errno 13] Permission denied: '' Thanks,


Question: scheme = template.split(':', 1)[0].lower()         return scheme in self.url_schemes <ins>    def apply_umask(self, old_path, new_path):        current_umask = os.umask(0)        os.umask(current_umask)        current_mode = stat.S_IMODE(os.stat(old_path).st_mode)        os.chmod(new_path, current_mode & ~current_umask)</ins>     def make_writeable(self, filename):         """         Make sure that the file is writeable.


Question: @register.filter(is_safe=True)def json_script(value, element_id):def json_script(value, element_id=None):     """     Output value JSON-encoded, wrapped in a

Make the elementid argument of jsonscript optional Description I recently had a use-case where I wanted to use json_script but I didn't need any id for it (I was including the <scriptinside a <templateso I didn't need an id to refer to it). I can't see any reason (security or otherwise) for the id to be required and making it optional doesn't seem to break any tests.


Question: }  def json_script(value, element_id):def json_script(value, element_id=None):     """     Escape all the HTML/XML special characters with their unicode escapes, so     value is safe to be output anywhere except for inside a tag attribute. Wrap


Question: """     from django.core.serializers.json import DjangoJSONEncoder     json_str = json.dumps(value, cls=DjangoJSONEncoder).translate(_json_script_escapes)    return format_html(        '',        element_id, mark_safe(json_str)    )    if element_id:        template = ''        args = (element_id, mark_safe(json_str))    else:        template = ''        args = (mark_safe(json_str),)    return format_html(template, *args)   def conditional_escape(text):


Question: for name, field in sorted(fields.items()):             deconstruction = self.deep_deconstruct(field)             if field.remote_field and field.remote_field.model:                del deconstruction[2]['to']                deconstruction[2].pop('to', None)             fields_def.append(deconstruction)         return fields_def

KeyError with migration autodetector and FK field with hardcoded reference Description Hi, I encountered this issue on an old Django project (probably 10 years old) with tons of models and probably a lot of questionable design decisions. The symptom is that running our test suite in verbose mode doesn't work: $ python test -v 2 Creating test database for alias 'default' ('testproject')... Operations to perform: Synchronize unmigrated apps: [... about 40 apps] Apply all migrations: (none) Synchronizing apps without migrations: Creating tables... Creating table authpermission Creating table authgroup Creating table authuser Creating table djangocontenttype Creating table djangosession Creating table djangoadminlog [... 100 or so more tables] Running deferred SQL... Running migrations: No migrations to apply.


Question: for name, field in sorted(fields.items()):             deconstruction = self.deep_deconstruct(field)             if field.remote_field and field.remote_field.model:                del deconstruction[2]['to']                deconstruction[2].pop('to', None)             fields_def.append(deconstruction)         return fields_def

Traceback (most recent call last): File "", line 17, in <moduleexecutefromcommandline() File "", line 401, in executefromcommandline utility.execute() File "", line 395, in execute self.fetchcommand(subcommand).runfromargv() File "", line 23, in runfromargv super().runfromargv(argv) File "", line 330, in runfromargv self.execute(args, cmdoptions) File "", line 371, in execute output = (args, options) File "", line 53, in handle failures = testrunner.runtests(testlabels) File "", line 697, in runtests oldconfig = self.setupdatabases(aliases=databases) File "", line 618, in setupdatabases self.parallel, kwargs File "", line 174, in setupdatabases serialize=connection.settingsdict['TEST'].get('SERIALIZE', True), File "", line 77, in createtestdb runsyncdb=True, File "", line 168, in callcommand return command.execute(args, defaults) File "", line 371, in execute output = (args, options) File "", line 85, in wrapped res = handlefunc(args, kwargs) File "", line 227, in handle changes = autodetector.changes() File "", line 43, in changes changes = self.detectchanges(convertapps, graph) File "", line 160, in detectchanges self.generaterenamedmodels() File "", line 476, in generaterenamedmodels modelfieldsdef = self.onlyrelationagnosticfields() File "", line 99, in onlyrelationagnosticfields del deconstruction[2]['to'] KeyError: 'to' I finally did some digging and found that the culprit is a custom ForeignKey field that hardcodes its to argument (and thus also removes it from its deconstructed kwargs). It seems that the autodetector doesn't like that.


Question: for name, field in sorted(fields.items()):             deconstruction = self.deep_deconstruct(field)             if field.remote_field and field.remote_field.model:                del deconstruction[2]['to']                deconstruction[2].pop('to', None)             fields_def.append(deconstruction)         return fields_def

Here's a self-contained reproduction test to replicate the issue: from import models from import MigrationAutodetector from import ModelState, ProjectState from import TestCase class CustomFKField(models.ForeignKey): def init(self, args, kwargs): kwargs['to'] = 'testapp.HardcodedModel' super().init(*args, kwargs) def deconstruct(self): name, path, args, kwargs = super().deconstruct() del kwargs["to"] return name, path, args, kwargs class ReproTestCase(TestCase): def testreprodution(self): before = ProjectState() before.addmodel(ModelState('testapp', 'HardcodedModel', [])) after = ProjectState() after.addmodel(ModelState('testapp', 'HardcodedModel', [])) after.addmodel(ModelState('testapp', 'TestModel', [('custom', CustomFKField(ondelete=models.CASCADE))])) changes = MigrationAutodetector(before, after).detectchanges() self.assertEqual(len(changes['testapp']), 1) While I'll happily admit that my custom field's design might be questionable, I don't think it's incorrect and I think the autodetector is at fault here. Changing del deconstruction[2]['to'] to deconstruction[2].pop('to', None) on the line indicated by the traceback makes my test suite run again, in all its glorious verbosity. Seems like an innocent enough fix to me.


Question: from django.db.models import F, OrderByfrom django.db.models.expressions import OrderByList   class OrderableAggMixin:      def __init__(self, *expressions, ordering=(), **extra):        if not isinstance(ordering, (list, tuple)):            ordering = [ordering]        ordering = ordering or []        # Transform minus sign prefixed strings into an OrderBy() expression.        ordering = (            (OrderBy(F(o[1:]), descending=True) if isinstance(o, str) and o[0] == '-' else o)            for o in ordering        )        if isinstance(ordering, (list, tuple)):            self.order_by = OrderByList(*ordering)        else:            self.order_by = OrderByList(ordering)         super().__init__(*expressions, **extra)        self.ordering = self._parse_expressions(*ordering)      def resolve_expression(self, *args, **kwargs):        self.ordering = [expr.resolve_expression(*args, **kwargs) for expr in self.ordering]        self.order_by = self.order_by.resolve_expression(*args, **kwargs)         return super().resolve_expression(*args, **kwargs)     def as_sql(self, compiler, connection):        if self.ordering:            ordering_params = []            ordering_expr_sql = []            for expr in self.ordering:                expr_sql, expr_params = compiler.compile(expr)                ordering_expr_sql.append(expr_sql)                ordering_params.extend(expr_params)            sql, sql_params = super().as_sql(compiler, connection, ordering=(                'ORDER BY ' + ', '.join(ordering_expr_sql)            ))            return sql, (*sql_params, *ordering_params)        return super().as_sql(compiler, connection, ordering='')    def get_source_expressions(self):        return super().get_source_expressions() + [self.order_by]      def set_source_expressions(self, exprs):        # Extract the ordering expressions because ORDER BY clause is handled        # in a custom way.        self.ordering = exprs[self._get_ordering_expressions_index():]        return super().set_source_expressions(exprs[:self._get_ordering_expressions_index()])        *exprs, self.order_by = exprs        return super().set_source_expressions(exprs)     def get_source_expressions(self):        return super().get_source_expressions() + self.ordering    def _get_ordering_expressions_index(self):        """Return the index at which the ordering expressions start."""        source_expressions = self.get_source_expressions()        return len(source_expressions) - len(self.ordering)    def as_sql(self, compiler, connection):        order_by_sql, order_by_params = compiler.compile(self.order_by)        sql, sql_params = super().as_sql(compiler, connection, ordering=order_by_sql)        return sql, (*sql_params, *order_by_params)diff --git a/django/db/models/expressions.py b/django/db/models/expressions.py-- a/django/db/models/expressions.py++ b/django/db/models/expressions.py

Window(orderby) should allow usage of descending string syntax to be used Description (last modified by Simon Charette) The QuerySet.orderby and some aggregates ordering kwarg allows for the leading dash syntax to be used but Window.orderby doesn't as it solely wraps the provided orderby in ExpressionList(expressions=orderby). This makes for an inconsistent API so I suggest we reuse the logic in in As a related note it seems most of the logic of OrderableAggMixin could be simplified by using ExpressionList. It's a shame that we used ordering and not orderby as a kwarg for OrderableAggMixin as it's now inconsistent. Also not sure how much of a public API the OrderBy expression is but I wish it was initially named Sort (or Ordering?) so that we could define class OrderBy(ExpressionList): template = 'ORDER BY %(expressions)s' def init(self, expressions, extra): expressions = [ (Sort(F(expr[1:]), descending=True) if isinstance(expr, str) and expr[0] == '-' else expr) for expr in expressions ] super().init(expressions, *extra) And then simply use this abstraction in Window and Postgres orderable aggregates. Assigning to myself as I plan to have a look at this in next few days.


Question: class ExpressionList(Func):     """     An expression containing multiple expressions. Can be used to provide a    list of expressions as an argument to another expression, like an    ordering clause.    list of expressions as an argument to another expression, like a partition    clause.     """     template = '%(expressions)s'


Question: return self.as_sql(compiler, connection, **extra_context)  <ins>class OrderByList(Func):    template = 'ORDER BY %(expressions)s'    def __init__(self, *expressions, **extra):        expressions = (            (                OrderBy(F(expr[1:]), descending=True)                if isinstance(expr, str) and expr[0] == '-'                else expr            )            for expr in expressions        )        super().__init__(*expressions, **extra)    def as_sql(self, *args, **kwargs):        if not self.source_expressions:            return '', ()        return super().as_sql(*args, **kwargs)</ins> class ExpressionWrapper(SQLiteNumericMixin, Expression):     """     An expression that can wrap another expression so that it can provide


Question: if self.order_by is not None:             if isinstance(self.order_by, (list, tuple)):                self.order_by = ExpressionList(*self.order_by)            elif not isinstance(self.order_by, BaseExpression):                self.order_by = OrderByList(*self.order_by)            elif isinstance(self.order_by, (BaseExpression, str)):                self.order_by = OrderByList(self.order_by)            else:                 raise ValueError(                    'order_by must be either an Expression or a sequence of '                    'expressions.'                    'Window.order_by must be either a string reference to a '                    'field, an expression, or a list or tuple of them.'                 )         super().__init__(output_field=output_field)         self.source_expression = self._parse_expressions(expression)[0]


Question: compiler=compiler, connection=connection,                 template='PARTITION BY %(expressions)s',             )            window_sql.extend(sql_expr)            window_sql.append(sql_expr)             window_params.extend(sql_params)          if self.order_by is not None:            window_sql.append(' ORDER BY ')             order_sql, order_params = compiler.compile(self.order_by)            window_sql.extend(order_sql)            window_sql.append(order_sql)             window_params.extend(order_params)          if self.frame:             frame_sql, frame_params = compiler.compile(self.frame)            window_sql.append(' ' + frame_sql)            window_sql.append(frame_sql)             window_params.extend(frame_params)          params.extend(window_params)


Question: return template % {             'expression': expr_sql,            'window': ''.join(window_sql).strip()            'window': ' '.join(window_sql).strip()         }, params      def as_sqlite(self, compiler, connection):


Question: return '{} OVER ({}{}{})'.format(             str(self.source_expression),             'PARTITION BY ' + str(self.partition_by) if self.partition_by else '',            'ORDER BY ' + str(self.order_by) if self.order_by else '',            str(self.order_by or ''),             str(self.frame or ''),         )


Question: import stat import tempfile from importlib import import_modulefrom urllib.request import urlretrievefrom urllib.request import build_opener  import django from django.conf import settings

django-admin start[project|app] doesn't send proper user agent header when HTTP(S) url is sent. Description (last modified by rsp2k) By default, when fetching a remote template, django-admin start[app|project] uses the default urllib User Agent which causes some sites to block requests (namely gitlab, since they use cloudflare See ). This patch sets the 'User Agent' header to Django/DJANGO_VERSION.


Question: if self.verbosity >= 2:             self.stdout.write('Downloading %s' % display_url)        the_path = os.path.join(tempdir, filename)        opener = build_opener()        opener.addheaders = [('User-Agent', f'Django/{django.__version__}')]         try:            the_path, info = urlretrieve(url, os.path.join(tempdir, filename))            with opener.open(url) as source, open(the_path, 'wb') as target:                headers = source.info()                target.write(source.read())         except OSError as e:             raise CommandError("couldn't download URL %s to %s: %s" %                                (url, filename, e))


Question: used_name = the_path.split('/')[-1]          # Trying to get better name from response headers        content_disposition = info.get('content-disposition')        content_disposition = headers['content-disposition']         if content_disposition:             _, params = cgi.parse_header(content_disposition)             guessed_filename = params.get('filename') or used_name


Question: # Falling back to content type guessing         ext = self.splitext(guessed_filename)[1]        content_type = info.get('content-type')        content_type = headers['content-type']         if not ext and content_type:             ext = mimetypes.guess_extension(content_type)             if ext:


Question: from django.apps import AppConfigfrom django.contrib.messages.storage import basefrom django.contrib.messages.utils import get_level_tagsfrom django.test.signals import setting_changed from django.utils.translation import gettext_lazy as _  def update_level_tags(setting, **kwargs):    if setting == 'MESSAGE_TAGS':        base.LEVEL_TAGS = get_level_tags() class MessagesConfig(AppConfig):     name = 'django.contrib.messages'     verbose_name = _("Messages")    def ready(self):        setting_changed.connect(update_level_tags)

LEVELTAGS not updated when using Description When reading messages inside tests, new message tags created using is not updated. That causes the property results to be an empty string and not know the new tags.


Question: if self.distinct_fields != rhs.distinct_fields:             raise TypeError('Cannot combine queries with different distinct fields.') <ins>        # If lhs and rhs shares the same alias prefix, it is possible to have        # conflicting alias changes like T4 -> T5, T5 -> T6, which might end up        # as T4 -> T6 while combining two querysets. To prevent this, change an        # alias prefix of the rhs and update current aliases accordingly,        # except if the alias is the base table since it must be present in the        # query on both sides.        initial_alias = self.get_initial_alias()        rhs.bump_prefix(self, exclude={initial_alias})</ins>         # Work out how to relabel the rhs aliases, if necessary.         change_map = {}         conjunction = (connector == AND)

Query.changealiases raises an AssertionError Description Python Version: 3.9.2 Django Version: 2.2.24, 3.2.9 (reproduced using two different versions) Code to Reproduce from import models class Foo(models.Model): qux = models.ForeignKey("app.Qux", ondelete=models.CASCADE, relatedname="foos") class Bar(models.Model): foo = models.ForeignKey("app.Foo", ondelete=models.CASCADE, relatedname="bars") anotherfoo = models.ForeignKey("app.Foo", ondelete=models.CASCADE, relatedname="otherbars") baz = models.ForeignKey("app.Baz", ondelete=models.CASCADE, relatedname="bars") class Baz(models.Model): pass class Qux(models.Model): bazes = models.ManyToManyField("app.Baz", relatedname="quxes") from import Q from import Foo, Qux qux = () qs1 = () qs2 = ( Q(()) | Q(()) ) qs2 | qs1 qs1 | qs2 Description I have encountered this bug during working on a project, recreated the code to reproduce as simple as I can. I have also examined the reason behind this bug, as far as I understand the reason is that during an or operation of two QuerySets, in Query.combine method of the variable combined, if rhs's Query currently have sequential aliases (e.g. T4 and T5) and related tablenames also exist in lhs.tablemap, calling Query.tablealias in will result in creation of aliases T5 for T4 and T6 for T5, thus changemap's keys intersect with changemap's values, so the AssertionError above is raised. Expectation Could you please fix this bug? Maybe aliasmap of rhs can be provided to and Query.tablealias, and suffix (number) of the new alias might be incremented until it is not in rhs.aliasmap, to prevent intersection between changemap's keys and values. Assertion in the first line of QuerySet.changealiases is not documented via a comment. As far as I understand, it is there because if keys and values intersects it means that an alias might be changed twice (e.g. first T4 -T5, and then T5 -T6) according to their order in the changemap. IMHO there can be a comment about what it assures, or an explanation can be to the AssertionError (like the assertions in the Query.combine method).


Question: if self.distinct_fields != rhs.distinct_fields:             raise TypeError('Cannot combine queries with different distinct fields.') <ins>        # If lhs and rhs shares the same alias prefix, it is possible to have        # conflicting alias changes like T4 -> T5, T5 -> T6, which might end up        # as T4 -> T6 while combining two querysets. To prevent this, change an        # alias prefix of the rhs and update current aliases accordingly,        # except if the alias is the base table since it must be present in the        # query on both sides.        initial_alias = self.get_initial_alias()        rhs.bump_prefix(self, exclude={initial_alias})</ins>         # Work out how to relabel the rhs aliases, if necessary.         change_map = {}         conjunction = (connector == AND)

It seems like QuerySet's OR operation is not commutative (they can create different queries, even though the results are the same), IMHO this can be explicitly declared on the documentation.


Question: # the AND case. The results will be correct but this creates too many         # joins. This is something that could be fixed later on.         reuse = set() if conjunction else set(self.alias_map)<del>        # Base table must be present in the query - this is the same        # table on both sides.        self.get_initial_alias()</del>         joinpromoter = JoinPromoter(connector, 2, False)         joinpromoter.add_votes(             j for j in self.alias_map if self.alias_map[j].join_type == INNER)


Question: # the AND case. The results will be correct but this creates too many         # joins. This is something that could be fixed later on.         reuse = set() if conjunction else set(self.alias_map)<del>        # Base table must be present in the query - this is the same        # table on both sides.        self.get_initial_alias()</del>         joinpromoter = JoinPromoter(connector, 2, False)         joinpromoter.add_votes(             j for j in self.alias_map if self.alias_map[j].join_type == INNER)


Question: relabelling any references to them in select columns and the where         clause.         """<ins>        # If keys and values of change_map were to intersect, an alias might be        # updated twice (e.g. T4 -> T5, T5 -> T6, so also T4 -> T6) depending        # on their order in change_map.</ins>         assert set(change_map).isdisjoint(change_map.values())          # 1. Update references in "select" (normal columns plus aliases),


Question: relabelling any references to them in select columns and the where         clause.         """<ins>        # If keys and values of change_map were to intersect, an alias might be        # updated twice (e.g. T4 -> T5, T5 -> T6, so also T4 -> T6) depending        # on their order in change_map.</ins>         assert set(change_map).isdisjoint(change_map.values())          # 1. Update references in "select" (normal columns plus aliases),


Question: for alias, aliased in self.external_aliases.items()         }     def bump_prefix(self, outer_query):    def bump_prefix(self, other_query, exclude=None):         """         Change the alias prefix to the next letter in the alphabet in a way        that the outer query's aliases and this query's aliases will not        that the other query's aliases and this query's aliases will not         conflict. Even tables that previously had no alias will get an alias        after this call.        after this call. To prevent changing aliases use the exclude parameter.         """         def prefix_gen():             """


Question: for alias, aliased in self.external_aliases.items()         }     def bump_prefix(self, outer_query):    def bump_prefix(self, other_query, exclude=None):         """         Change the alias prefix to the next letter in the alphabet in a way        that the outer query's aliases and this query's aliases will not        that the other query's aliases and this query's aliases will not         conflict. Even tables that previously had no alias will get an alias        after this call.        after this call. To prevent changing aliases use the exclude parameter.         """         def prefix_gen():             """


Question: yield ''.join(s)                 prefix = None         if self.alias_prefix != outer_query.alias_prefix:        if self.alias_prefix != other_query.alias_prefix:             # No clashes between self and outer query should be possible.             return


Question: yield ''.join(s)                 prefix = None         if self.alias_prefix != outer_query.alias_prefix:        if self.alias_prefix != other_query.alias_prefix:             # No clashes between self and outer query should be possible.             return


Question: 'Maximum recursion depth exceeded: too many subqueries.'                 )         self.subq_aliases = self.subq_aliases.union([self.alias_prefix])        outer_query.subq_aliases = outer_query.subq_aliases.union(self.subq_aliases)        other_query.subq_aliases = other_query.subq_aliases.union(self.subq_aliases)        if exclude is None:            exclude = {}         self.change_aliases({             alias: '%s%d' % (self.alias_prefix, pos)             for pos, alias in enumerate(self.alias_map)            if alias not in exclude         })      def get_initial_alias(self):


Question: 'Maximum recursion depth exceeded: too many subqueries.'                 )         self.subq_aliases = self.subq_aliases.union([self.alias_prefix])        outer_query.subq_aliases = outer_query.subq_aliases.union(self.subq_aliases)        other_query.subq_aliases = other_query.subq_aliases.union(self.subq_aliases)        if exclude is None:            exclude = {}         self.change_aliases({             alias: '%s%d' % (self.alias_prefix, pos)             for pos, alias in enumerate(self.alias_map)            if alias not in exclude         })      def get_initial_alias(self):


Question: using=using, raw=raw,         )     def _prepare_related_fields_for_save(self, operation_name):    def _prepare_related_fields_for_save(self, operation_name, fields=None):         # Ensure that a model instance without a PK hasn't been assigned to         # a ForeignKey or OneToOneField on this model. If the field is         # nullable, allowing the save would result in silent data loss.         for field in self._meta.concrete_fields:            if fields and field not in fields:                continue             # If the related field isn't cached, then an instance hasn't been             # assigned and there's no need to worry about this check.             if field.is_relation and field.is_cached(self):diff --git a/django/db/models/query.py b/django/db/models/query.py-- a/django/db/models/query.py++ b/django/db/models/query.py

Saving parent object after setting on child leads to unexpected data loss in bulkupdate(). Description Consider following example: class Child(models.Model): pass class Parent(models.Model): child = models.ForeignKey(Child, ondelete=models.CASCADE, null=True) parent = (child=None) = Child() () Parent.objects.bulkupdate([parent], fields=["child"]) Expected behavior: parent model instance was updated with the ID to the child in the database. Actual behavior: Parent model is still referencing Null. There should probably be some check for ForeignKeys in the bulkupdate logic, and if one is updated, the ID of the child model should be re-copied to the child_id field that is actually being written to the database.


Question: raise ValueError('bulk_update() cannot be used with primary key fields.')         if not objs:             return 0<ins>        for obj in objs:            obj._prepare_related_fields_for_save(operation_name='bulk_update', fields=fields)</ins>         # PK is used twice in the resulting update query, once in the filter         # and once in the WHEN. Each field will also have one CAST.         connection = connections[self.db]


Question: if not expiry:   # Checks both None and 0 cases             return self.get_session_cookie_age()        if not isinstance(expiry, datetime):        if not isinstance(expiry, (datetime, str)):             return expiry        if isinstance(expiry, str):            expiry = datetime.fromisoformat(expiry)         delta = expiry - modification         return delta.days * 86400 + delta.seconds

Deprecate PickleSerializer and move it out of core Description Pickle serializer has long been known to be dangerous. This is mitigated by requiring MAC on pickle in cookies, but nevertheless, RCEs continue to happen:  To further discourage it's use, we should consider deprecating PickleSerializer and moving it into a third party package.


Question: if isinstance(expiry, datetime):             return expiry<ins>        elif isinstance(expiry, str):            return datetime.fromisoformat(expiry)</ins>         expiry = expiry or self.get_session_cookie_age()         return modification + timedelta(seconds=expiry)


Question: return         if isinstance(value, timedelta):             value = timezone.now() + value<ins>        if isinstance(value, datetime):            value = value.isoformat()</ins>         self['_session_expiry'] = value      def get_expire_at_browser_close(self):


Question: ``get_expiry_date()`` or ``get_expiry_age()`` to find the actual expiry         date/age, if there is one.         """        if self.get('_session_expiry') is None:        if (expiry := self.get('_session_expiry')) is None:             return settings.SESSION_EXPIRE_AT_BROWSER_CLOSE        return self.get('_session_expiry') == 0        return expiry == 0      def flush(self):         """diff --git a/django/contrib/sessions/serializers.py b/django/contrib/sessions/serializers.py-- a/django/contrib/sessions/serializers.py++ b/django/contrib/sessions/serializers.py


Question: # RemovedInDjango50Warning. from django.core.serializers.base import (     PickleSerializer as BasePickleSerializer, )diff --git a/django/core/cache/backends/redis.py b/django/core/cache/backends/redis.py-- a/django/core/cache/backends/redis.py++ b/django/core/cache/backends/redis.py


Question: """Redis cache backend.""" import pickle import random import re  from django.core.cache.backends.base import DEFAULT_TIMEOUT, BaseCachefrom django.core.serializers.base import PickleSerializer from django.utils.functional import cached_property from django.utils.module_loading import import_string  class RedisSerializer(PickleSerializer):    """    Similar to PickSerializer, except integers are serialized as native Redis    integers for better incr() and decr() atomicity.    """class RedisSerializer:    def __init__(self, protocol=None):        self.protocol = pickle.HIGHEST_PROTOCOL if protocol is None else protocol     def dumps(self, obj):         # Only skip pickling for integers, a int subclasses as bool should be         # pickled.         if type(obj) is int:             return obj        return super().dumps(obj)        return pickle.dumps(obj, self.protocol)      def loads(self, data):         try:             return int(data)         except ValueError:            return super().loads(data)            return pickle.loads(data)   class RedisCacheClient:diff --git a/django/core/serializers/base.py b/django/core/serializers/base.py-- a/django/core/serializers/base.py++ b/django/core/serializers/base.py


Question: Module for abstract serializer/unserializer base classes. """ import pickleimport warnings from io import StringIO  from django.core.exceptions import ObjectDoesNotExist from django.db import modelsfrom django.utils.deprecation import RemovedInDjango50Warning  DEFER_FIELD = object()


Question: cache backends.     """     def __init__(self, protocol=None):<ins>        warnings.warn(            'PickleSerializer is deprecated due to its security risk. Use '            'JSONSerializer instead.',            RemovedInDjango50Warning,        )</ins>         self.protocol = pickle.HIGHEST_PROTOCOL if protocol is None else protocol      def dumps(self, obj):


Question: return [             constraint             for constraint in self.constraints            if isinstance(constraint, UniqueConstraint) and constraint.condition is None            if (                isinstance(constraint, UniqueConstraint) and                constraint.condition is None and                not constraint.contains_expressions            )         ]      @cached_property

Detecting uniqueness doesn't work for models with functional unique constraints. Description When creating a new object from Django Administration site, I'm getting an error "Tag with this already exists.". (Tag is my model name) I'm using on this model the new functional unique constraints introducted in Django 4.0 (). I suppose this is related to the contraints put on the model. However, creating a Tag object with code - e.g. (...) - works fine. It only fails in Django admin. I'm not sure if it's a bug or misunderstanding/misconfiguration on my side? This is my model: class Tag(models.Model): """Basic tag model.""" # When using getorcreate, we need to pass tag name as a default value, like this: # Tag.objects.getorcreate(defaults={'name': tag}, nameiexact=tag, ) user = models.ForeignKey(settings.AUTHUSERMODEL, ondelete=models.CASCADE, verbosename=('user')) name = models.CharField(maxlength=50, dbindex=False, verbosename=('name'), blank=False, # It's the default behavior, but let's be explicit validators=[RegexValidator(regex=r'[A-z0-9À-ž\s]+', message='This field accepts only letters, digits and space.')]) slug = models.SlugField(maxlength=50, verbosename=('slug')) class Meta: ordering = ['name'] # Make sure tag name: # - are unique per user # - are case insensitive (prevent adding "test" if "Test" already exists) # - aren't empty constraints = [models.UniqueConstraint(fields=['user', 'name'], name='%(applabel)s%(class)snameuniqueperuser'), models.UniqueConstraint(Lower('name'), name='%(applabel)s%(class)snamecaseinsensitive'), models.CheckConstraint(check=models.Q(namelengthgt=0), name='%(applabel)s%(class)snamenotempty')] (snip) This is the admin configuration: class TagAdmin(admin.ModelAdmin): listdisplay = ('pk', 'user', 'name', 'slug') listdisplaylinks = ['pk'] fields = ('user', 'name',) listfilter = ('useremail',) search_fields = ('name',)


Question: return self  <ins>@deconstructible(path='django.db.models.Func')</ins> class Func(SQLiteNumericMixin, Expression):     """An SQL function call."""     function = None

Use simplified paths for deconstruct of expressions Description Previously F() deconstructed to: (). But since it can also be imported from , PR changed it to deconstruct to (). This simplifies generated migration code where it will be referenced only as from import models / models.F(). As Mariusz pointed out on the PR, the same technique can be applied to other expressions, further simplifying generated migrations.


Question: return copy  <ins>@deconstructible(path='django.db.models.Value')</ins> class Value(SQLiteNumericMixin, Expression):     """Represent a wrapped value as a node within an expression."""     # Provide a default value for `for_save` in order to allow unresolved


Question: return super().as_sql(*args, **kwargs)  <ins>@deconstructible(path='django.db.models.ExpressionWrapper')</ins> class ExpressionWrapper(SQLiteNumericMixin, Expression):     """     An expression that can wrap another expression so that it can provide


Question: return "{}({})".format(self.__class__.__name__, self.expression)  <ins>@deconstructible(path='django.db.models.When')</ins> class When(Expression):     template = 'WHEN %(condition)s THEN %(result)s'     # This isn't a complete conditional expression, must be used in Case().


Question: return cols  <ins>@deconstructible(path='django.db.models.Case')</ins> class Case(SQLiteNumericMixin, Expression):     """     An SQL searched CASE expression:


Question: return sql, params  <ins>@deconstructible(path='django.db.models.OrderBy')</ins> class OrderBy(Expression):     template = '%(expression)s %(ordering)s'     conditional = False


Question: with connection.cursor() as cursor:             cursor.execute(                'SELECT %s FROM %s WHERE %s = %%s and expires > %%s' % (                'SELECT %s FROM %s WHERE %s = %%s and %s > %%s' % (                     quote_name('cache_key'),                     quote_name(self._table),                     quote_name('cache_key'),                    quote_name('expires'),                 ),                 [key, connection.ops.adapt_datetimefield_value(now)]             )

DatabaseCache backend doesn't quote all fields in queries Description Snowflake requires all fields to be quoted, otherwise they're treated as uppercase. The attached patch works with stable/3.2.x, but I'll have to review it once django-snowflake development is caught up to Django's main branch.


Question: else:             connection = connections[db]             table = connection.ops.quote_name(self._table)            cursor.execute("DELETE FROM %s WHERE expires < %%s" % table,                           [connection.ops.adapt_datetimefield_value(now)])            cursor.execute('DELETE FROM %s WHERE %s < %%s' % (                table,                connection.ops.quote_name('expires'),            ), [connection.ops.adapt_datetimefield_value(now)])             deleted_count = cursor.rowcount             remaining_num = num - deleted_count             if remaining_num > self._max_entries:


Question: last_cache_key = cursor.fetchone()                 if last_cache_key:                     cursor.execute(                        'DELETE FROM %s WHERE cache_key < %%s' % table,                        'DELETE FROM %s WHERE %s < %%s' % (                            table,                            connection.ops.quote_name('cache_key'),                        ),                         [last_cache_key[0]],                     ) diff --git a/django/db/backends/base/operations.py b/django/db/backends/base/operations.py-- a/django/db/backends/base/operations.py++ b/django/db/backends/base/operations.py


Question: This is used by the 'db' cache backend to determine where to start         culling.         """        return "SELECT cache_key FROM %s ORDER BY cache_key LIMIT 1 OFFSET %%s"        cache_key = self.quote_name('cache_key')        return f'SELECT {cache_key} FROM %s ORDER BY {cache_key} LIMIT 1 OFFSET %%s'      def unification_cast_sql(self, output_field):         """diff --git a/django/db/backends/oracle/operations.py b/django/db/backends/oracle/operations.py-- a/django/db/backends/oracle/operations.py++ b/django/db/backends/oracle/operations.py


Question: }      def cache_key_culling_sql(self):        return 'SELECT cache_key FROM %s ORDER BY cache_key OFFSET %%s ROWS FETCH FIRST 1 ROWS ONLY'        cache_key = self.quote_name('cache_key')        return (            f'SELECT {cache_key} '            f'FROM %s '            f'ORDER BY {cache_key} OFFSET %%s ROWS FETCH FIRST 1 ROWS ONLY'        )      def date_extract_sql(self, lookup_type, field_name):         if lookup_type == 'week_day':


Question: def _path(route, view, kwargs=None, name=None, Pattern=None):     from django.views import View <ins>    if kwargs is not None and not isinstance(kwargs, dict):        raise TypeError(            f'kwargs argument must be a dict, but got {kwargs.__class__.__name__}.'        )</ins>     if isinstance(view, (list, tuple)):         # For include(...) processing.         pattern = Pattern(route, is_endpoint=False)

path()/repath() should raise a TypeError when kwargs is not a dict. Description Apparently, however many years into using Django, I'm still capable of making a "newbie" mistake and getting confused. So perhaps other actual new users encounter similar, especially given the lack of typing specifiers. I defined a URL like so: urlpatterns = [ path("path/to/thing", (), "myview"), ] which ... well, you either spot the issue immediately or you don't, and end up with the following. If you try and resolve() the path (eg: by making a request in your browser), you'll get something like: In [3]: resolve("/path/to/thing") in resolve(path, urlconf) 22 if urlconf is None: 23 urlconf = geturlconf() ---24 return getresolver(urlconf).resolve(path) 25 26 in resolve(self, path) 586 for pattern in self.urlpatterns: 587 try: --588 submatch = pattern.resolve(newpath) 589 except Resolver404 as e: 590 self.extendtried(tried, pattern, e.args[0].get('tried')) in resolve(self, path) 388 newpath, args, kwargs = match 389 # Pass any extrakwargs as kwargs. --390 (self.defaultargs) 391 return ResolverMatch(self.callback, args, kwargs, , route=str(self.pattern)) 392 ValueError: dictionary update sequence element has length 1; 2 is required The crux of the issue being that I meant to give the URL a name, and it's a super unfortunate history that kwargs comes before the name argument (because nearly everyone gives a URL a name, but passing static kwargs is comparatively infrequent). So what's actually happened is that kwargs = "myview" and eventually self.defaultargs = "myview".


Question: def _path(route, view, kwargs=None, name=None, Pattern=None):     from django.views import View <ins>    if kwargs is not None and not isinstance(kwargs, dict):        raise TypeError(            f'kwargs argument must be a dict, but got {kwargs.__class__.__name__}.'        )</ins>     if isinstance(view, (list, tuple)):         # For include(...) processing.         pattern = Pattern(route, is_endpoint=False)

If I update to path("path/to/thing", (), "myview", name="myview"), leaving the type incorrect, I can get the following error via reverse, too: In [4]: reverse("myview") in reverse(viewname, urlconf, args, kwargs, currentapp) 84 resolver = getnsresolver(nspattern, resolver, tuple(())) 85 ---86 return resolver.reversewithprefix(view, prefix, args, kwargs) 87 88 in reversewithprefix(self, lookupview, prefix, args, **kwargs) 669 if set(kwargs).symmetricdifference(params).difference(defaults): 670 continue --671 if any((k, v) != v for k, v in ()): 672 continue 673 candidatesubs = kwargs AttributeError: 'str' object has no attribute 'items' Both of these suggest that either there should be a type-guard in path to assert it's dict-ish (if not None), or a system check on URLPattern to raise a friendly message. Well, they actually continue to suggest to me that everything after the view argument should be keyword-only, or that kwargs should come later, but I suspect those to be a harder sell ;) This is specifically around the kwargs, but it doesn't look like there's any guarding on the name either, and I feel like a name of {'test': 'test'} (i.e. accidentally swapped both positionals) is likely to bite & cause an issue somewhere.


Question: import copy import itertools import operator<ins>import warnings</ins> from functools import total_ordering, wraps

Deprecate cachedproperty's name argument Description Django 2.2 is the last version to support Python 3.5 where cachedproperty's name argument is required. Following the release of Django 4.0, most apps will drop support for Django 2.2 (and hence Python 3.5), so Django 4.0 can deprecate the name argument without giving warnings that aren't actionable as long as apps want to keep support for Python 3.5.


Question: A cached property can be made out of an existing method:     (e.g. ``url = cached_property(get_absolute_url)``).<del>    The optional ``name`` argument is obsolete as of Python 3.6 and will be    deprecated in Django 4.0 (#30127).</del>     """     name = None


Question: )      def __init__(self, func, name=None):<ins>        from django.utils.deprecation import RemovedInDjango50Warning        if name is not None:            warnings.warn(                "The name argument is deprecated as it's unnecessary as of "                "Python 3.6.",                RemovedInDjango50Warning,                stacklevel=2,            )</ins>         self.real_func = func         self.__doc__ = getattr(func, '__doc__')


Question: raise ValidationError(self.message, code=self.code, params={'value': value})          # Then check full URL        try:            splitted_url = urlsplit(value)        except ValueError:            raise ValidationError(self.message, code=self.code, params={'value': value})         try:             super().__call__(value)         except ValidationError as e:             # Trivial case failed. Try for possible IDN domain             if value:                try:                    scheme, netloc, path, query, fragment = urlsplit(value)                except ValueError:  # for example, "Invalid IPv6 URL"                    raise ValidationError(self.message, code=self.code, params={'value': value})                scheme, netloc, path, query, fragment = splitted_url                 try:                     netloc = punycode(netloc)  # IDN -> ACE                 except UnicodeError:  # invalid domain part

URLField throws ValueError instead of ValidationError on clean Description forms.URLField( ).clean('////] results in: ValueError: Invalid IPv6 URL Traceback (most recent call last): File "", line 22, in TestOneInput File "", line 350, in testformsURLField File "", line 151, in clean File "", line 136, in runvalidators File "", line 130, in call File "", line 440, in urlsplit


Question: raise         else:             # Now verify IPv6 in the netloc part            host_match = re.search(r'^\[(.+)\](?::\d{1,5})?$', urlsplit(value).netloc)            host_match = re.search(r'^\[(.+)\](?::\d{1,5})?$', splitted_url.netloc)             if host_match:                 potential_ip = host_match[1]                 try:


Question: # section 3.1. It's defined to be 255 bytes or less, but this includes         # one byte for the length of the name and one byte for the trailing dot         # that's used to indicate absolute names in DNS.        if len(urlsplit(value).hostname) > 253:        if splitted_url.hostname is None or len(splitted_url.hostname) > 253:             raise ValidationError(self.message, code=self.code, params={'value': value})


Question: iso8601_duration_re = _lazy_re_compile(     r'^(?P<sign>[-+]?)'     r'P'    r'(?:(?P<days>\d+(.\d+)?)D)?'    r'(?:(?P<days>\d+([\.,]\d+)?)D)?'     r'(?:T'    r'(?:(?P<hours>\d+(.\d+)?)H)?'    r'(?:(?P<minutes>\d+(.\d+)?)M)?'    r'(?:(?P<seconds>\d+(.\d+)?)S)?'    r'(?:(?P<hours>\d+([\.,]\d+)?)H)?'    r'(?:(?P<minutes>\d+([\.,]\d+)?)M)?'    r'(?:(?P<seconds>\d+([\.,]\d+)?)S)?'     r')?'     r'$' )

fails to handle broken data Description (last modified by Florian Apolloner) The actual input string was 'P3(3D' === Uncaught Python exception: === ValueError: could not convert string to float: '3(3' Traceback (most recent call last): File "", line 22, in TestOneInput File "", line 294, in testformsDurationField File "", line 149, in clean File "", line 502, in topython File "", line 154, in parse_duration File "", line 154, in <dictcomp


Question: from functools import wraps <del>from django.http import HttpRequest</del> from django.middleware.cache import CacheMiddleware from django.utils.cache import add_never_cache_headers, patch_cache_control from django.utils.decorators import decorator_from_middleware_with_args

nevercache()/cachecontrol() decorators raise error on duck-typed requests. Description The cache decorators cachecontrol, nevercache and sensitivepostparameters no longer work with Django REST framework because they strictly check for an HttpRequest instance.


Question: def _cache_controller(viewfunc):         @wraps(viewfunc)         def _cache_controlled(request, *args, **kw):            if not isinstance(request, HttpRequest):            # Ensure argument looks like a request.            if not hasattr(request, 'META'):                 raise TypeError(                     "cache_control didn't receive an HttpRequest. If you are "                     "decorating a classmethod, be sure to use "


Question: """     @wraps(view_func)     def _wrapped_view_func(request, *args, **kwargs):        if not isinstance(request, HttpRequest):        # Ensure argument looks like a request.        if not hasattr(request, 'META'):             raise TypeError(                 "never_cache didn't receive an HttpRequest. If you are "                 "decorating a classmethod, be sure to use @method_decorator."


Question: defaults = {'form_class': form_class, 'required': False}         return super().formfield(**{**defaults, **kwargs}) <ins>    def select_format(self, compiler, sql, params):        sql, params = super().select_format(compiler, sql, params)        # Filters that match everything are handled as empty strings in the        # WHERE clause, but in SELECT or GROUP BY list they must use a        # predicate that's always True.        if sql == '':            sql = '1'        return sql, params</ins>  class CharField(Field):     description = _("String (up to %(max_length)s)")

ExpressionWrapper for ~Q(pkin=[]) crashes. Description (last modified by Stefan Brand) Problem Description I'm reducing some Q objects (similar to what is described in ticket:. Everything is fine for the case where the result is ExpressionWrapper(Q(pkin=[])). However, when I reduce to ExpressionWrapper(~Q(pkin=[])) the query breaks. Symptoms Working for ExpressionWrapper(Q(pkin=[])) print(queryset.annotate(foo=ExpressionWrapper(Q(pkin=[]), outputfield=BooleanField())).values("foo").query) SELECT 0 AS "foo" FROM "table" Not working for ExpressionWrapper(~Q(pkin=[])) print(queryset.annotate(foo=ExpressionWrapper(~Q(pkin=[]), outputfield=BooleanField())).values("foo").query) SELECT AS "foo" FROM "table"


Question: '--check', action='store_true', dest='check_unapplied',             help='Exits with a non-zero status if unapplied migrations exist.',         )<ins>        parser.add_argument(            '--prune', action='store_true', dest='prune',            help='Delete nonexistent migrations from the django_migrations table.',        )</ins>      @no_translations     def handle(self, *args, **options):

Delete nonexistent migrations from djangomigrations table Description Django adds a djangomigrations table to the database which list all of the migrations that have been applied (and when). With the introduction of squashmigrations, it is possible for this table to contain a lot of old migrations that no longer exist. This can be problematic if naming duplication occurs: Example: I have an app with: myapp/migrations/ I squash and delete replaced migrations: myapp/migrations/ I create a new migration and use poor naming: myapp/migrations/ My new migration never runs because the djangomigrations table thinks it has already been applied. I propose truncation of the djangomigrations table so that it includes only migrations that actually exist in the django project. This could be done automatically (when executor runs, or inside the migrate mcommand). Or have its own mcommand that requires it be run manually. I prefer the automatic approach though. Pros: Cleans up old data that just bloats the database. Protects users from the trap mentioned above where a new migration is created with the same name as one that was applied in the past. Cons: A loss of historical information. Note: Need to be careful with implementation to avoid a possible new trap if a user squashes migrations and then proceeds to delete the replaced migrations before running the squashed migrations on their database -django will think squashed migrations haven't been applied and will attempt to reapply them. This can be remedied simply by not removing migrations mentioned in replaces lists of other migrations from djangomigrations (ie. we'd consider replaced migrations as still existing, even if their actual files have already been removed).


Question: else:             targets = executor.loader.graph.leaf_nodes() <ins>        if options['prune']:            if not options['app_label']:                raise CommandError(                    'Migrations can be pruned only when an app is specified.'                )            if self.verbosity > 0:                self.stdout.write('Pruning migrations:', self.style.MIGRATE_HEADING)            to_prune = set(executor.loader.applied_migrations) - set(executor.loader.disk_migrations)            squashed_migrations_with_deleted_replaced_migrations = [                migration_key                for migration_key, migration_obj in executor.loader.replacements.items()                if any(replaced in to_prune for replaced in migration_obj.replaces)            ]            if squashed_migrations_with_deleted_replaced_migrations:                self.stdout.write(self.style.NOTICE(                    "  Cannot use --prune because the following squashed "                    "migrations have their 'replaces' attributes and may not "                    "be recorded as applied:"                ))                for migration in squashed_migrations_with_deleted_replaced_migrations:                    app, name = migration                    self.stdout.write(f'    {app}.{name}')                self.stdout.write(self.style.NOTICE(                    "  Re-run 'manage.py migrate' if they are not marked as "                    "applied, and remove 'replaces' attributes in their "                    "Migration classes."                ))            else:                to_prune = sorted(                    migration                    for migration in to_prune                    if migration[0] == app_label                )                if to_prune:                    for migration in to_prune:                        app, name = migration                        if self.verbosity > 0:                            self.stdout.write(self.style.MIGRATE_LABEL(                                f'  Pruning {app}.{name}'                            ), ending='')                        executor.recorder.record_unapplied(app, name)                        if self.verbosity > 0:                            self.stdout.write(self.style.SUCCESS(' OK'))                elif self.verbosity > 0:                    self.stdout.write('  No migrations to prune.')</ins>         plan = executor.migration_plan(targets)         exit_dry = plan and options['check_unapplied']


Question: return         if exit_dry:             sys.exit(1)<ins>        if options['prune']:            return</ins>          # At this point, ignore run_syncdb if there aren't any apps to sync.         run_syncdb = options['run_syncdb'] and executor.loader.unmigrated_apps


Question: return [], {}, set(), []     else:         using = router.db_for_write(obj._meta.model)    collector = NestedObjects(using=using)    collector = NestedObjects(using=using, origin=objs)     collector.collect(objs)     perms_needed = set() diff --git a/django/contrib/contenttypes/management/commands/remove_stale_contenttypes.py b/django/contrib/contenttypes/management/commands/remove_stale_contenttypes.py-- a/django/contrib/contenttypes/management/commands/remove_stale_contenttypes.py++ b/django/contrib/contenttypes/management/commands/remove_stale_contenttypes.py

Add ability to know where an object was deleted from to pre/post delete signals Description Since deleting objects deletes related objects, it would be good to know the origin of the delete. It lets the signal listeners know why an object is being delete. If it's deleted from a (), you know which instance initiated the delete and if it's from a () you know which queryset initiated the delete, with the models in that queryset. Using this you know if the instance is being deleted directly or because it's related to another instance.


Question: ct_info = []                     for ct in to_remove:                         ct_info.append('    - Content type for %s.%s' % (ct.app_label, ct.model))                        collector = NoFastDeleteCollector(using=using)                        collector = NoFastDeleteCollector(using=using, origin=ct)                         collector.collect([ct])                          for obj_type, objs in collector.data.items():diff --git a/django/db/models/base.py b/django/db/models/base.py-- a/django/db/models/base.py++ b/django/db/models/base.py


Question: "to None." % (self._meta.object_name, self._meta.pk.attname)             )         using = using or router.db_for_write(self.__class__, instance=self)        collector = Collector(using=using)        collector = Collector(using=using, origin=self)         collector.collect([self], keep_parents=keep_parents)         return collector.delete() diff --git a/django/db/models/deletion.py b/django/db/models/deletion.py-- a/django/db/models/deletion.py++ b/django/db/models/deletion.py


Question: class Collector:    def __init__(self, using):    def __init__(self, using, origin=None):         self.using = using        # A Model or QuerySet object.        self.origin = origin         # Initially, {model: {instances}}, later values become lists.         self.data = defaultdict(set)         # {model: {(field, value): {instances}}}


Question: for model, obj in self.instances_with_model():                 if not model._meta.auto_created:                     signals.pre_delete.send(                        sender=model, instance=obj, using=self.using                        sender=model, instance=obj, using=self.using,                        origin=self.origin,                     )              # fast deletes