Question: if not model._meta.auto_created:                     for obj in instances:                         signals.post_delete.send(                            sender=model, instance=obj, using=self.using                            sender=model, instance=obj, using=self.using,                            origin=self.origin,                         )          # update collected instancesdiff --git a/django/db/models/query.py b/django/db/models/query.py-- a/django/db/models/query.py++ b/django/db/models/query.py


Question: del_query.query.select_related = False         del_query.query.clear_ordering(force=True)         collector = Collector(using=del_query.db)        collector = Collector(using=del_query.db, origin=self)         collector.collect(del_query)         deleted, _rows_count = collector.delete()

Add ability to know where an object was deleted from to pre/post delete signals Description Since deleting objects deletes related objects, it would be good to know the origin of the delete. It lets the signal listeners know why an object is being delete. If it's deleted from a (), you know which instance initiated the delete and if it's from a () you know which queryset initiated the delete, with the models in that queryset. Using this you know if the instance is being deleted directly or because it's related to another instance.


Question: (un)applied and in a second step run all the database operations.         """         # The django_migrations table must be present to record applied        # migrations.        self.recorder.ensure_schema()        # migrations, but don't create it if there are no migrations to apply.        if plan == []:            if not self.recorder.has_table():                return self._create_project_state(with_applied_migrations=False)        else:            self.recorder.ensure_schema()          if plan is None:             plan = self.migration_plan(targets)

MigrationRecorder does not obey dbrouter allowmigrate rules Description Hi, We have a multi-db setup. We have one connection that is for the django project, and several connections that talk to other dbs for information (ie models with managed = False). Django should only create tables in the first connection, never in any of the other connections. We have a simple router that does the following: class Router(object): def allowmigrate(self, db, model): if db == 'default': return True return False Current Behaviour We run our functional tests and the migrate command is called against each connection when the test databases are created (see , setupdatabases, line 300-ish, which calls , createtestdb, line 377-ish) When this migrate runs, it tries to apply our migrations, which tries to record that a migration has been applied (see , applymigration, which has several calls to self.recorder.recordapplied). The first thing that recordapplied does is a call to self.ensureschema() (see , recordapplied, lien 66-ish). ensureschema checks to see if the Migration model is in the tables in the connection. If it does not find the table then it tries to create the table. I believe that this is incorrect behaviour when a dbrouter has been provided. If using the router above, my expectation would be that the table is not created on any connection other than the 'default' connection. Looking at the other methods on the MigrationRecorder, I would expect that there will be similar issues with appliedmigrations and record_unapplied.


Question: def reduce(self, operation, app_label):         return (             super().reduce(operation, app_label) or            not operation.references_model(self.name, app_label)            self.can_reduce_through(operation, app_label)         )     def can_reduce_through(self, operation, app_label):        return not operation.references_model(self.name, app_label)  class CreateModel(ModelOperation):     """Create a model's table."""

Optimize multiple AlterFooTogether operations into one Description Hi, In we split the AlterFooTogether (AlterUniqueTogether and AlterIndexTogether) operations into two types of operations. First, a migration will have operations to remove constraints, and then other operations adds the new constraints. This allows field alterations to work as expected during in between operations. In some cases, this introduced two operations that can actually easily be reduced to one. See for instance the test case:  Example: operations = [ migrations.AlterUniqueTogether( name='mymodel', uniquetogether=set(), ), migrations.AlterIndexTogether( name='mymodel', indextogether=set(), ), migrations.AlterUniqueTogether( name='mymodel', uniquetogether={("col",)}, ), migrations.AlterIndexTogether( name='mymodel', indextogether={("col",)}, ), ] should be optimized to operations = [ migrations.AlterUniqueTogether( name='mymodel', uniquetogether={("col",)}, ), migrations.AlterIndexTogether( name='mymodel', indextogether={("col",)}, ), ] So that we don't do two operations on each constraint, but only one.


Question: def migration_name_fragment(self):         return 'alter_%s_%s' % (self.name_lower, self.option_name) <ins>    def can_reduce_through(self, operation, app_label):        return (            super().can_reduce_through(operation, app_label) or (                isinstance(operation, AlterTogetherOptionOperation) and                type(operation) is not type(self)            )        )</ins>  class AlterUniqueTogether(AlterTogetherOptionOperation):     """


Question: <ins>import sysfrom django.apps import appsfrom django.core.management.base import BaseCommand, CommandErrorfrom django.core.management.utils import run_formattersfrom django.db import migrationsfrom django.db.migrations.exceptions import AmbiguityErrorfrom django.db.migrations.loader import MigrationLoaderfrom django.db.migrations.optimizer import MigrationOptimizerfrom django.db.migrations.writer import MigrationWriterfrom django.utils.version import get_docs_versionclass Command(BaseCommand):    help = "Optimizes the operations for the named migration."    def add_arguments(self, parser):        parser.add_argument(            "app_label",            help="App label of the application to optimize the migration for.",        )        parser.add_argument(            "migration_name", help="Migration name to optimize the operations for."        )        parser.add_argument(            "--check",            action="store_true",            help="Exit with a non-zero status if the migration can be optimized.",        )    def handle(self, *args, **options):        verbosity = options["verbosity"]        app_label = options["app_label"]        migration_name = options["migration_name"]        check = options["check"]        # Validate app_label.        try:            apps.get_app_config(app_label)        except LookupError as err:            raise CommandError(str(err))        # Load the current graph state.        loader = MigrationLoader(None)        if app_label not in loader.migrated_apps:            raise CommandError(f"App '{app_label}' does not have migrations.")        # Find a migration.        try:            migration = loader.get_migration_by_prefix(app_label, migration_name)        except AmbiguityError:            raise CommandError(                f"More than one migration matches '{migration_name}' in app "                f"'{app_label}'. Please be more specific."            )        except KeyError:            raise CommandError(                f"Cannot find a migration matching '{migration_name}' from app "                f"'{app_label}'."            )        # Optimize the migration.        optimizer = MigrationOptimizer()        new_operations = optimizer.optimize(migration.operations, migration.app_label)        if len(migration.operations) == len(new_operations):            if verbosity > 0:                self.stdout.write("No optimizations possible.")            return        else:            if verbosity > 0:                self.stdout.write(                    "Optimizing from %d operations to %d operations."                    % (len(migration.operations), len(new_operations))                )            if check:                sys.exit(1)        # Set the new migration optimizations.        migration.operations = new_operations        # Write out the optimized migration file.        writer = MigrationWriter(migration)        migration_file_string = writer.as_string()        if writer.needs_manual_porting:            if migration.replaces:                raise CommandError(                    "Migration will require manual porting but is already a squashed "                    "migration.\nTransition to a normal migration first: "                    "https://docs.djangoproject.com/en/%s/topics/migrations/"                    "#squashing-migrations" % get_docs_version()                )            # Make a new migration with those operations.            subclass = type(                "Migration",                (migrations.Migration,),                {                    "dependencies": migration.dependencies,                    "operations": new_operations,                    "replaces": [(migration.app_label, migration.name)],                },            )            optimized_migration_name = "%s_optimized" % migration.name            optimized_migration = subclass(optimized_migration_name, app_label)            writer = MigrationWriter(optimized_migration)            migration_file_string = writer.as_string()            if verbosity > 0:                self.stdout.write(                    self.style.MIGRATE_HEADING("Manual porting required") + "\n"                    "  Your migrations contained functions that must be manually "                    "copied over,\n"                    "  as we could not safely copy their implementation.\n"                    "  See the comment at the top of the optimized migration for "                    "details."                )        with open(writer.path, "w", encoding="utf-8") as fh:            fh.write(migration_file_string)        run_formatters([writer.path])        if verbosity > 0:            self.stdout.write(                self.style.MIGRATE_HEADING(f"Optimized migration {writer.path}")            )</ins>

Add a management command to optimize a migration Description (last modified by Raphael Gaschignard) Because the migration optimizer still has a bit of trouble with reducing operations, you often want to edit squashed migrations and re-run it through the optimizer. The attached patch contains an implementation of a management command, optimizemigration, that will help speed up this process. optimizemigration appname migrationname reads a single migration, pass it through the optimizer, and then rewrite the result to disk. Unlike squashmigrations, this ignores things like the squashed-ness of the migration.


Question: def __init__(self, *args, db_collation=None, **kwargs):         super().__init__(*args, **kwargs)         self.db_collation = db_collation        self.validators.append(validators.MaxLengthValidator(self.max_length))        if self.max_length is not None:            self.validators.append(validators.MaxLengthValidator(self.max_length))      def check(self, **kwargs):         databases = kwargs.get('databases') or []

Micro-optimisation for Value.resolveoutputfield (by modifying ) Description Currently, when you do something like annotate(x=Value('test')) that will eventually probably call down into Value.resolveoutputfield() and run the following code: if isinstance(, str): return fields.CharField() which is innocuous enough. However, CharField currently expects that self.maxlength is always a non null value of sensible data, and AFAIK this is caught for users at system-check time as a requirement for use. So what currently happens is that the CharField internally gets granted a MaxLengthValidator which cannot work and must be demonstrably extraneous (i.e. validators aren't used the outputfield, at least for Value) x = Value('test') y = x.resolveoutputfield() y.validators [ at 0x105e3d940] y.clean('1', modelinstance=None) in compare(self, a, b): TypeError: '' not supported between instances of 'int' and 'NoneType' Further compounding this is that MaxLengthValidator is decorated by (both directly and indirectly via BaseValidator ...?). So, baseline (as of ): In [1]: from import Value In [2]: x = Value('test') In [3]: %timeit x.resolveoutputfield() 1 µs ± 39.6 ns per loop (mean ± std. dev. of 7 runs, loops each) (Note: a previous run was faster at 7.6µs, so normal CPU workfload flux is in effect). We can see how much of the time is because of (see my comment here on a PR about deconstructible being a source to potentially optimise away) by just commenting it out from both validator classes: In [1]: from import Value In [2]: x = Value('test') In [3]: %timeit x.resolveoutputfield() 96 µs ± 130 ns per loop (mean ± std. dev. of 7 runs, loops each) But ignoring the class instantiation altogether is faster, easier and more correct at this juncture: In [1]: from import Value In [2]: x = Value('test') In [3]: %timeit x.resolveoutputfield() 86 µs ± 45.4 ns per loop (mean ± std. dev.


Question: def __init__(self, *args, db_collation=None, **kwargs):         super().__init__(*args, **kwargs)         self.db_collation = db_collation        self.validators.append(validators.MaxLengthValidator(self.max_length))        if self.max_length is not None:            self.validators.append(validators.MaxLengthValidator(self.max_length))      def check(self, **kwargs):         databases = kwargs.get('databases') or []

of 7 runs, loops each) So roughly a 2µs improvement. How do we get to that? Change the to: if self.maxlength is not None: (validators.MaxLengthValidator(self.max_length)) which incidentally and happily is the same process taken by for precedent. I have a branch locally with this change, and all existing tests currently pass. I'll push it to CI once I get a ticket number out of this submission, and see if it causes any issues elsewhere, and we can decide if it can be accepted from there.


Question: def add_field(self, model, field):         """Create a field on a model."""        # Fields with default values cannot by handled by ALTER TABLE ADD        # COLUMN statement because DROP DEFAULT is not supported in        # ALTER TABLE.        if not field.null or self.effective_default(field) is not None:        if (            # Primary keys and unique fields are not supported in ALTER TABLE            # ADD COLUMN.            field.primary_key or field.unique or            # Fields with default values cannot by handled by ALTER TABLE ADD            # COLUMN statement because DROP DEFAULT is not supported in            # ALTER TABLE.            not field.null or self.effective_default(field) is not None        ):             self._remake_table(model, create_field=field)         else:             super().add_field(model, field)

Adding nullable OneToOneField crashes on SQLite. Description This new sqlite3 error has cropped up between building django-oauth-toolkit between Django 4.0 and main branch for migrations.AddField of a OneToOneField (see ): self = object at 0x10b8038b0query = 'ALTER TABLE "oauth2provideraccesstoken" ADD COLUMN "sourcerefreshtokenid" bigint NULL UNIQUE REFERENCES "oauth2providerrefreshtoken" ("id") DEFERRABLE INITIALLY DEFERRED' params = [] def execute(self, query, params=None): if params is None: return Database.Cursor.execute(self, query) query = self.convertquery(query) E Cannot add a UNIQUE column Here's the relevant migration snippet: migrations.AddField( modelname='AccessToken', name='sourcerefreshtoken', field=models.OneToOneField(blank=True, null=True, , to=oauth2settings.REFRESHTOKENMODEL, relatedname="refreshedaccess_token"), ), I see there have been a lot of sqlite3 changes in since the 4.0 release....


Question: # Since we just bypassed this class' get_queryset(), we must manage             # the reverse relation manually.             for rel_obj in queryset:                instance = instances_dict[rel_obj_attr(rel_obj)]                setattr(rel_obj, self.field.name, instance)                if not self.field.is_cached(rel_obj):                    instance = instances_dict[rel_obj_attr(rel_obj)]                    setattr(rel_obj, self.field.name, instance)             cache_name = self.field.remote_field.get_cache_name()             return queryset, rel_obj_attr, instance_attr, False, cache_name, False

Deferred fields incorrect when following prefetches back to the "parent" object Description Given the following models: class User(models.Model): email = models.EmailField() kind = models.CharField( maxlength=10, choices=[("ADMIN", "Admin"), ("REGULAR", "Regular")] ) class Profile(models.Model): fullname = models.CharField(maxlength=255) user = models.OneToOneField(User, ondelete=models.CASCADE) I'd expect the following test case to pass: def testonlyrelatedqueryset(self): user = ( email="", kind="ADMIN", ) (user=user, fullname="Test Tester") queryset = ("email").prefetchrelated( Prefetch( "profile", queryset=Profile.objects.prefetchrelated( Prefetch("user", ("kind")) ), ) ) with self.assertNumQueries(3): user = () with self.assertNumQueries(0): self.assertEqual(, "ADMIN") The second assertNumQueries actually fails with: AssertionError: 1 != 0 : 1 queries executed, 0 expected Captured queries were: "testsuser"."id", "testsuser"."kind" FROM "testsuser" WHERE "testsuser"."id" = 1 This is exactly the query I'd expect to see if kind on the inner User queryset had been deferred, which it hasn't. The three queries executed when iterating the main queryset (ie when executing user = ()) look correct: "testsuser"."id", "testsuser"."email" FROM "testsuser" ORDER BY "testsuser"."id" ASC LIMIT 1 "testsprofile"."id", "testsprofile"."fullname", "testsprofile"."userid" FROM "testsprofile" WHERE "testsprofile"."userid" IN (1) "testsuser"."id", "testsuser"."kind" FROM "testsuser" WHERE "testsuser"."id" IN (1) Printing () returns {'kind'}. It looks to me like Django is correctly evaluating the set of deferred fields when executing the "inner" User queryset, but somehow the instances are inheriting the set of fields they "think" have been deferred from the outer User queryset, so when the attribute is accessed it causes a database query to be executed. It appears that this also happens if the relationship between Profile and User is a ForeignKey rather than a OneToOneField (in that case, a query is executed when accessing ()[0]).


Question: # Since we just bypassed this class' get_queryset(), we must manage             # the reverse relation manually.             for rel_obj in queryset:                instance = instances_dict[rel_obj_attr(rel_obj)]                setattr(rel_obj, self.field.name, instance)                if not self.field.is_cached(rel_obj):                    instance = instances_dict[rel_obj_attr(rel_obj)]                    setattr(rel_obj, self.field.name, instance)             cache_name = self.field.remote_field.get_cache_name()             return queryset, rel_obj_attr, instance_attr, False, cache_name, False

I'm happy to attempt to tackle this if someone can (a) confirm it's actually a bug and (b) point me in the right direction! Thanks :)


Question: obj = resolver_match.func          if hasattr(obj, 'view_class'):            caller = obj.view_class        elif hasattr(obj, '__name__'):            obj = obj.view_class        if hasattr(obj, '__name__'):             caller = obj.__name__         elif hasattr(obj, '__class__') and hasattr(obj.__class__, '__name__'):             caller = obj.__class__.__name__

Technical 404 debug page reported incorrect view name for CBVs. Description Technical 404 debug page reported incorrect view name for CBVs, e.g. Raised by: <class ''instead of Raised by: Regression in . Thanks Keryn Knight for the report.


Question: self.namespaces = [x for x in namespaces if x] if namespaces else []         self.namespace = ':'.join(self.namespaces) <ins>        if hasattr(func, 'view_class'):            func = func.view_class</ins>         if not hasattr(func, '__name__'):             # A class-based view             self._func_path = func.__class__.__module__ + '.' + func.__class__.__name__

ResolverMatch repr is incorrect for Class Based Views Description The following test applies cleanly to 3.2.9, and AFAIK would apply roughly correctly all the way back to when CBVs were introduced (I can't easily test without going back to a super old Python and finding the test cases, which have moved around): """ add to class: """ def testclassbasedrepr(self): self.assertEqual( repr(resolve('/redirect/')), "ResolverMatch(, " "args=(), kwargs={}, urlname=None, appnames=[], " "namespaces=[], route=redirect/)", ) The funcpath as AFAIK always been a representation to the fully qualified dotted callable where possible, that is for a CBV it's the CBV module + the class name. As of 4.0, the funcpath has become , I believe because of removing the use of updatewrapper and intentionally not setting the name and qualname in favour using the viewclass attribute, as per the comment viewclass should be used to robustly determine the name of the view (see Pull Request) Unfortunately I think that means the detection of class based views in ResolverMatch no longer works correctly, and this can probably only be resolved by making ResolverMatch CBV aware again by embedding detection of viewclass therein. Noted it as a question in this PR for ticket , but hoisting it here properly to be considered separately. The fix appears to ostensibly the same as for (see PR) class ResolverMatch: def init(...): # ... if hasattr(func, 'viewclass'): func = func.viewclass if not hasattr(func, 'name'): # ... I have a branch which I shall push shortly to confirm it works generally.


Question: return NotImplemented      def __hash__(self):        return hash((            self.creation_counter,            self.model._meta.app_label if hasattr(self, 'model') else None,            self.model._meta.model_name if hasattr(self, 'model') else None,        ))        return hash(self.creation_counter)      def __deepcopy__(self, memodict):         # We don't have to deepcopy very much here, since most things are not

Model () should be immutable. Description changes value when a field is assigned to a model class. This code crashes with an AssertionError: from import models f = models.CharField(max_length=200) d = {f: 1} class Book(models.Model): title = f assert f in d The bug was introduced in . It's unlikely to have been encountered because there are few use cases to put a field in a dict before it's assigned to a model class. But I found a reason to do so whilst implementing and the behaviour had me stumped for a little. IMO we can revert the hash change from . Objects with the same hash are still checked for equality, which was fixed in that ticket. But it's bad if an object's hash changes, since it breaks its use in dicts.


Question: for name, urlbase in ROLES.items():         create_reference_role(name, urlbase) # Match the beginning of a named or unnamed group.# Match the beginning of a named, unnamed, or non-capturing groups. named_group_matcher = _lazy_re_compile(r'\(\?P(<\w+>)') unnamed_group_matcher = _lazy_re_compile(r'\(')non_capturing_group_matcher = _lazy_re_compile(r'\(\?\:')   def replace_metacharacters(pattern):

simplifyregex() doesn't handle non-capturing groups Description (last modified by Mariusz Felisiak) While using Django REST Framework's Schema generator, I found out they're using simplifyregex(); however, current version has a few shortcomings, namely non-capturing groups are broken. simplifyregex() doesn't handle non-capturing groups Description (last modified by Mariusz Felisiak) While using Django REST Framework's Schema generator, I found out they're using simplifyregex(); however, current version has a few shortcomings, namely non-capturing groups are broken.


Question: final_pattern += pattern[:start] + '<var>'         prev_end = end     return final_pattern + pattern[prev_end:]def remove_non_capturing_groups(pattern):    r"""    Find non-capturing groups in the given `pattern` and remove them, e.g.    1. (?P\w+)/b/(?:\w+)c(?:\w+) => (?P\\w+)/b/c    2. ^(?:\w+(?:\w+))a => ^a    3. ^a(?:\w+)/b(?:\w+) => ^a/b    """    group_start_end_indices = _find_groups(pattern, non_capturing_group_matcher)    final_pattern, prev_end = '', None    for start, end, _ in group_start_end_indices:        final_pattern += pattern[prev_end:start]        prev_end = end    return final_pattern + pattern[prev_end:]diff --git a/django/contrib/admindocs/views.py b/django/contrib/admindocs/views.py-- a/django/contrib/admindocs/views.py++ b/django/contrib/admindocs/views.py


Question: from django.contrib.admin.views.decorators import staff_member_required from django.contrib.admindocs import utils from django.contrib.admindocs.utils import (    replace_metacharacters, replace_named_groups, replace_unnamed_groups,    remove_non_capturing_groups, replace_metacharacters, replace_named_groups,    replace_unnamed_groups, ) from django.core.exceptions import ImproperlyConfigured, ViewDoesNotExist from django.db import models


Question: example, turn "^(?P<sport_slug>\w+)/athletes/(?P<athlete_slug>\w+)/$"     into "/<sport_slug>/athletes/<athlete_slug>/".     """<ins>    pattern = remove_non_capturing_groups(pattern)</ins>     pattern = replace_named_groups(pattern)     pattern = replace_unnamed_groups(pattern)     pattern = replace_metacharacters(pattern)


Question: self.core_filters = {self.field.name: instance} <ins>            # Even if this relation is not to pk, we require still pk value.            # The wish is that the instance has been already saved to DB,            # although having a pk value isn't a guarantee of that.            if self.instance.pk is None:                raise ValueError(                    f"{instance.__class__.__name__!r} instance needs to have a primary "                    f"key value before this relationship can be used."                )</ins>         def __call__(self, *, manager):             manager = getattr(self.model, manager)             manager_class = create_reverse_many_to_one_manager(manager.__class__, rel)

Unify reverse foreign key and m2m unsaved model querying Description Currently when querying unsaved reverse relations the behavior differs. Using model: class Foo(models.Model): fk = models.ForeignKey('self', relatedname='fkrev') m2m = models.ManyToManyField('self') and test case: print(Foo()()) print(Foo()()) We get [] from the first filter, but an error ValueError: "<Foo: Foo object" needs to have a value for field "fromfoo" before this many-to-many relationship can be used. from the second filter. So, m2m fields can't be filtered if the object isn't saved, but reverse fk fields can be filtered. There is a (slightly stale) patch for which makes fk fields and m2m fields work consistently. The changes in behavior are: Nullable many-to-many and foreign key relations will return an empty queryset when the relation field is null. For many-to-many this was previously an error (no change for foreign keys). Trying to add objects to a foreign key relation when the relation field is null is now an error (no change for m2m). Trying to use a relation of any kind when the object isn't saved is now an error (no change for m2m). I think we can squeeze these changes in as bug-fixes. These are slight backwards compatibility changes, but to me it seems that almost always the changes will be visible only in code that isn't working as intended. If these are seen as something likely breaking working code, then I don't see any way to make the APIs consistent. The patch is available from:


Question: do_not_call_in_templates = True <ins>        def _check_fk_val(self):            for field in self.field.foreign_related_fields:                if getattr(self.instance, field.attname) is None:                    raise ValueError(                        f'"{self.instance!r}" needs to have a value for field '                        f'"{field.attname}" before this relationship can be used.'                    )</ins>         def _apply_rel_filters(self, queryset):             """             Filter the queryset for the instance this manager is bound to.


Question: return queryset, rel_obj_attr, instance_attr, False, cache_name, False          def add(self, *objs, bulk=True):<ins>            self._check_fk_val()</ins>             self._remove_prefetched_objects()             db = router.db_for_write(self.model, instance=self.instance)


Question: add.alters_data = True          def create(self, **kwargs):<ins>            self._check_fk_val()</ins>             kwargs[self.field.name] = self.instance             db = router.db_for_write(self.model, instance=self.instance)             return super(RelatedManager, self.db_manager(db)).create(**kwargs)


Question: create.alters_data = True          def get_or_create(self, **kwargs):<ins>            self._check_fk_val()</ins>             kwargs[self.field.name] = self.instance             db = router.db_for_write(self.model, instance=self.instance)             return super(RelatedManager, self.db_manager(db)).get_or_create(**kwargs)


Question: get_or_create.alters_data = True          def update_or_create(self, **kwargs):<ins>            self._check_fk_val()</ins>             kwargs[self.field.name] = self.instance             db = router.db_for_write(self.model, instance=self.instance)             return super(RelatedManager, self.db_manager(db)).update_or_create(**kwargs)


Question: def remove(self, *objs, bulk=True):                 if not objs:                     return<ins>                self._check_fk_val()</ins>                 val = self.field.get_foreign_related_value(self.instance)                 old_ids = set()                 for obj in objs:


Question: remove.alters_data = True              def clear(self, *, bulk=True):<ins>                self._check_fk_val()</ins>                 self._clear(self, bulk)              clear.alters_data = True


Question: _clear.alters_data = True          def set(self, objs, *, bulk=True, clear=False):<ins>            self._check_fk_val()</ins>             # Force evaluation of `objs` in case it's a queryset whose value             # could be affected by `manager.clear()`. Refs #19816.             objs = tuple(objs)


Question: def __init__(self, queryset, output_field=None, **extra):         # Allow the usage of both QuerySet and sql.Query objects.        self.query = getattr(queryset, 'query', queryset)        self.query = getattr(queryset, 'query', queryset).clone()        self.query.subquery = True         self.extra = extra         super().__init__(output_field)

() generates invalid SQL. Description (last modified by M1ha Shvn) Since this commit (...) method returns incorrect SQL removing first and last symbols instead of absent breakets. Adding Subquery().query.subquery = True attribute fixes the problem. From my point of view, it should be set in Subquery constructor. from import connection from import App q = Subquery(()) print(str(q.query)) print(q.assql(q.query.getcompiler('default'), connection)) q.query.subquery = True print(q.assql(q.query.getcompiler('default'), connection)) ('(SELECT "appsapp"."id", "appsapp"."name" FROM "apps_app")', ())


Question: file_name = html.unescape(file_name)         file_name = file_name.rsplit('/')[-1]         file_name = file_name.rsplit('\\')[-1]<ins>        # Remove non-printable characters.        file_name = ''.join([char for char in file_name if char.isprintable()])</ins>          if file_name in {'', '.', '..'}:             return None

File upload crash when a file extension contains null characters. Description A 2.5M file uploaded with a raw null byte anyplace after the . in its filename means that Django attempts to create a tempfile with that same "extension," which errors out with ValueError: embedded null byte. It's almost certainly a violation of RFC to have a filename with a raw null byte in it, but it shouldn't result in a 500 when parsing the form. Here's code to generate a bad request: #!/usr/bin/env python3 import io import requests contents = io.StringIO("." (1024 1024 * 3)) files = {"docfile": (b"!", contents, "text/plain")} req = requests.Request("POST", "http://localhost:8000/", files=files, data={}) prepared = req.prepare() body = assert isinstance(body, bytes) = body.replace(b"!", b"\x00") requests.Session().send(prepared) produces an error with the view: from django import forms from import HttpResponseRedirect from django.shortcuts import render from import csrfexempt class UploadFileForm(forms.Form): docfile = forms.FileField() def index(request): if == 'POST': form = UploadFileForm(request.POST, request.FILES) if form.is_valid(): print(repr(request.FILES['docfile'])) return HttpResponseRedirect('/') else: print("Not valid!") return HttpResponseRedirect('/') else: form = UploadFileForm() return render(request, '', {'form': form}) I'm not sure what the goal is of preserving the "extension" of the uploaded file in the tempfile that is made; if that's important enough a behaviour to keep, some amount of escaping on the parsed-out extension may be necessary.


Question: import copy import operator import warningsfrom itertools import chainfrom itertools import chain, islice  import django from django.conf import settings

Support prefetchrelated() with Queryset.iterator() Description (last modified by Asif Saifuddin Auvi) I was surprised when I found out that prefetchrelated calls are ignored when using Queryset.iterator. I noticed in the docs here  that it is because the "these two optimizations do not make sense together." That may have been true in the past, but it is definitely not the case now. The iterator allows chunking (by default 2000) and it would be very helpful to prefetch related for each chunk.


Question: from django.db.models.sql.constants import CURSOR, GET_ITERATOR_CHUNK_SIZE from django.db.models.utils import create_namedtuple_class, resolve_callables from django.utils import timezone<ins>from django.utils.deprecation import RemovedInDjango50Warning</ins> from django.utils.functional import cached_property, partition  # The maximum number of results to fetch in a get() query.


Question: ####################################      def _iterator(self, use_chunked_fetch, chunk_size):        yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)        iterable = self._iterable_class(            self,            chunked_fetch=use_chunked_fetch,            chunk_size=chunk_size or 2000,        )        if not self._prefetch_related_lookups or chunk_size is None:            yield from iterable            return        iterator = iter(iterable)        while results := list(islice(iterator, chunk_size)):            prefetch_related_objects(results, *self._prefetch_related_lookups)            yield from results     def iterator(self, chunk_size=2000):    def iterator(self, chunk_size=None):         """         An iterator over the results from applying this QuerySet to the        database.        """        if chunk_size <= 0:        database. chunk_size must be provided for QuerySets that prefetch        related objects. Otherwise, a default chunk_size of 2000 is supplied.        """        if chunk_size is None:            if self._prefetch_related_lookups:                # When the deprecation ends, replace with:                # raise ValueError(                #     'chunk_size must be provided when using '                #     'QuerySet.iterator() after prefetch_related().'                # )                warnings.warn(                    'Using QuerySet.iterator() after prefetch_related() '                    'without specifying chunk_size is deprecated.',                    category=RemovedInDjango50Warning,                    stacklevel=2,                )        elif chunk_size <= 0:             raise ValueError('Chunk size must be strictly positive.')         use_chunked_fetch = not connections[self.db].settings_dict.get('DISABLE_SERVER_SIDE_CURSORS')         return self._iterator(use_chunked_fetch, chunk_size)


Question: return self.name.lower()      def get_field(self, field_name):        field_name = (            self.options['order_with_respect_to']            if field_name == '_order'            else field_name        )        if field_name == '_order':            field_name = self.options.get('order_with_respect_to', field_name)         return self.fields[field_name]      @classmethod

Migration autodetector crashes on models with field named order, but not using orderwithrespectto. Description (last modified by Fabian Büchler) The commit  introduced a new function ModelState.getfield in This converts the field name order to the one defined in options['orderwithrespectto'] automatically, which fails if the model has a field order but isn't using Meta.orderwithrespectto. That is the case for models generated by django-simple-history () for models that are originally using Meta.orderwithrespectto: the resulting historical records model has only order but is not using the Meta option.


Question: return self.name.lower()      def get_field(self, field_name):        field_name = (            self.options['order_with_respect_to']            if field_name == '_order'            else field_name        )        if field_name == '_order':            field_name = self.options.get('order_with_respect_to', field_name)         return self.fields[field_name]      @classmethod

This shows when running migrate or makemigrations: $ makemigrations --dry-run Waiting for port 'mysql:3306' timeout 1s (attempt 1/60) Port 'mysql:3306' is open Traceback (most recent call last): File "", line 42, in <modulemain() File "", line 36, in main executefromcommandline() File "/usr/local/lib/python3.8/site-", line 425, in executefromcommandline utility.execute() File "/usr/local/lib/python3.8/site-", line 419, in execute self.fetchcommand(subcommand).runfromargv() File "/usr/local/lib/python3.8/site-", line 373, in runfromargv self.execute(args, cmdoptions) File "/usr/local/lib/python3.8/site-", line 417, in execute output = (args, options) File "/usr/local/lib/python3.8/site-", line 90, in wrapped res = handlefunc(args, *kwargs) File "/usr/local/lib/python3.8/site-", line 172, in handle changes = autodetector.changes( File "/usr/local/lib/python3.8/site-", line 43, in changes changes = self.detectchanges(convertapps, graph) File "/usr/local/lib/python3.8/site-", line 189, in detectchanges self.generatealteredfields() File "/usr/local/lib/python3.8/site-", line 928, in generatealteredfields oldfield = [applabel, oldmodelname].getfield(oldfieldname) File "/usr/local/lib/python3.8/site-", line 689, in getfield self.options['orderwithrespectto'] KeyError: 'orderwithrespectto' I believe this could be solved using a bit more defensive code, like: def getfield(self, fieldname): if fieldname == 'order' and 'orderwithrespectto' in self.options: fieldname = self.options['orderwithrespectto'] return [fieldname]


Question: # Using 0/1 here instead of False/True to produce more compact json             is_safedata = 1 if isinstance(obj.message, SafeData) else 0             message = [self.message_key, is_safedata, obj.level, obj.message]            if obj.extra_tags:            if obj.extra_tags is not None:                 message.append(obj.extra_tags)             return message         return super().default(obj)

Messages framework incorrectly serializes/deserializes extratags when it's an empty string Description When a message is serialised and then deserialised with any of the built in storage backends, then extratags=="" is converted to extratags==None. This is because MessageEncoder checks for the truthyness of extratags rather than checking it is not None. To replicate this bug from import settings settings.configure() # Just to allow the following import from import Message from import MessageEncoder, MessageDecoder originalmessage = Message(10, "Here is a message", extratags="") encodedmessage = MessageEncoder().encode(originalmessage) decodedmessage = MessageDecoder().decode(encodedmessage) originalmessage.extratags == "" True decodedmessage.extratags is None True Effect of the bug in application behaviour This error occurred in the wild with a template tag similar to the following: {% if x not in message.extratags %} When the message was displayed as part of a redirect, it had been serialised and deserialized which meant that extratags was None instead of the empty string. This caused an error. It's important to note that this bug affects all of the standard API (, etc. all have a default value of extra_tags equal to "").


Question: cleansed_substitute = "********************"     hidden_settings = _lazy_re_compile(        "API|TOKEN|KEY|SECRET|PASS|SIGNATURE", flags=re.I        "API|TOKEN|KEY|SECRET|PASS|SIGNATURE|HTTP_COOKIE", flags=re.I     )      def cleanse_setting(self, key, value):

Session ID should be cleansed from error reporting Description the session ID should be cleansed when reporting errors, just like other credentials. A patch is available at  See also and  A quick github search yielded multiple occasions where session IDs ended up in public bug reports:     I am sure you could find many more. This could potentially be exploited by automatically searching for such requests and hijacking the associated accounts.


Question: Cleanse an individual setting key/value of sensitive content. If the         value is a dictionary, recursively cleanse the keys in that dictionary.         """        try:            is_sensitive = self.hidden_settings.search(key)        except TypeError:            is_sensitive = False        if key == settings.SESSION_COOKIE_NAME:            is_sensitive = True        else:            try:                is_sensitive = self.hidden_settings.search(key)            except TypeError:                is_sensitive = False          if is_sensitive:             cleansed = self.cleansed_substitute


Question: return {}         return {k: self.cleanse_setting(k, v) for k, v in request.META.items()} <ins>    def get_safe_cookies(self, request):        """        Return a dictionary of request.COOKIES with sensitive values redacted.        """        if not hasattr(request, "COOKIES"):            return {}        return {k: self.cleanse_setting(k, v) for k, v in request.COOKIES.items()}</ins>     def is_active(self, request):         """         This filter is to add safety in production environments (i.e. DEBUG


Question: "frames": frames,             "request": self.request,             "request_meta": self.filter.get_safe_request_meta(self.request),<ins>            "request_COOKIES_items": self.filter.get_safe_cookies(self.request).items(),</ins>             "user_str": user_str,             "filtered_POST_items": list(                 self.filter.get_post_parameters(self.request).items()


Question: if self.request is not None:             c["request_GET_items"] = self.request.GET.items()             c["request_FILES_items"] = self.request.FILES.items()<del>            c["request_COOKIES_items"] = self.request.COOKIES.items()</del>             c["request_insecure_uri"] = self._get_raw_insecure_uri()             c["raising_view_name"] = get_caller(self.request)


Question: from django.db.models import AutoField, DateField, DateTimeField, sql from django.db.models.constants import LOOKUP_SEP, OnConflict from django.db.models.deletion import Collectorfrom django.db.models.expressions import Case, Expression, F, Ref, Value, Whenfrom django.db.models.expressions import Case, F, Ref, Value, When from django.db.models.functions import Cast, Trunc from django.db.models.query_utils import FilteredRelation, Q from django.db.models.sql.constants import CURSOR, GET_ITERATOR_CHUNK_SIZE

bulkupdate() does not work with plain F('...') expressions. Description Repro: assign plain F(...) to some model instance field save with bulkupdate Example: Code highlighting: from import SelfRef o = ().first() o.c8 = F('name') # model has char fields 'c8' and 'name' SelfRef.objects.bulkupdate([o], ['c8']) 1 o.refreshfromdb() o.c8 'F(name)' from import connection connection.queries[-2] {'sql': 'UPDATE "exampleappselfref" SET "c8" = CASE WHEN ("exampleappselfref"."id" = ) THEN \'F(name)\' ELSE NULL END WHERE "exampleappselfref"."id" IN ()', 'time': '0.001'} The created SQL contains the string repr of F(), instead of resolving to the column name. Looking at the source code, the culprit seems to be a too narrow type check in  It works, if the type check gets replaced by one of these: Code highlighting: if not hasattr(attr, 'resolve_expression'): ... if not isinstance(attr, (Expression, F)): ...


Question: when_statements = []                 for obj in batch_objs:                     attr = getattr(obj, field.attname)                    if not isinstance(attr, Expression):                    if not hasattr(attr, 'resolve_expression'):                         attr = Value(attr, output_field=field)                     when_statements.append(When(pk=obj.pk, then=attr))                 case_statement = Case(*when_statements, output_field=field)


Question: class SafeData:<ins>    __slots__ = ()</ins>     def __html__(self):         """         Return the html representation of a string for interoperability.

Introduce empty slots protocol for SafeString & SafeData Description (last modified by Keryn Knight) This is a case-by-case proposal ultimately referencing Because SafeString is used a lot and is otherwise supposed to be treatable as a untainted str we should be able to (AFAIK) update it + it's inheritance chain to use slots = () whilst still allowing custom subclasses of either to add additional attributes. By defining slots as empty on SafeString (and SafeData) we'd avoid creation of a dict on the instance, which mirrors the str() behaviour. According to pympler, currently in Python 3.10 using the following back of the napkins strings: In [4]: s = "test" # this might be interned, as a short string? In [5]: s2 = "test" 100 In [6]: s3 = SafeString("test") In [7]: s4 = SafeString("test" 100) we get: In [8]: asizeof(s) # str Out[8]: 56 In [9]: asizeof(s2) # str Out[9]: 456 In [10]: asizeof(s3) # SafeString Out[10]: 208 In [11]: asizeof(s4) # SafeString Out[11]: 608 But if we swap out the implementation to be slots'd, it looks more like: In [8]: asizeof(s) # str Out[8]: 56 In [9]: asizeof(s2) # str Out[9]: 456 In [10]: asizeof(s3) # SafeString Out[10]: 104 In [11]: asizeof(s4) # SafeString Out[11]: 504 So we're "saving" 104 bytes per SafeString created, by the look of it. I presume it to be some fun implementation detail of something somewhere that it is allegedly accounting for more than 64 bytes, which is the asizeof({}) A quick and dirty check over the test suite suggests that for me locally, running tests in 512.912s accounted for 949.0 MB of SafeStrings, checked by just incrementing a global integer of bytes (using and --parallel=1) and piping that to filesizeformat, so y'know, room for error.


Question: class SafeData:<ins>    __slots__ = ()</ins>     def __html__(self):         """         Return the html representation of a string for interoperability.

After the patch, the same tests accounted for 779.4 MB of SafeString, "saving" 170 MB overall. The only functionality this would preclude -- as far as I know -- is no longer being able to bind arbitrary values to an instance like so: s = SafeString('test') s.test = 1 which would raise AttributeError if slots were , just like trying to assign attributes to str() directly does. I don't believe this will have any marked performance change, as neither SafeString nor SafeData actually have any extra attributes, only methods. I have a branch which implements this, and tests pass for me locally.


Question: A str subclass that has been specifically marked as "safe" for HTML output     purposes.     """<ins>    __slots__ = ()</ins>     def __add__(self, rhs):         """         Concatenating a safe string with another safe bytestring or


Question: A str subclass that has been specifically marked as "safe" for HTML output     purposes.     """<ins>    __slots__ = ()</ins>     def __add__(self, rhs):         """         Concatenating a safe string with another safe bytestring or


Question: if hasattr(default, 'resolve_expression'):             default = default.resolve_expression(query, allow_joins, reuse, summarize)         c.default = None  # Reset the default argument before wrapping.        return Coalesce(c, default, output_field=c._output_field_or_none)        coalesce = Coalesce(c, default, output_field=c._output_field_or_none)        coalesce.is_summary = c.is_summary        return coalesce      @property     def default_alias(self):

aggregate() with 'default' after annotate() crashes. Description I saw this on a PostgreSQL project and reproduced it with SQLite. Django 4.0.1. Annotate (anything) then aggregate works fine: $ shell Python 3.10.2 (main, Jan 21 2022, 19:45:54) [Clang 13.0.0 (clang-1300.0.29.30)] Type 'copyright', 'credits' or 'license' for more information IPython 7.30.1 -- An enhanced Interactive Python. Type '?' for help. In [1]: from import In [2]: from import In [3]: from import * In [4]: () Out[4]: 95 In [5]: Book.objects.annotate(idx=F("id")).aggregate(Sum("id")) Out[5]: {'idsum': 4560} But add the aggregate classes’ default argument (new in 4.0), and it breaks: In [6]: Book.objects.annotate(idx=F("id")).aggregate(Sum("id", default=0)) OperationalError Traceback (most recent call last) ... OperationalError: near "FROM": syntax error The generated SQL: In [7]: %debug 414 return Database.Cursor.execute(self, query) 415 query = self.convertquery(query) --416 return Database.Cursor.execute(self, query, params) 417 418 def executemany(self, query, paramlist): ipdbquery 'SELECT FROM (SELECT "corebook"."id" AS "idx", COALESCE(SUM("corebook"."id"), ?) AS "idsum" FROM "core_book") subquery' ipdbparams (0,) ipdbThe “long form” using Coalesce works: In [8]: Book.objects.annotate(idx=F("id")).aggregate(x=Coalesce(Sum("id"), 0)) Out[8]: {'x': 4560}


Question: for app_label, model_name, field_name in sorted(self.new_field_keys - self.old_field_keys):             old_model_name = self.renamed_models.get((app_label, model_name), model_name)             old_model_state = self.from_state.models[app_label, old_model_name]            new_model_state = self.to_state.models[app_label, old_model_name]            new_model_state = self.to_state.models[app_label, model_name]             field = new_model_state.get_field(field_name)             # Scan to see if this is actually a rename!             field_dec = self.deep_deconstruct(field)

Migration autodetector crashes when renaming a model and field. Description Migration autodetector crashes when renaming a model and field in a single step: $ python makemigrations Did you rename the testone.MyModel model to MyModel2? [y/N] y Traceback (most recent call last): File "", line 22, in <modulemain() File "", line 18, in main executefromcommandline() File "", line 419, in executefromcommandline utility.execute() File "", line 413, in execute self.fetchcommand(subcommand).runfromargv() File "", line 354, in runfromargv self.execute(args, cmdoptions) File "", line 398, in execute output = (args, options) File "", line 89, in wrapped res = handlefunc(args, *kwargs) File "", line 172, in handle changes = autodetector.changes( File "", line 43, in changes changes = self.detectchanges(convertapps, graph) File "", line 182, in detectchanges self.generaterenamedfields() File "", line 823, in generaterenamedfields newmodelstate = [applabel, oldmodelname] KeyError: ('testone', 'mymodel') Reported by HoskeOwl. Regression in .


Question: def as_sql(self, compiler, connection, template=None, **extra_context):         query = self.query.exists(using=connection.alias)        sql, params = super().as_sql(            compiler,            connection,            template=template,            query=query,            **extra_context,        )        try:            sql, params = super().as_sql(                compiler,                connection,                template=template,                query=query,                **extra_context,            )        except EmptyResultSet:            if self.negated:                return '', ()            raise         if self.negated:             sql = 'NOT {}'.format(sql)         return sql, params

filter on exists-subquery with empty queryset removes whole WHERE block Description (last modified by Tobias Bengfort) qs = (~models.Exists(()), name='test') qs <QuerySet []print() EmptyResultSet With django-debug-toolbar I can still see the query, but there WHERE block is missing completely. This seems to be very similar to .


Question: @receiver(file_changed, dispatch_uid='template_loaders_file_changed') def template_changed(sender, file_path, **kwargs):<ins>    if file_path.suffix == '.py':        return</ins>     for template_dir in get_template_directories():         if template_dir in file_path.parents:             reset_loaders()

Dev Server fails to restart after adding BASEDIR to TEMPLATES[0]['DIRS'] in settings Description Repro steps: $ pip install -U django $ django-admin startproject <nameOpen , copy the BASEDIR variable from line 16 and paste it into the empty DIRS list on line 57 $ runserver Back in your IDE, save a file and watch the dev server NOT restart. Back in , remove BASE_DIR from the templates DIRS list. Manually CTRL-C your dev server (as it won't restart on its own when you save), restart the dev server. Now return to your file, re-save it, and notice the development server once again detects changes and restarts. This bug prevents the dev server from restarting no matter where you make changes - it is not just scoped to edits to


Question: return result         return copy.deepcopy(self._wrapped, memo) <ins>    __add__ = new_method_proxy(operator.add)    @new_method_proxy    def __radd__(self, other):        return other + self</ins>  def partition(predicate, values):     """

SimpleLazyObject doesn't implement radd Description Technically, there's a whole bunch of magic methods it doesn't implement, compared to a complete proxy implementation, like that of wrapt.ObjectProxy, but radd being missing is the one that's biting me at the moment. As far as I can tell, the implementation can't just be radd = newmethodproxy() because that doesn't exist, which is rubbish. radd = newmethodproxy(operator.attrgetter("radd")) also won't work because types may not have that attr, and attrgetter doesn't supress the exception (correctly) The minimal implementation I've found that works for me is: def radd(self, other): if self.wrapped is empty: () return other + self._wrapped


Question: <ins>import warnings</ins> from django.db.models.lookups import (     Exact,     GreaterThan,

Deprecate passing unsaved objects to related filters. Description Consider this filter: (relatedobj=bar) Where 'bar' is an unsaved object instance. In Django 1.11, this would always return an empty QuerySet (since no Foo object is related to unsaved 'bar'). In Django 2.0 through 2.2, this is equivalent to doing (which can return a non-empty QuerySet): (relatedobj=None) I found a somewhat related issue that touches on this subject: My questions: What is the intended behaviour? In the aforementioned issue Simon Charette suggests that unsaved objects should be prevented from being used in related filters. I agree with that. Is this documented anywhere? I couldn't find anything. At the very least this should be documented somewhere.


Question: LessThan,     LessThanOrEqual, )<ins>from django.utils.deprecation import RemovedInDjango50Warning</ins>   class MultiColSource:


Question: from django.db.models import Model      if isinstance(value, Model):<ins>        if value.pk is None:            # When the deprecation ends, replace with:            # raise ValueError(            #     "Model instances passed to related filters must be saved."            # )            warnings.warn(                "Passing unsaved model instances to related filters is deprecated.",                RemovedInDjango50Warning,            )</ins>         value_list = []         sources = lhs.output_field.path_infos[-1].target_fields         for source in sources:


Question: if is_async:             if not method_is_async:                 if debug:                    logger.debug("Synchronous %s adapted.", name)                    logger.debug("Synchronous handler adapted for %s.", name)                 return sync_to_async(method, thread_sensitive=True)         elif method_is_async:             if debug:                logger.debug("Asynchronous %s adapted.", name)                logger.debug("Asynchronous handler adapted for %s.", name)             return async_to_sync(method)         return method

"Synchronous middleware ... adapted" is not logged for synchronous middleware Description This is a bug in log messages that contradicts the guide in a warning in the docs; the feature works. The guide at , emphasis mine: Warning You will only get the benefits of a fully-asynchronous request stack if you have no synchronous middleware loaded into your site. If there is a piece of synchronous middleware, then Django must use a thread per request to safely emulate a synchronous environment for it. Middleware can be built to support both sync and async contexts. Some of Django’s middleware is built like this, but not all. To see what middleware Django has to adapt, you can turn on debug logging for the django.request logger and look for log messages about “Synchronous middleware … adapted”. The test for "Synchronous middleware ... adapted" is instead testing an async middleware over a sync method:  'middlewareexceptions.middleware.asyncpaymentmiddleware', ]) def testasyncmiddleware(self): with self.assertLogs('django.request', 'DEBUG') as cm: response = ('/middlewareexceptions/view/') self.assertEqual(response.statuscode, 402) self.assertEqual( cm.records[0].getMessage(), "Synchronous middleware " "middlewareexceptions.middleware.asyncpaymentmiddleware " "adapted.", ) About the existing implementation: BaseHandler.loadmiddleware passes the middleware name as name to BaseHandler.adaptmethodmode:  adaptedhandler = self.adaptmethodmode( middlewareisasync, handler, handlerisasync, debug=settings.DEBUG, name='middleware %s' % middlewarepath, ) BaseHandler.adaptmethodmode adapts the method and treats name as the method name rather than the middleware name; when the middleware name is used, it implies a method has been adapted for the middleware, not that the middleware was adapted: if debug and not name: name = name or 'method %s()' % method.qualname if isasync: if not methodisasync: if debug: ('Synchronous %s adapted.', name) return synctoasync(method, threadsensitive=True) elif methodisasync: if debug: ('Asynchronous %s adapted.


Question: if is_async:             if not method_is_async:                 if debug:                    logger.debug("Synchronous %s adapted.", name)                    logger.debug("Synchronous handler adapted for %s.", name)                 return sync_to_async(method, thread_sensitive=True)         elif method_is_async:             if debug:                logger.debug("Asynchronous %s adapted.", name)                logger.debug("Asynchronous handler adapted for %s.", name)             return async_to_sync(method)         return method

', name) return asynctosync(method) Proposed fix: Handle middleware name and method name separately within BaseHandler.adaptmethodmode: def adaptmethodmode( self, isasync, method, methodisasync=None, debug=False, name=None, ): """ Adapt a method to be in the correct "mode": If isasync is False: Synchronous methods are left alone Asynchronous methods are wrapped with asynctosync If isasync is True: Synchronous methods are wrapped with synctoasync() Asynchronous methods are left alone """ methodname = None if methodisasync is None: methodisasync = asyncio.iscoroutinefunction(method) if debug and not name: name = name or 'method %s()' % method.qualname methodname = 'method %s()' % method.qualname if isasync: if not methodisasync: if debug: ('Synchronous %s adapted.', name) if name: ('Asynchronous %s adapted.', name) else: ('Synchronous %s adapted.', methodname) return synctoasync(method, threadsensitive=True) elif methodisasync: if debug: ('Asynchronous %s adapted.', name) if name: ('Synchronous %s adapted.', name) else: ('Asynchronous %s adapted.', methodname) return asynctosync(method)


Question: )         # PK is used twice in the resulting update query, once in the filter         # and once in the WHEN. Each field will also have one CAST.<ins>        self._for_write = True</ins>         connection = connections[self.db]         max_batch_size = connection.ops.bulk_batch_size(["pk", "pk"] + fields, objs)         batch_size = min(batch_size, max_batch_size) if batch_size else max_batch_size

QuerySet.bulkupdate() should perform atomic writes against routed dbforwrite. Description Discovered in but the summary is that bulkupdate doesn't set self.forwrite = True prior to accessing so the latter is actually dbforread and that includes the block creation.


Question: update_kwargs[field.attname] = case_statement             updates.append(([obj.pk for obj in batch_objs], update_kwargs))         rows_updated = 0        queryset = self.using(self.db)         with transaction.atomic(using=self.db, savepoint=False):             for pks, update_kwargs in updates:                rows_updated += self.filter(pk__in=pks).update(**update_kwargs)                rows_updated += queryset.filter(pk__in=pks).update(**update_kwargs)         return rows_updated      bulk_update.alters_data = True


Question: import multiprocessing import os import shutilimport sqlite3 import sys from pathlib import Path from django.db import NotSupportedError from django.db.backends.base.creation import BaseDatabaseCreation

Allow parallel test runner to work with Windows/macOS process start method. Description (last modified by Brandon Navra) Python 3.8 on MacOS has changed the default start method for the multiprocessing module from fork to spawn:  When running tests with the --parallel flag, this causes the worker processes to fail with Apps aren't loaded yet. as they no longer have a copy of the parent memory state. It can also cause the workers to fail to find the cloned dbs ( {{ FATAL: database "xxx1" does not exist}} ) as the db test prefix is missing. I have attached a patch which changes (the worker initialiser for ParallelTestSuite) to run () and set the db name to one with the test prefix.


Question: def get_test_db_clone_settings(self, suffix):         orig_settings_dict = self.connection.settings_dict         source_database_name = orig_settings_dict["NAME"]        if self.is_in_memory_db(source_database_name):        if not self.is_in_memory_db(source_database_name):            root, ext = os.path.splitext(source_database_name)            return {**orig_settings_dict, "NAME": f"{root}_{suffix}{ext}"}        start_method = multiprocessing.get_start_method()        if start_method == "fork":             return orig_settings_dict        else:            root, ext = os.path.splitext(orig_settings_dict["NAME"])            return {**orig_settings_dict, "NAME": "{}_{}{}".format(root, suffix, ext)}        if start_method == "spawn":            return {                **orig_settings_dict,                "NAME": f"{self.connection.alias}_{suffix}.sqlite3",            }        raise NotSupportedError(            f"Cloning with start method {start_method!r} is not supported."        )      def _clone_test_db(self, suffix, verbosity, keepdb=False):         source_database_name = self.connection.settings_dict["NAME"]         target_database_name = self.get_test_db_clone_settings(suffix)["NAME"]        # Forking automatically makes a copy of an in-memory database.         if not self.is_in_memory_db(source_database_name):             # Erase the old test database             if os.access(target_database_name, os.F_OK):


Question: except Exception as e:                 self.log("Got an error cloning the test database: %s" % e)                 sys.exit(2)<ins>        # Forking automatically makes a copy of an in-memory database.        # Spawn requires migrating to disk which will be re-opened in        # setup_worker_connection.        elif multiprocessing.get_start_method() == "spawn":            ondisk_db = sqlite3.connect(target_database_name, uri=True)            self.connection.connection.backup(ondisk_db)</ins>      def _destroy_test_db(self, test_database_name, verbosity):         if test_database_name and not self.is_in_memory_db(test_database_name):


Question: else:             sig.append(test_database_name)         return tuple(sig)    def setup_worker_connection(self, _worker_id):        settings_dict = self.get_test_db_clone_settings(_worker_id)        # connection.settings_dict must be updated in place for changes to be        # reflected in django.db.connections. Otherwise new threads would        # connect to the default database instead of the appropriate clone.        start_method = multiprocessing.get_start_method()        if start_method == "fork":            # Update settings_dict in place.            self.connection.settings_dict.update(settings_dict)            self.connection.close()        elif start_method == "spawn":            alias = self.connection.alias            connection_str = (                f"file:memorydb_{alias}_{_worker_id}?mode=memory&cache=shared"            )            source_db = self.connection.Database.connect(                f"file:{alias}_{_worker_id}.sqlite3", uri=True            )            target_db = sqlite3.connect(connection_str, uri=True)            source_db.backup(target_db)            source_db.close()            # Update settings_dict in place.            self.connection.settings_dict.update(settings_dict)            self.connection.settings_dict["NAME"] = connection_str            # Re-open connection to in-memory database before closing copy            # connection.            self.connection.connect()            target_db.close()            if os.environ.get("RUNNING_DJANGOS_TEST_SUITE") == "true":                self.mark_expected_failures_and_skips()diff --git a/django/utils/autoreload.py b/django/utils/autoreload.py-- a/django/utils/autoreload.py++ b/django/utils/autoreload.py


Question: # cause issues here.         if not isinstance(module, ModuleType):             continue        if module.__name__ == "__main__":        if module.__name__ in ("__main__", "__mp_main__"):             # __main__ (usually manage.py) doesn't always have a __spec__ set.             # Handle this by falling back to using __file__, resolved below.             # See https://docs.python.org/reference/import.html#main-spec


Question: def USE_L10N(self):         stack = traceback.extract_stack()         # Show a warning if the setting is used outside of Django.        # Stack index: -1 this line, -2 the caller.        filename, _, _, _ = stack[-2]        # Stack index: -1 this line, -2 the LazyObject __getattribute__(),        # -3 the caller.        filename, _, _, _ = stack[-3]         if not filename.startswith(os.path.dirname(django.__file__)):             warnings.warn(                 USE_L10N_DEPRECATED_MSG,diff --git a/django/utils/functional.py b/django/utils/functional.py-- a/django/utils/functional.py++ b/django/utils/functional.py

LazyObject defines attribute that don't exist on wrapped object Description LazyObject defines magic methods (getitem, iter) which may be missing from the wrapped object. This leads to the following errors: somevariable = if hasattr(somevariable, "getitem"): foo = somevariable["foo"] # raises TypeError: 'User' object has no attribute 'getitem' if hasattr(somevariable, "iter"): for item in some_variable: # raises TypeError: 'User' object is not iterable


Question: self._setup()         return func(self._wrapped, *args) <ins>    inner._mask_wrapped = False</ins>     return inner


Question: # override __copy__() and __deepcopy__() as well.         self._wrapped = empty <ins>    def __getattribute__(self, name):        value = super().__getattribute__(name)        # If attribute is a proxy method, raise an AttributeError to call        # __getattr__() and use the wrapped object method.        if not getattr(value, "_mask_wrapped", True):            raise AttributeError        return value</ins>     __getattr__ = new_method_proxy(getattr)      def __setattr__(self, name, value):


Question: kwargs["db_table"] = self.db_table         if self.remote_field.db_constraint is not True:             kwargs["db_constraint"] = self.remote_field.db_constraint        # Rel needs more work.        # Lowercase model names as they should be treated as case-insensitive.         if isinstance(self.remote_field.model, str):            kwargs["to"] = self.remote_field.model            if "." in self.remote_field.model:                app_label, model_name = self.remote_field.model.split(".")                kwargs["to"] = "%s.%s" % (app_label, model_name.lower())            else:                kwargs["to"] = self.remote_field.model.lower()         else:            kwargs["to"] = self.remote_field.model._meta.label            kwargs["to"] = self.remote_field.model._meta.label_lower         if getattr(self.remote_field, "through", None) is not None:             if isinstance(self.remote_field.through, str):                 kwargs["through"] = self.remote_field.through

ManyToManyField to lowercased swappable setting causes generating infinite migrations. Description If I create a custom user model that extends AbstractUser and then try to add a ManyToManyField that references this custom User model, django keeps making the same AlterField migration over and over again unnecessarily. I've attached a Git repository that I used to reproduce this issue with very simple code. You can see the two erroneous migrations after the initial migration. If I use the built in user, there is no issue. It seems to appear once I extend AbstractUser. Git repository:  Here is the accounts app code. from import settings from import AbstractUser from import models class User(AbstractUser): pass class Test(models.Model): members = models.ManyToManyField(settings.AUTHUSERMODEL) Here is one of the erroneous migrations. Notice it depends on 0002altertestmembers which is an erroneous migrations file that looks just like this one. from import settings from import migrations, models class Migration(migrations.Migration): dependencies = [ ('accounts', '0002altertestmembers'), ] operations = [ migrations.AlterField( modelname='test', name='members', field=models.ManyToManyField(to=settings.AUTHUSER_MODEL), ), ]


Question: SELECT2_TRANSLATIONS.update({"zh-hans": "zh-CN", "zh-hant": "zh-TW"})  <ins>def get_select2_language():    lang_code = get_language()    supported_code = SELECT2_TRANSLATIONS.get(lang_code)    if supported_code is None:        # If 'zh-hant-tw' is not supported, try subsequent language codes i.e.        # 'zh-hant' and 'zh'.        i = None        while (i := lang_code.rfind("-", 0, i)) > -1:            if supported_code := SELECT2_TRANSLATIONS.get(lang_code[:i]):                return supported_code    return supported_code</ins> class AutocompleteMixin:     """     Select widget mixin that loads options from AutocompleteJsonView via AJAX.

Fallback to a more generic language variant for Select2 translations. Description When using for example the setting LANGUAGECODE="de-ch", the autocompletefield widget will be in English as select2 does not provide the specific regional translation for "de-ch". However the existing translation language "de" would be a much better approximation. I suggest using the language without region from the LANGUAGE_CODE in case an exact match can not be found. This is not a duplicate of .


Question: self.db = using         self.choices = choices         self.attrs = {} if attrs is None else attrs.copy()        self.i18n_name = SELECT2_TRANSLATIONS.get(get_language())        self.i18n_name = get_select2_language()      def get_url(self):         return reverse(self.url_name % self.admin_site.name)


Question: from functools import wraps <ins>from django.utils.functional import keep_lazy</ins>  class SafeData:     __slots__ = ()

forces evaluation of lazy objects Description Consider the following example: from import marksafe from import activate, ugettextlazy as s = marksafe(("username")) tpl = Template('{{ s }}') activate('fr') print((Context({'s': s}))) I would expect this to output nom d'utilisateur (which is the french translation of username) but what happens instead is that it outputs username. The reason for this is that marksafe will force the evaluation of the lazy string provided by ugettextlazy when it's called. Unfortunately, the solution to this it trickier than simply wrapping marksafe with , because marksafe can operate both on bytes and text (and allowlazy needs to know the type of object return by the wrapped function). I wrote some tests and a proposed solution on my branch:


Question: return wrapped  <ins>@keep_lazy(SafeString)</ins> def mark_safe(s):     """     Explicitly mark a string as safe for (HTML) output purposes. The returned


Question: "class": get_ul_class(self.radio_fields[db_field.name]),                     }                 )                kwargs["empty_label"] = _("None") if db_field.blank else None                kwargs["empty_label"] = (                    kwargs.get("empty_label", _("None")) if db_field.blank else None                )          if "queryset" not in kwargs:             queryset = self.get_field_queryset(db, db_field, request)

ModelAdmin with defined radiofields override emptylabel Description ModelAdmin drops my "emptylabel" and set "defaultemptylabel". For example: class MyModelAdmin(ModelAdmin): radiofields = 'myfield', def formfieldforforeignkey(self, dbfield, args, kwargs): if == 'myfield': kwargs['emptylabel'] = "I WANT TO SET MY OWN EMPTY LABEL" return super().formfieldforforeignkey(dbfield, args, kwargs) You get never the "I WANT TO SET MY OWN EMPTY LABEL" How to fix it: In django\contrib\admin\, row 234: kwargs['emptylabel'] = ('None') if else None Should be changed on: kwargs['emptylabel'] = (('emptylabel') or ('None')) if else None


Question: def add_legacy_name(apps, schema_editor):    alias = schema_editor.connection.alias     ContentType = apps.get_model("contenttypes", "ContentType")    for ct in ContentType.objects.all():    for ct in ContentType.objects.using(alias):         try:             ct.name = apps.get_model(ct.app_label, ct.model)._meta.object_name         except LookupError:

Unapplying ContentType migration 0002 does not populate legacy name field on non-default database Description Given a database with the alias other that has all contenttypes migrations applied, roll the state back to initial: $ migrate --database=other contenttypes 0001 All ContentType rows in the other database will be null instead of their intended value.


Question: - a naive ``datetime.datetime`` object in UTC,         - an aware ``datetime.datetime`` object in any time zone.         If it is a ``datetime.datetime`` object then calculate ``max_age``.<ins>        ``max_age`` can be:        - int/float specifying seconds,        - ``datetime.timedelta`` object.</ins>         """         self.cookies[key] = value         if expires is not None:

setcookie and setsignedcookie should accept timedelta object for maxage argument Description This already works for getsignedcookie: request.getsignedcookie("mykey", maxage=timedelta(days=3)) This is due to the underlying behaviour of TimestampSigner, which was fixed to do this in . But for setcookie and setsignedcookie it accepts only a number: response = HttpResponse() response.setcookie("mykey", maxage=timedelta(days=3)) TypeError: int() argument must be a string, a bytes-like object or a number, not 'datetime.timedelta'


Question: delta = delta + datetime.timedelta(seconds=1)                 # Just set max_age - the max_age logic will set expires.                 expires = None                if max_age is not None:                    raise ValueError("'expires' and 'max_age' can't be used together.")                 max_age = max(0, delta.days * 86400 + delta.seconds)             else:                 self.cookies[key]["expires"] = expires         else:             self.cookies[key]["expires"] = ""         if max_age is not None:            if isinstance(max_age, datetime.timedelta):                max_age = max_age.total_seconds()             self.cookies[key]["max-age"] = int(max_age)             # IE requires expires, so set it if hasn't been already.             if not expires:


Question: "models": [model_dict],                 }         if label:            return app_dict.get(label)         return app_dict     def get_app_list(self, request):    def get_app_list(self, request, app_label=None):         """         Return a sorted list of all the installed apps that have been         registered in this site.         """        app_dict = self._build_app_dict(request)        app_dict = self._build_app_dict(request, app_label)          # Sort the apps alphabetically.         app_list = sorted(app_dict.values(), key=lambda x: x["name"].lower())

AppAdmin class for customizing app listing in admin index Description See  for details. This supplements the app directive. As discussed with brosner and jkocherhans in #django-dev: <brosnerit looks reasonable, but haven't spent much time thinking about it <jkocherhansmrts: I think this is clearly backwards incompatible with the current nfa api and has to go in pre 1.0 if it goes in at all <jkocherhansI'm a big -1 on the order attribute and -0 on models (maybe just a different syntax), but the other stuff seems reasonable <mrtsjkocherhans: what's wrong with ordering? <jkocherhansit just feels like the wrong place to specify it <jkocherhansit's a global issue, and an issue any particular app should handle <mrtsmy use case: I have a lot of functionality exposed to somewhat dumb users <mrtsand they have trouble finding the right bits in the admin interface ordering is only used in context of admin index I would like to put the important apps to top and collapse the rest <jkocherhansexactly. what should 3rd party apps put there? therein lies my objection. <mrtswell, I'd say decouple admin from models (as nfa already does) and don't specify any admin options at all -- users are free to customize things with AppAdmin <jkocherhansI guess not if using a AppAdmin class is optional. I was originally thinking it would replace model registration with an admin site. <mrtsjkocherhans: yeah, that's what I kinda meant... it looks more coherent this way jkocherhans: and it may solve some of the issues register() currently has <jkocherhansmrts: I'm gonna have to let it sit for awhile. I'm trying to think of what else an AdminApp class would do besides being a coathanger for a few attributes, nothing is coming to mind. <mrtsjkocherhans: but jezdez has a point -- it would also provide easy bridging for app instances Example syntax follows.


Question: "models": [model_dict],                 }         if label:            return app_dict.get(label)         return app_dict     def get_app_list(self, request):    def get_app_list(self, request, app_label=None):         """         Return a sorted list of all the installed apps that have been         registered in this site.         """        app_dict = self._build_app_dict(request)        app_dict = self._build_app_dict(request, app_label)          # Sort the apps alphabetically.         app_list = sorted(app_dict.values(), key=lambda x: x["name"].lower())

class BarModelAdmin(admin.ModelAdmin): description = 'A bar is a bar is a bar' ... class FooAppAdmin(admin.AppAdmin): app = settings.INSTALLED_APPS[0] name = "Foo" # overrides app().name description = "An application that does foo" style = {'classes' : ('collapse',)} order = 1 models = ( # model order in this list determines their display order in app block (BarModel, BarModelAdmin), (BazModel, None), # use default ModelAdmin, don't show description ) (FooAppAdmin) # no need for the tedious for model in [A, B, C, D]: (model)


Question: )      def app_index(self, request, app_label, extra_context=None):        app_dict = self._build_app_dict(request, app_label)        if not app_dict:        app_list = self.get_app_list(request, app_label)        if not app_list:             raise Http404("The requested admin page does not exist.")        # Sort the models alphabetically within each app.        app_dict["models"].sort(key=lambda x: x["name"])         context = {             **self.each_context(request),            "title": _("%(app)s administration") % {"app": app_dict["name"]},            "title": _("%(app)s administration") % {"app": app_list[0]["name"]},             "subtitle": None,            "app_list": [app_dict],            "app_list": app_list,             "app_label": app_label,             **(extra_context or {}),         }


Question: )      def app_index(self, request, app_label, extra_context=None):        app_dict = self._build_app_dict(request, app_label)        if not app_dict:        app_list = self.get_app_list(request, app_label)        if not app_list:             raise Http404("The requested admin page does not exist.")        # Sort the models alphabetically within each app.        app_dict["models"].sort(key=lambda x: x["name"])         context = {             **self.each_context(request),            "title": _("%(app)s administration") % {"app": app_dict["name"]},            "title": _("%(app)s administration") % {"app": app_list[0]["name"]},             "subtitle": None,            "app_list": [app_dict],            "app_list": app_list,             "app_label": app_label,             **(extra_context or {}),         }


Question: else:             # Raise if a developer-specified template doesn't exist.             raise    return HttpResponseForbidden(t.render(c), content_type="text/html")    return HttpResponseForbidden(t.render(c))diff --git a/django/views/debug.py b/django/views/debug.py-- a/django/views/debug.py++ b/django/views/debug.py

Builtin csrffailure() view uses wrong charset Description When Django detects wrong CSRF token, it shows an error using view That file ends with return HttpResponseForbidden(t.render(c), contenttype="text/html;") When the template (CSRFFAILURETEMPLATENAME) is written using UTF-8, it is rendered incorrectly. I suggest changing that line to return HttpResponseForbidden(t.render(c), contenttype="text/html;"+ f" charset={settings.DEFAULTCHARSET};") or perhaps leaving out the content_type entirely. Currently I'm using a workaround, by adding <meta http-equiv="Content-type" content="text/html; charset=utf-8" /to the template's HEAD, but it seems to me that the suggested fix is a better solution.


Question: reporter = get_exception_reporter_class(request)(request, exc_type, exc_value, tb)     if request.accepts("text/html"):         html = reporter.get_traceback_html()        return HttpResponse(html, status=status_code, content_type="text/html")        return HttpResponse(html, status=status_code)     else:         text = reporter.get_traceback_text()         return HttpResponse(


Question: "raising_view_name": get_caller(request),         }     )    return HttpResponseNotFound(t.render(c), content_type="text/html")    return HttpResponseNotFound(t.render(c))   def default_urlconf(request):


Question: }     )     return HttpResponse(t.render(c), content_type="text/html")    return HttpResponse(t.render(c))diff --git a/django/views/defaults.py b/django/views/defaults.py-- a/django/views/defaults.py++ b/django/views/defaults.py


Question: try:         template = loader.get_template(template_name)         body = template.render(context, request)<del>        content_type = None  # Django will use 'text/html'.</del>     except TemplateDoesNotExist:         if template_name != ERROR_404_TEMPLATE_NAME:             # Reraise if it's a missing custom template.


Question: },         )         body = template.render(Context(context))        content_type = "text/html"    return HttpResponseNotFound(body, content_type=content_type)    return HttpResponseNotFound(body)   @requires_csrf_token


Question: raise         return HttpResponseServerError(             ERROR_PAGE_TEMPLATE % {"title": "Server Error (500)", "details": ""},<del>            content_type="text/html",</del>         )     return HttpResponseServerError(template.render())


Question: raise         return HttpResponseBadRequest(             ERROR_PAGE_TEMPLATE % {"title": "Bad Request (400)", "details": ""},<del>            content_type="text/html",</del>         )     # No exception content is passed to the template, to not disclose any     # sensitive information.


Question: raise         return HttpResponseForbidden(             ERROR_PAGE_TEMPLATE % {"title": "403 Forbidden", "details": ""},<del>            content_type="text/html",</del>         )     return HttpResponseForbidden(         template.render(request=request, context={"exception": str(exception)})


Question: import warnings from django.contrib.auth import get_user_model from django.contrib.auth.models import Permission from django.db.models import Exists, OuterRef, Qfrom django.utils.deprecation import RemovedInDjango50Warningfrom django.utils.inspect import func_supports_parameter  UserModel = get_user_model()

Allow syncing user attributes on every authentication with RemoteUserBackend. Description If using a custom RemoteUserBackend, it is possible to want to synchronize any changes from the remote system back into the Django user records whenever authentication happens. Currently, if any user attributes change in the remote system there is no easy way to reflect these changes back into the users in the Django system. The goal of this feature is to introduce a new method in the class called synchronizeuser with a method signature equal to that of configureuser (which it complements) that will be called in the authenticate method of said class right after fetching the user from the database (if any), regardless of whether the user was unknown and created or otherwise. Implementors can then override this method and implement data synchronization between the remote user and the Django user that will be applied on every user authentication attempt.


Question: """         if not remote_user:             return<ins>        created = False</ins>         user = None         username = self.clean_username(remote_user)


Question: user, created = UserModel._default_manager.get_or_create(                 **{UserModel.USERNAME_FIELD: username}             )            if created:                user = self.configure_user(request, user)         else:             try:                 user = UserModel._default_manager.get_by_natural_key(username)             except UserModel.DoesNotExist:                 pass        # RemovedInDjango50Warning: When the deprecation ends, replace with:        #   user = self.configure_user(request, user, created=created)        if func_supports_parameter(self.configure_user, "created"):            user = self.configure_user(request, user, created=created)        else:            warnings.warn(                f"`created=True` must be added to the signature of "                f"{self.__class__.__qualname__}.configure_user().",                category=RemovedInDjango50Warning,            )            if created:                user = self.configure_user(request, user)         return user if self.user_can_authenticate(user) else None      def clean_username(self, username):


Question: """         return username     def configure_user(self, request, user):    def configure_user(self, request, user, created=True):         """        Configure a user after creation and return the updated user.        Configure a user and return the updated user.          By default, return the user unmodified.         """


Question: )             header_value = self.META.get(header)             if header_value is not None:                return "https" if header_value == secure_value else "http"                header_value, *_ = header_value.split(",", 1)                return "https" if header_value.strip() == secure_value else "http"         return self._get_scheme()      def is_secure(self):

Add support for multiple values for the x-forwarded-proto header Description When Django is deployed behind more than one proxy, the proxy behavior is sometimes to list the protocol as a comma-separated list. However, currently, Django expects only one value for the x-forwarded-proto header, instead of parsing it as a list of values and setting the protocol accordingly. x-forwarded-proto is a non-standard header, so there isn't a specification for its use, but different reverse-proxy vendors do use it in different ways, and some append the protocol as a comma-separated value from left-to-right (left being the furthermost proxy and rightmost being the closest). Similar issues have been raised and implemented in other projects, for example: Tornado: Issue:  Implementation:  Ruby: Issue:  Implemenation:  Reactor-Netty:  Implementation:  Most implementation use the leftmost-value or rightmost value. I would expect that provided that you are certain that the initial proxy can be trusted, that the left-most value makes the most sense, since it represent the original value at the entry-point for the HTTP request which is often where TLS is being terminated. Common example of this behavior is when using mulitple AWS proxies such as API Gateway proxying to an elastic load balancer.


Question: if header is None:             raise ValueError         matches = re.match(r"^([^;]+)(; length=([0-9]+))?$", header, re.IGNORECASE)        if matches is None:            raise ValueError         header_mtime = parse_http_date(matches[1])         header_len = matches[3]         if header_len and int(header_len) != size:             raise ValueError         if int(mtime) > header_mtime:             raise ValueError    except (AttributeError, ValueError, OverflowError):    except (ValueError, OverflowError):         return True     return False

Fix handling empty string for If-Modified-Since header Description Empty string used to be ignored for If-Modified-Since header, but now raises exception since . Fix handling empty string for If-Modified-Since header Description Empty string used to be ignored for If-Modified-Since header, but now raises exception since .


Question: managers=self.managers,                 ),             ]<ins>        elif (            isinstance(operation, AlterModelManagers)            and self.name_lower == operation.name_lower        ):            return [                CreateModel(                    self.name,                    fields=self.fields,                    options=self.options,                    bases=self.bases,                    managers=operation.managers,                ),            ]</ins>         elif (             isinstance(operation, AlterTogetherOptionOperation)             and self.name_lower == operation.name_lower

Optimize CreateModel + AlterModelManagers to CreateModel Description During migration optimization, CreateModel + AlterModelOptions is reduced to just CreateModel, with the model options. Similarly, CreateModel + AlterModelManagers can become just CreateModel.


Question: class HasKeyLookup(PostgresOperatorLookup):     logical_operator = None <ins>    def compile_json_path_final_key(self, key_transform):        # Compile the final key without interpreting ints as array elements.        return ".%s" % json.dumps(key_transform)</ins>     def as_sql(self, compiler, connection, template=None):         # Process JSON path from the left-hand side.         if isinstance(self.lhs, KeyTransform):

haskey, haskeys, and hasanykeys JSONField() lookups don't handle numeric keys on SQLite, MySQL, and Oracle. Description (last modified by TheTerrasque) Problem When using models.JSONField() haskey lookup with numerical keys on SQLite database it fails to find the keys. Versions: Django: 4.0.3 Python: 3.9.6 (tags/v3.9.6:, Jun 28 2021, 15:26:21) [MSC v.1929 64 bit (AMD64)] on win32 sqlite3.version: '2.6.0' sqlite3.sqliteversion: '3.35.5' Example: Database DATABASES = { 'default': { 'ENGINE': '', 'NAME': 'db.sqlite3', } } Model class JsonFieldHasKeyTest(models.Model): data = models.JSONField() Test from import TestCase from .models import JsonFieldHasKeyTest class JsonFieldHasKeyTestCase(TestCase): def setUp(self) -None: test = JsonFieldHasKeyTest(data={'foo': 'bar'}) test2 = JsonFieldHasKeyTest(data={'1111': 'bar'}) () () def testjsonfieldhaskey(self): c1 = (datahaskey='foo').count() c2 = (datahaskey='1111').count() self.assertEqual(c1, 1, "Should have found 1 entry with key 'foo'") self.assertEqual(c2, 1, "Should have found 1 entry with key '1111'") Result FAIL: testjsonfieldhaskey () Traceback (most recent call last): File "H:\Files\Projects\Electaco\Webservice\elecserve\markers\", line 16, in testjsonfieldhaskey self.assertEqual(c2, 1, "Should have found 1 entry with key '1111'") AssertionError: 0 != 1 : Should have found 1 entry with key '1111' Additional info This has been tested on SQLite and Postgresql backend, it works on postgresql but fails on sqlite.


Question: *_, rhs_key_transforms = key.preprocess_lhs(compiler, connection)             else:                 rhs_key_transforms = [key]            rhs_params.append(                "%s%s"                % (                    lhs_json_path,                    compile_json_path(rhs_key_transforms, include_root=False),                )            )            *rhs_key_transforms, final_key = rhs_key_transforms            rhs_json_path = compile_json_path(rhs_key_transforms, include_root=False)            rhs_json_path += self.compile_json_path_final_key(final_key)            rhs_params.append(lhs_json_path + rhs_json_path)         # Add condition for each key.         if self.logical_operator:             sql = "(%s)" % self.logical_operator.join([sql] * len(rhs_params))


Question: logical_operator = " OR "  <ins>class HasKeyOrArrayIndex(HasKey):    def compile_json_path_final_key(self, key_transform):        return compile_json_path([key_transform], include_root=False)</ins> class CaseInsensitiveMixin:     """     Mixin to allow case-insensitive comparison of JSON values on MySQL.


Question: class KeyTransformIsNull(lookups.IsNull):     # key__isnull=False is the same as has_key='key'     def as_oracle(self, compiler, connection):        sql, params = HasKey(        sql, params = HasKeyOrArrayIndex(             self.lhs.lhs,             self.lhs.key_name,         ).as_oracle(compiler, connection)


Question: template = "JSON_TYPE(%s, %%s) IS NULL"         if not self.rhs:             template = "JSON_TYPE(%s, %%s) IS NOT NULL"        return HasKey(self.lhs.lhs, self.lhs.key_name).as_sql(        return HasKeyOrArrayIndex(self.lhs.lhs, self.lhs.key_name).as_sql(             compiler,             connection,             template=template,


Question: rhs, rhs_params = super().process_rhs(compiler, connection)         if rhs_params == ["null"]:             # Field has key and it's NULL.            has_key_expr = HasKey(self.lhs.lhs, self.lhs.key_name)            has_key_expr = HasKeyOrArrayIndex(self.lhs.lhs, self.lhs.key_name)             has_key_sql, has_key_params = has_key_expr.as_oracle(compiler, connection)             is_null_expr = self.lhs.get_lookup("isnull")(self.lhs, True)             is_null_sql, is_null_params = is_null_expr.as_sql(compiler, connection)


Question: )  <ins>def is_valid_locale(locale):    return re.match(r"^[a-z]+$", locale) or re.match(r"^[a-z]+_[A-Z].*$", locale)</ins> @total_ordering class TranslatableFile:     def __init__(self, dirpath, file_name, locale_dir):

makemessages command skips the nl-nl-x-informal folder Description There is a mismatch on the DjangoTranslation folder and the makemessages command for the value nl-nl-x-informal DjangoTranslation uses the tolocale method to determine the language folder to read the tolocale translates nl-nl-x-informal correctly to the nlNL-x-informal folder. However makemessages skips the nlNL-x-informal folder and displays the following message invalid locale nlNL-x-informal, did you mean nlNLxinformal?  The check for - in the locale should only be for the first section a.k.a. nl_NL


Question: # Build po files for each selected locale             for locale in locales:                if "-" in locale:                    self.stdout.write(                        "invalid locale %s, did you mean %s?"                        % (                            locale,                            locale.replace("-", "_"),                        ),                if not is_valid_locale(locale):                    # Try to guess what valid locale it could be                    # Valid examples are: en_GB, shi_Latn_MA and nl_NL-x-informal                    # Search for characters followed by a non character (i.e. separator)                    match = re.match(                        r"^(?P<language>[a-zA-Z]+)"                        r"(?P<separator>[^a-zA-Z])"                        r"(?P<territory>.+)$",                        locale,                     )                    if match:                        locale_parts = match.groupdict()                        language = locale_parts["language"].lower()                        territory = (                            locale_parts["territory"][:2].upper()                            + locale_parts["territory"][2:]                        )                        proposed_locale = f"{language}_{territory}"                    else:                        # It could be a language in uppercase                        proposed_locale = locale.lower()                    # Recheck if the proposed locale is valid                    if is_valid_locale(proposed_locale):                        self.stdout.write(                            "invalid locale %s, did you mean %s?"                            % (                                locale,                                proposed_locale,                            ),                        )                    else:                        self.stdout.write("invalid locale %s" % locale)                     continue                 if self.verbosity > 0:                     self.stdout.write("processing locale %s" % locale)