Question: Test runner setupdatabases crashes with "TEST": {"MIGRATE": False}. Description I'm trying to upgrade a project from Django 3.0 to Django 3.1 and wanted to try out the new "TEST": {"MIGRATE": False} database setting. Sadly I'm running into an issue immediately when running test. Removing the "TEST": {"MIGRATE": False} line allows the tests to run. So this is not blocking the upgrade for us, but it would be nice if we were able to use the new feature to skip migrations during testing. For reference, this project was recently upgraded from Django 1.4 all the way to 3.0 so there might be some legacy cruft somewhere that triggers this. Here's the trackeback. I'll try to debug this some more. Traceback (most recent call last): File "/usr/local/lib/python3.6/site-", line 84, in execute return (sql, params) relation "djangoadminlog" does not exist LINE 1: ", "djangoadminlog"."changemessage" FROM "djangoad... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.6/site-", line 1156, in executesql cursor.execute(sql, params) File "/usr/local/lib/python3.6/site-", line 66, in execute return self.executewithwrappers(sql, params, many=False, executor=self.execute) File "/usr/local/lib/python3.6/site-", line 75, in executewithwrappers return executor(sql, params, many, context) File "/usr/local/lib/python3.6/site-", line 84, in execute return (sql, params) File "/usr/local/lib/python3.6/site-", line 90, in exit raise djexcvalue.withtraceback(traceback) from excvalue File "/usr/local/lib/python3.6/site-", line 84, in execute return (sql, params) relation "djangoadminlog" does not exist LINE 1: ", "djangoadminlog"."changemessage" FROM "djangoad... ^ During handling of the above exception, another exception occurred: Traceback (most recent call last): File "", line 15, in <modulemain() File "", line 11, in main executefromcommandline() File "/usr/local/lib/python3.6/site-", line 401, in executefromcommandline utility.execute() File "/usr/local/lib/python3.6/site-", line 395, in execute self.fetchcommand(subcommand).runfromargv() File "/usr/local/lib/python3.6/site-", line 23, in runfromargv super().runfromargv(argv) File "/usr/local/lib/python3.6/site-", line 330, in runfromargv self.execute(args, cmdoptions) File "/usr/local/lib/python3.6/site-", line 371, in execute output = (args, options) File "/usr/local/lib/python3.6/site-", line 53, in handle failures = testrunner.runtests(testlabels) File "/usr/local/lib/python3.6/site-", line 695, in runtests oldconfig = self.setupdatabases(aliases=databases) File "/usr/local/lib/python3.6/site-", line 616, in setupdatabases self.parallel, kwargs File "/usr/local/lib/python3.6/site-", line 174, in setupdatabases serialize=connection.settingsdict['TEST'].get('SERIALIZE', True), File "/usr/local/lib/python3.6/site-", line 78, in createtestdb self.connection.testserializedcontents = self.serializedbtostring() File "/usr/local/lib/python3.6/site-", line 121, in serializedbtostring serializers.serialize("json", getobjects(), indent=None, stream=out) File "/usr/local/lib/python3.6/site-", line 128, in serialize s.serialize(queryset, options) File "/usr/local/lib/python3.6/site-", line 90, in serialize for count, obj in enumerate(queryset, start=1): File "/usr/local/lib/python3.6/site-", line 118, in getobjects yield from queryset.iterator() File "/usr/local/lib/python3.6/site-", line 360, in iterator yield from self.iterableclass(self, chunkedfetch=usechunkedfetch, chunksize=chunksize) File "/usr/local/lib/python3.6/site-", line 53, in iter results = compiler.executesql(chunkedfetch=self.chunkedfetch, chunksize=self.chunksize) File "/usr/local/lib/python3.6/site-", line 1159, in executesql () cursor "djangocurs139860821038912sync_1" does not exist

settings.DATABASES[self.connection.alias]["NAME"] = test_database_name         self.connection.settings_dict["NAME"] = test_database_name         if self.connection.settings_dict['TEST']['MIGRATE']:        try:            if self.connection.settings_dict['TEST']['MIGRATE'] is False:                # Disable migrations for all apps.                old_migration_modules = settings.MIGRATION_MODULES                settings.MIGRATION_MODULES = {                    app.label: None                    for app in apps.get_app_configs()                }             # We report migrate messages at one level lower than that             # requested. This ensures we don't get flooded with messages during             # testing (unless you really ask to be flooded).


Question: Test runner setupdatabases crashes with "TEST": {"MIGRATE": False}. Description I'm trying to upgrade a project from Django 3.0 to Django 3.1 and wanted to try out the new "TEST": {"MIGRATE": False} database setting. Sadly I'm running into an issue immediately when running test. Removing the "TEST": {"MIGRATE": False} line allows the tests to run. So this is not blocking the upgrade for us, but it would be nice if we were able to use the new feature to skip migrations during testing. For reference, this project was recently upgraded from Django 1.4 all the way to 3.0 so there might be some legacy cruft somewhere that triggers this. Here's the trackeback. I'll try to debug this some more. Traceback (most recent call last): File "/usr/local/lib/python3.6/site-", line 84, in execute return (sql, params) relation "djangoadminlog" does not exist LINE 1: ", "djangoadminlog"."changemessage" FROM "djangoad... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.6/site-", line 1156, in executesql cursor.execute(sql, params) File "/usr/local/lib/python3.6/site-", line 66, in execute return self.executewithwrappers(sql, params, many=False, executor=self.execute) File "/usr/local/lib/python3.6/site-", line 75, in executewithwrappers return executor(sql, params, many, context) File "/usr/local/lib/python3.6/site-", line 84, in execute return (sql, params) File "/usr/local/lib/python3.6/site-", line 90, in exit raise djexcvalue.withtraceback(traceback) from excvalue File "/usr/local/lib/python3.6/site-", line 84, in execute return (sql, params) relation "djangoadminlog" does not exist LINE 1: ", "djangoadminlog"."changemessage" FROM "djangoad... ^ During handling of the above exception, another exception occurred: Traceback (most recent call last): File "", line 15, in <modulemain() File "", line 11, in main executefromcommandline() File "/usr/local/lib/python3.6/site-", line 401, in executefromcommandline utility.execute() File "/usr/local/lib/python3.6/site-", line 395, in execute self.fetchcommand(subcommand).runfromargv() File "/usr/local/lib/python3.6/site-", line 23, in runfromargv super().runfromargv(argv) File "/usr/local/lib/python3.6/site-", line 330, in runfromargv self.execute(args, cmdoptions) File "/usr/local/lib/python3.6/site-", line 371, in execute output = (args, options) File "/usr/local/lib/python3.6/site-", line 53, in handle failures = testrunner.runtests(testlabels) File "/usr/local/lib/python3.6/site-", line 695, in runtests oldconfig = self.setupdatabases(aliases=databases) File "/usr/local/lib/python3.6/site-", line 616, in setupdatabases self.parallel, kwargs File "/usr/local/lib/python3.6/site-", line 174, in setupdatabases serialize=connection.settingsdict['TEST'].get('SERIALIZE', True), File "/usr/local/lib/python3.6/site-", line 78, in createtestdb self.connection.testserializedcontents = self.serializedbtostring() File "/usr/local/lib/python3.6/site-", line 121, in serializedbtostring serializers.serialize("json", getobjects(), indent=None, stream=out) File "/usr/local/lib/python3.6/site-", line 128, in serialize s.serialize(queryset, options) File "/usr/local/lib/python3.6/site-", line 90, in serialize for count, obj in enumerate(queryset, start=1): File "/usr/local/lib/python3.6/site-", line 118, in getobjects yield from queryset.iterator() File "/usr/local/lib/python3.6/site-", line 360, in iterator yield from self.iterableclass(self, chunkedfetch=usechunkedfetch, chunksize=chunksize) File "/usr/local/lib/python3.6/site-", line 53, in iter results = compiler.executesql(chunkedfetch=self.chunkedfetch, chunksize=self.chunksize) File "/usr/local/lib/python3.6/site-", line 1159, in executesql () cursor "djangocurs139860821038912sync_1" does not exist

database=self.connection.alias,                 run_syncdb=True,             )<ins>        finally:            if self.connection.settings_dict['TEST']['MIGRATE'] is False:                settings.MIGRATION_MODULES = old_migration_modules</ins>          # We then serialize the current state of the database into a string         # and store it on the connection. This slightly horrific process is so people


Question: Lag() with DecimalField crashes on SQLite. Description On Django 3.0.7 with a SQLite database using the following model: from import models class LagTest(models.Model): modified = models.DateField() data = models.FloatField() amount = models.DecimalField(decimalplaces=4, maxdigits=7) and the following query from import F from import Lag from import Window from import LagTest w = Window(expression=Lag('amount',7), partitionby=[F('modified')], orderby=F('modified').asc()) q = ().annotate(w=w) generates the following error: In [12]: print(q) OperationalError Traceback (most recent call last) C:\ProgramData\Anaconda3\envs\djbase\lib\site-packages\django\db\backends\ in execute(self, sql, params, ignoredwrapperargs) 85 else: ---86 return (sql, params) 87 C:\ProgramData\Anaconda3\envs\djbase\lib\site-packages\django\db\backends\sqlite3\ in execute(self, query, params) 395 query = self.convertquery(query) --396 return Database.Cursor.execute(self, query, params) 397 OperationalError: near "OVER": syntax error The above exception was the direct cause of the following exception: OperationalError Traceback (most recent call last) <ipython-input-12-996617e96a38in <module----1 print(q) C:\ProgramData\Anaconda3\envs\djbase\lib\site-packages\django\db\models\ in repr(self) 250 251 def repr(self): --252 data = list(self[:REPROUTPUTSIZE + 1]) 253 if len(data) REPROUTPUTSIZE: 254 data[-1] = "...(remaining elements truncated)..." C:\ProgramData\Anaconda3\envs\djbase\lib\site-packages\django\db\models\ in iter(self) 274 - Responsible for turning the rows into model objects. 275 """ --276 self.fetchall() 277 return iter(self.resultcache) 278 C:\ProgramData\Anaconda3\envs\djbase\lib\site-packages\django\db\models\ in fetchall(self) 1259 def fetchall(self): 1260 if self.resultcache is None: -1261 self.resultcache = list(self.iterableclass(self)) 1262 if self.prefetchrelatedlookups and not self.prefetchdone: 1263 self.prefetchrelatedobjects() C:\ProgramData\Anaconda3\envs\djbase\lib\site-packages\django\db\models\ in iter(self) 55 # Execute the query. This will also fill , klassinfo, 56 # and annotations. ---57 results = compiler.executesql(chunkedfetch=self.chunkedfetch, chunksize=self.chunksize) 58 select, klassinfo, annotationcolmap = (, compiler.klassinfo, 59 compiler.annotationcolmap) C:\ProgramData\Anaconda3\envs\djbase\lib\site-packages\django\db\models\sql\ in executesql(self, resulttype, chunkedfetch, chunksize) 1150 cursor = () 1151 try: -1152 cursor.execute(sql, params) 1153 except Exception: 1154 # Might fail for server-side cursors (e.g. connection closed) C:\ProgramData\Anaconda3\envs\djbase\lib\site-packages\django\db\backends\ in execute(self, sql, params) 98 def execute(self, sql, params=None): 99 with self.debugsql(sql, params, uselastexecutedquery=True): --100 return super().execute(sql, params) 101 102 def executemany(self, sql, paramlist): C:\ProgramData\Anaconda3\envs\djbase\lib\site-packages\django\db\backends\ in execute(self, sql, params) 66 67 def execute(self, sql, params=None): ---68 return self.executewithwrappers(sql, params, many=False, executor=self.execute) 69 70 def executemany(self, sql, paramlist): C:\ProgramData\Anaconda3\envs\djbase\lib\site-packages\django\db\backends\ in executewithwrappers(self, sql, params, many, executor) 75 for wrapper in reversed(): 76 executor = functools.partial(wrapper, executor) ---77 return executor(sql, params, many, context) 78 79 def execute(self, sql, params, ignoredwrapperargs): C:\ProgramData\Anaconda3\envs\djbase\lib\site-packages\django\db\backends\ in execute(self, sql, params, ignoredwrapperargs) 84 return (sql) 85 else: ---86 return (sql, params) 87 88 def executemany(self, sql, paramlist, ignoredwrapperargs): C:\ProgramData\Anaconda3\envs\djbase\lib\site-packages\django\db\ in exit(self, exctype, excvalue, traceback) 88 if djexctype not in (DataError, IntegrityError): 89 self.wrapper.errorsoccurred = True ---90 raise djexcvalue.withtraceback(traceback) from excvalue 91 92 def call(self, func): C:\ProgramData\Anaconda3\envs\djbase\lib\site-packages\django\db\backends\ in execute(self, sql, params, ignoredwrapperargs) 84 return (sql) 85 else: ---86 return (sql, params) 87 88 def executemany(self, sql, paramlist, ignoredwrapperargs): C:\ProgramData\Anaconda3\envs\djbase\lib\site-packages\django\db\backends\sqlite3\ in execute(self, query, params) 394 return Database.Cursor.execute(self, query) 395 query = self.convertquery(query) --396 return Database.Cursor.execute(self, query, params) 397 398 def executemany(self, query, paramlist): OperationalError: near "OVER": syntax error The generated SQL query is: SELECT "test1lagtest"."id", "test1lagtest"."modified", "test1lagtest"."data", "test1lagtest"."amount", CAST(LAG("test1lagtest"."amount", 7) AS NUMERIC) OVER (PARTITION BY "test1lagtest"."modified" ORDER BY "test1lagtest"."modified" ASC) AS "w" FROM "test1lagtest" I believe this fails as the CAST() statement ends after LAG whereas it should be around the whole statement up until "w" This only applies where the lagged field is a DecimalField e.g. w = Window(expression=Lag('data',7), partitionby=[F('modified')], orderby=F('modified').asc()) works correctly. I can override it by adding outputfield=FloatField() to the Lag function e.g. w = Window(expression=Lag('amount',7,outputfield=FloatField()), partitionby=[F('modified')], orderby=F('modified').asc())

self.descending = True  class Window(Expression):class Window(SQLiteNumericMixin, Expression):     template = '%(expression)s OVER (%(window)s)'     # Although the main expression may either be an aggregate or an     # expression with an aggregate function, the GROUP BY that will


Question: Lag() with DecimalField crashes on SQLite. Description On Django 3.0.7 with a SQLite database using the following model: from import models class LagTest(models.Model): modified = models.DateField() data = models.FloatField() amount = models.DecimalField(decimalplaces=4, maxdigits=7) and the following query from import F from import Lag from import Window from import LagTest w = Window(expression=Lag('amount',7), partitionby=[F('modified')], orderby=F('modified').asc()) q = ().annotate(w=w) generates the following error: In [12]: print(q) OperationalError Traceback (most recent call last) C:\ProgramData\Anaconda3\envs\djbase\lib\site-packages\django\db\backends\ in execute(self, sql, params, ignoredwrapperargs) 85 else: ---86 return (sql, params) 87 C:\ProgramData\Anaconda3\envs\djbase\lib\site-packages\django\db\backends\sqlite3\ in execute(self, query, params) 395 query = self.convertquery(query) --396 return Database.Cursor.execute(self, query, params) 397 OperationalError: near "OVER": syntax error The above exception was the direct cause of the following exception: OperationalError Traceback (most recent call last) <ipython-input-12-996617e96a38in <module----1 print(q) C:\ProgramData\Anaconda3\envs\djbase\lib\site-packages\django\db\models\ in repr(self) 250 251 def repr(self): --252 data = list(self[:REPROUTPUTSIZE + 1]) 253 if len(data) REPROUTPUTSIZE: 254 data[-1] = "...(remaining elements truncated)..." C:\ProgramData\Anaconda3\envs\djbase\lib\site-packages\django\db\models\ in iter(self) 274 - Responsible for turning the rows into model objects. 275 """ --276 self.fetchall() 277 return iter(self.resultcache) 278 C:\ProgramData\Anaconda3\envs\djbase\lib\site-packages\django\db\models\ in fetchall(self) 1259 def fetchall(self): 1260 if self.resultcache is None: -1261 self.resultcache = list(self.iterableclass(self)) 1262 if self.prefetchrelatedlookups and not self.prefetchdone: 1263 self.prefetchrelatedobjects() C:\ProgramData\Anaconda3\envs\djbase\lib\site-packages\django\db\models\ in iter(self) 55 # Execute the query. This will also fill , klassinfo, 56 # and annotations. ---57 results = compiler.executesql(chunkedfetch=self.chunkedfetch, chunksize=self.chunksize) 58 select, klassinfo, annotationcolmap = (, compiler.klassinfo, 59 compiler.annotationcolmap) C:\ProgramData\Anaconda3\envs\djbase\lib\site-packages\django\db\models\sql\ in executesql(self, resulttype, chunkedfetch, chunksize) 1150 cursor = () 1151 try: -1152 cursor.execute(sql, params) 1153 except Exception: 1154 # Might fail for server-side cursors (e.g. connection closed) C:\ProgramData\Anaconda3\envs\djbase\lib\site-packages\django\db\backends\ in execute(self, sql, params) 98 def execute(self, sql, params=None): 99 with self.debugsql(sql, params, uselastexecutedquery=True): --100 return super().execute(sql, params) 101 102 def executemany(self, sql, paramlist): C:\ProgramData\Anaconda3\envs\djbase\lib\site-packages\django\db\backends\ in execute(self, sql, params) 66 67 def execute(self, sql, params=None): ---68 return self.executewithwrappers(sql, params, many=False, executor=self.execute) 69 70 def executemany(self, sql, paramlist): C:\ProgramData\Anaconda3\envs\djbase\lib\site-packages\django\db\backends\ in executewithwrappers(self, sql, params, many, executor) 75 for wrapper in reversed(): 76 executor = functools.partial(wrapper, executor) ---77 return executor(sql, params, many, context) 78 79 def execute(self, sql, params, ignoredwrapperargs): C:\ProgramData\Anaconda3\envs\djbase\lib\site-packages\django\db\backends\ in execute(self, sql, params, ignoredwrapperargs) 84 return (sql) 85 else: ---86 return (sql, params) 87 88 def executemany(self, sql, paramlist, ignoredwrapperargs): C:\ProgramData\Anaconda3\envs\djbase\lib\site-packages\django\db\ in exit(self, exctype, excvalue, traceback) 88 if djexctype not in (DataError, IntegrityError): 89 self.wrapper.errorsoccurred = True ---90 raise djexcvalue.withtraceback(traceback) from excvalue 91 92 def call(self, func): C:\ProgramData\Anaconda3\envs\djbase\lib\site-packages\django\db\backends\ in execute(self, sql, params, ignoredwrapperargs) 84 return (sql) 85 else: ---86 return (sql, params) 87 88 def executemany(self, sql, paramlist, ignoredwrapperargs): C:\ProgramData\Anaconda3\envs\djbase\lib\site-packages\django\db\backends\sqlite3\ in execute(self, query, params) 394 return Database.Cursor.execute(self, query) 395 query = self.convertquery(query) --396 return Database.Cursor.execute(self, query, params) 397 398 def executemany(self, query, paramlist): OperationalError: near "OVER": syntax error The generated SQL query is: SELECT "test1lagtest"."id", "test1lagtest"."modified", "test1lagtest"."data", "test1lagtest"."amount", CAST(LAG("test1lagtest"."amount", 7) AS NUMERIC) OVER (PARTITION BY "test1lagtest"."modified" ORDER BY "test1lagtest"."modified" ASC) AS "w" FROM "test1lagtest" I believe this fails as the CAST() statement ends after LAG whereas it should be around the whole statement up until "w" This only applies where the lagged field is a DecimalField e.g. w = Window(expression=Lag('data',7), partitionby=[F('modified')], orderby=F('modified').asc()) works correctly. I can override it by adding outputfield=FloatField() to the Lag function e.g. w = Window(expression=Lag('amount',7,outputfield=FloatField()), partitionby=[F('modified')], orderby=F('modified').asc())

'window': ''.join(window_sql).strip()         }, params <ins>    def as_sqlite(self, compiler, connection):        if isinstance(self.output_field, fields.DecimalField):            # Casting to numeric must be outside of the window expression.            copy = self.copy()            source_expressions = copy.get_source_expressions()            source_expressions[0].output_field = fields.FloatField()            copy.set_source_expressions(source_expressions)            return super(Window, copy).as_sqlite(compiler, connection)        return self.as_sql(compiler, connection)</ins>     def __str__(self):         return '{} OVER ({}{}{})'.format(             str(self.source_expression),


Question: Using EmptyFieldListFilter with GenericForeignKey and GenericRelation crashes. Description I have a GenericForeignKey in my model: contenttyperesourcecontract = models.ForeignKey( ContentType, ondelete=models.CASCADE, blank=True, null=True, ) resourcecontractid = models.PositiveIntegerField(blank=True, null=True) resourcecontract = GenericForeignKey('contenttyperesource', 'resourcecontractid') and I want to use the new admin.EmptyFieldListFilter in my model admin: class myAdmin(admin.ModelAdmin): listfilter = (('resourcecontract', admin.EmptyFieldListFilter),) But when I try to run it I get a "'GenericForeignKey' object has no attribute 'emptystringsallowed'". It will work fine if I use the resourcecontractid field. Would it make sense to extend the GenericForeignKey to use those emptystrings_allowed attributes from fields that are used in this generic relation?

# Field flags     auto_created = False<ins>    empty_strings_allowed = False</ins>      many_to_many = False     many_to_one = False


Question: Using EmptyFieldListFilter with GenericForeignKey and GenericRelation crashes. Description I have a GenericForeignKey in my model: contenttyperesourcecontract = models.ForeignKey( ContentType, ondelete=models.CASCADE, blank=True, null=True, ) resourcecontractid = models.PositiveIntegerField(blank=True, null=True) resourcecontract = GenericForeignKey('contenttyperesource', 'resourcecontractid') and I want to use the new admin.EmptyFieldListFilter in my model admin: class myAdmin(admin.ModelAdmin): listfilter = (('resourcecontract', admin.EmptyFieldListFilter),) But when I try to run it I get a "'GenericForeignKey' object has no attribute 'emptystringsallowed'". It will work fine if I use the resourcecontractid field. Would it make sense to extend the GenericForeignKey to use those emptystrings_allowed attributes from fields that are used in this generic relation?

limit_choices_to=limit_choices_to,         ) <ins>        # Reverse relations are always nullable (Django can't enforce that a        # foreign key on the related model points to this model).        kwargs['null'] = True</ins>         kwargs['blank'] = True         kwargs['on_delete'] = models.CASCADE         kwargs['editable'] = False


Question: Incorrect messaging when validatemin/validatemax and minnum == maxnum. Description When using Django Formset, the error messaging isn't checking for the condition where both minnum and maxnum are equal and both validatemin and validatemax are set to true. Code highlighting: class TestForm(forms.Form): msg = forms.CharField() testformset = formsetfactory( TestForm, minnum=2, maxnum=2, validatemin=True, validatemax=True) When formset is created in the following way and both validate flags are set True the following error messages show up If the supplied forms are less than two - please submit 2 or more forms expected please submit 2 forms similarly the reverse is also true when the forms are more than two it gives the error message saying please submit 2 or fewer forms expected please submit 2 forms This was a  bug reported on Wagtail and after investigating a little I noticed the incorrect messaging was coming from  this part in the validation

self.total_form_count() - len(self.deleted_forms) > self.max_num) or \                     self.management_form.cleaned_data[TOTAL_FORM_COUNT] > self.absolute_max:                 raise ValidationError(ngettext(                    "Please submit %d or fewer forms.",                    "Please submit %d or fewer forms.", self.max_num) % self.max_num,                    "Please submit at most %d form.",                    "Please submit at most %d forms.", self.max_num) % self.max_num,                     code='too_many_forms',                 )             if (self.validate_min and                     self.total_form_count() - len(self.deleted_forms) - empty_forms_count < self.min_num):                 raise ValidationError(ngettext(                    "Please submit %d or more forms.",                    "Please submit %d or more forms.", self.min_num) % self.min_num,                    "Please submit at least %d form.",                    "Please submit at least %d forms.", self.min_num) % self.min_num,                     code='too_few_forms')             # Give self.clean() a chance to do cross-form validation.             self.clean()


Question: String formatting error when passing floats as values in {% blocktrans %} tags Description With the following template code: {% blocktrans count counter=person.distanceinmiles|floatformat:0 %}{{ counter }} mile away{% plural %}{{ counter }} miles away{% endblocktrans %} And a russian translation: #, python-format msgid "%(counter)s mile away" msgidplural "%(counter)s miles away" msgstr[0] "На расстоянии %(counter)s мили" msgstr[1] "На расстоянии %(counter)s миль" msgstr[2] "На расстоянии %(counter)s миль" Rendering the template fails with a String formatting error: "TypeError: not all arguments converted during string formatting". This happens because gettext string formatting fails when a float is passed. Removing the floatformat and casting the value as int works fine. This could be improved by either: Swallow TypeError and throw a friendlier message saying the type might not be compatible with gettext's string formatting (not a fan of swallowing TypeError though, we need to make sure nothing else in dotranslate throws such an exception) Actively checking the type's compatibility with the string format

<ins>from decimal import Decimal</ins> from django.conf import settings from django.template import Library, Node, TemplateSyntaxError, Variable from django.template.base import TokenType, render_value_in_context


Question: String formatting error when passing floats as values in {% blocktrans %} tags Description With the following template code: {% blocktrans count counter=person.distanceinmiles|floatformat:0 %}{{ counter }} mile away{% plural %}{{ counter }} miles away{% endblocktrans %} And a russian translation: #, python-format msgid "%(counter)s mile away" msgidplural "%(counter)s miles away" msgstr[0] "На расстоянии %(counter)s мили" msgstr[1] "На расстоянии %(counter)s миль" msgstr[2] "На расстоянии %(counter)s миль" Rendering the template fails with a String formatting error: "TypeError: not all arguments converted during string formatting". This happens because gettext string formatting fails when a float is passed. Removing the floatformat and casting the value as int works fine. This could be improved by either: Swallow TypeError and throw a friendlier message saying the type might not be compatible with gettext's string formatting (not a fan of swallowing TypeError though, we need to make sure nothing else in dotranslate throws such an exception) Actively checking the type's compatibility with the string format

singular, vars = self.render_token_list(self.singular)         if self.plural and self.countervar and self.counter:             count = self.counter.resolve(context)<ins>            if not isinstance(count, (Decimal, float, int)):                raise TemplateSyntaxError(                    "%r argument to %r tag must be a number."                    % (self.countervar, self.tag_name)                )</ins>             context[self.countervar] = count             plural, plural_vars = self.render_token_list(self.plural)             if message_context:


Question: Required mutually exclusive groups don't work with boolean arguments. Description I have the following management command (called ): from import BaseCommand class Command(BaseCommand): def addarguments(self, parser): group = parser.addmutuallyexclusivegroup(required=True) group.addargument('--value', type=str) group.addargument('--flag', action='storetrue') def handle(self, args, *options): pass Running test --flag or --value foo, the command works as expected. Using callcommand to call the command from my code fails if I pass --flag, e.g.: from import callcommand callcommand('test', flag=True) This gets me the following traceback: argparse.ArgumentError: argument --flag: ignored explicit argument 'True' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "<input", line 1, in <moduleFile "", line 147, in callcommand defaults = parser.parseargs(args=parseargs) File "", line 57, in parseargs return super().parseargs(args, namespace) File "", line 1780, in parseargs args, argv = self.parseknownargs(args, namespace) File "", line 1819, in parseknownargs (str(err)) File "", line 63, in error raise CommandError("Error: %s" % message) Error: argument --flag: ignored explicit argument 'True'

import os import pkgutil import sysfrom argparse import _SubParsersActionfrom argparse import (    _AppendConstAction, _CountAction, _StoreConstAction, _SubParsersAction,) from collections import defaultdict from difflib import get_close_matches from importlib import import_module


Question: Required mutually exclusive groups don't work with boolean arguments. Description I have the following management command (called ): from import BaseCommand class Command(BaseCommand): def addarguments(self, parser): group = parser.addmutuallyexclusivegroup(required=True) group.addargument('--value', type=str) group.addargument('--flag', action='storetrue') def handle(self, args, *options): pass Running test --flag or --value foo, the command works as expected. Using callcommand to call the command from my code fails if I pass --flag, e.g.: from import callcommand callcommand('test', flag=True) This gets me the following traceback: argparse.ArgumentError: argument --flag: ignored explicit argument 'True' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "<input", line 1, in <moduleFile "", line 147, in callcommand defaults = parser.parseargs(args=parseargs) File "", line 57, in parseargs return super().parseargs(args, namespace) File "", line 1780, in parseargs args, argv = self.parseknownargs(args, namespace) File "", line 1819, in parseknownargs (str(err)) File "", line 63, in error raise CommandError("Error: %s" % message) Error: argument --flag: ignored explicit argument 'True'

# Any required arguments which are passed in via **options must be passed     # to parse_args().     parse_args += [        '{}={}'.format(min(opt.option_strings), arg_options[opt.dest])        min(opt.option_strings)        if isinstance(opt, (_AppendConstAction, _CountAction, _StoreConstAction))        else '{}={}'.format(min(opt.option_strings), arg_options[opt.dest])         for opt in parser_actions if (             opt.dest in options and             (opt.required or opt in mutually_exclusive_required_options)


Question: Queryset crashes when recreated from a pickled query with FilteredRelation used in aggregation. Description I am pickling query objects () for later re-evaluation as per  However, when I tried to rerun a query that contains a FilteredRelation inside a filter, I get an missing FROM-clause entry for table "t3" error. I created a minimum reproducible example. from import models class Publication(models.Model): title = models.CharField(maxlength=64) class Session(models.Model): TYPECHOICES = (('A', 'A'), ('B', 'B')) publication = models.ForeignKey(Publication, ondelete=models.CASCADE) sessiontype = models.CharField(choices=TYPECHOICES, default='A', maxlength=1) place = models.CharField(maxlength=16) value = models.PositiveIntegerField(default=1) The actual code to cause the crash: import pickle from import FilteredRelation, Q, Sum from import Publication, Session p1 = (title='Foo') p2 = (title='Bar') (publication=p1, sessiontype='A', place='X', value=1) (publication=p1, sessiontype='B', place='X', value=2) (publication=p2, sessiontype='A', place='X', value=4) (publication=p2, sessiontype='B', place='X', value=8) (publication=p1, sessiontype='A', place='Y', value=1) (publication=p1, sessiontype='B', place='Y', value=2) (publication=p2, sessiontype='A', place='Y', value=4) (publication=p2, sessiontype='B', place='Y', value=8) qs = ().annotate( relevantsessions=FilteredRelation('session', condition=Q(sessionsessiontype='A')) ).annotate(x=Sum('relevantsessionsvalue')) print(list(qs)) qs2 = () = (()) print(list(qs2)) In the crashing query, there seems to be a difference in the tablemap attribute - this is probably where the t3 table is coming from. Please let me know if there is any more info required for hunting this down. Cheers Beda p.s.- I also tried in Django 3.1 and the behavior is the same. p.p.s.- just to make sure, I am not interested in ideas on how to rewrite the query - the above is a very simplified version of what I use, so it would probably not be applicable anyway.

from django.core import exceptions from django.utils.functional import cached_property<ins>from django.utils.hashable import make_hashable</ins>  from . import BLANK_CHOICE_DASH from .mixins import FieldCacheMixin


Question: Queryset crashes when recreated from a pickled query with FilteredRelation used in aggregation. Description I am pickling query objects () for later re-evaluation as per  However, when I tried to rerun a query that contains a FilteredRelation inside a filter, I get an missing FROM-clause entry for table "t3" error. I created a minimum reproducible example. from import models class Publication(models.Model): title = models.CharField(maxlength=64) class Session(models.Model): TYPECHOICES = (('A', 'A'), ('B', 'B')) publication = models.ForeignKey(Publication, ondelete=models.CASCADE) sessiontype = models.CharField(choices=TYPECHOICES, default='A', maxlength=1) place = models.CharField(maxlength=16) value = models.PositiveIntegerField(default=1) The actual code to cause the crash: import pickle from import FilteredRelation, Q, Sum from import Publication, Session p1 = (title='Foo') p2 = (title='Bar') (publication=p1, sessiontype='A', place='X', value=1) (publication=p1, sessiontype='B', place='X', value=2) (publication=p2, sessiontype='A', place='X', value=4) (publication=p2, sessiontype='B', place='X', value=8) (publication=p1, sessiontype='A', place='Y', value=1) (publication=p1, sessiontype='B', place='Y', value=2) (publication=p2, sessiontype='A', place='Y', value=4) (publication=p2, sessiontype='B', place='Y', value=8) qs = ().annotate( relevantsessions=FilteredRelation('session', condition=Q(sessionsessiontype='A')) ).annotate(x=Sum('relevantsessionsvalue')) print(list(qs)) qs2 = () = (()) print(list(qs2)) In the crashing query, there seems to be a difference in the tablemap attribute - this is probably where the t3 table is coming from. Please let me know if there is any more info required for hunting this down. Cheers Beda p.s.- I also tried in Django 3.1 and the behavior is the same. p.p.s.- just to make sure, I am not interested in ideas on how to rewrite the query - the above is a very simplified version of what I use, so it would probably not be applicable anyway.

self.related_model._meta.model_name,         ) <ins>    @property    def identity(self):        return (            self.field,            self.model,            self.related_name,            self.related_query_name,            tuple(sorted(make_hashable(self.limit_choices_to))),            self.parent_link,            self.on_delete,            self.symmetrical,            self.multiple,        )    def __eq__(self, other):        if not isinstance(other, self.__class__):            return NotImplemented        return self.identity == other.identity    def __hash__(self):        return hash(self.identity)</ins>     def get_choices(         self, include_blank=True, blank_choice=BLANK_CHOICE_DASH,         limit_choices_to=None, ordering=(),


Question: Queryset crashes when recreated from a pickled query with FilteredRelation used in aggregation. Description I am pickling query objects () for later re-evaluation as per  However, when I tried to rerun a query that contains a FilteredRelation inside a filter, I get an missing FROM-clause entry for table "t3" error. I created a minimum reproducible example. from import models class Publication(models.Model): title = models.CharField(maxlength=64) class Session(models.Model): TYPECHOICES = (('A', 'A'), ('B', 'B')) publication = models.ForeignKey(Publication, ondelete=models.CASCADE) sessiontype = models.CharField(choices=TYPECHOICES, default='A', maxlength=1) place = models.CharField(maxlength=16) value = models.PositiveIntegerField(default=1) The actual code to cause the crash: import pickle from import FilteredRelation, Q, Sum from import Publication, Session p1 = (title='Foo') p2 = (title='Bar') (publication=p1, sessiontype='A', place='X', value=1) (publication=p1, sessiontype='B', place='X', value=2) (publication=p2, sessiontype='A', place='X', value=4) (publication=p2, sessiontype='B', place='X', value=8) (publication=p1, sessiontype='A', place='Y', value=1) (publication=p1, sessiontype='B', place='Y', value=2) (publication=p2, sessiontype='A', place='Y', value=4) (publication=p2, sessiontype='B', place='Y', value=8) qs = ().annotate( relevantsessions=FilteredRelation('session', condition=Q(sessionsessiontype='A')) ).annotate(x=Sum('relevantsessionsvalue')) print(list(qs)) qs2 = () = (()) print(list(qs2)) In the crashing query, there seems to be a difference in the tablemap attribute - this is probably where the t3 table is coming from. Please let me know if there is any more info required for hunting this down. Cheers Beda p.s.- I also tried in Django 3.1 and the behavior is the same. p.p.s.- just to make sure, I am not interested in ideas on how to rewrite the query - the above is a very simplified version of what I use, so it would probably not be applicable anyway.

state.pop('related_model', None)         return state <ins>    @property    def identity(self):        return super().identity + (self.field_name,)</ins>     def get_related_field(self):         """         Return the Field in the 'to' object to which this relationship is tied.


Question: Queryset crashes when recreated from a pickled query with FilteredRelation used in aggregation. Description I am pickling query objects () for later re-evaluation as per  However, when I tried to rerun a query that contains a FilteredRelation inside a filter, I get an missing FROM-clause entry for table "t3" error. I created a minimum reproducible example. from import models class Publication(models.Model): title = models.CharField(maxlength=64) class Session(models.Model): TYPECHOICES = (('A', 'A'), ('B', 'B')) publication = models.ForeignKey(Publication, ondelete=models.CASCADE) sessiontype = models.CharField(choices=TYPECHOICES, default='A', maxlength=1) place = models.CharField(maxlength=16) value = models.PositiveIntegerField(default=1) The actual code to cause the crash: import pickle from import FilteredRelation, Q, Sum from import Publication, Session p1 = (title='Foo') p2 = (title='Bar') (publication=p1, sessiontype='A', place='X', value=1) (publication=p1, sessiontype='B', place='X', value=2) (publication=p2, sessiontype='A', place='X', value=4) (publication=p2, sessiontype='B', place='X', value=8) (publication=p1, sessiontype='A', place='Y', value=1) (publication=p1, sessiontype='B', place='Y', value=2) (publication=p2, sessiontype='A', place='Y', value=4) (publication=p2, sessiontype='B', place='Y', value=8) qs = ().annotate( relevantsessions=FilteredRelation('session', condition=Q(sessionsessiontype='A')) ).annotate(x=Sum('relevantsessionsvalue')) print(list(qs)) qs2 = () = (()) print(list(qs2)) In the crashing query, there seems to be a difference in the tablemap attribute - this is probably where the t3 table is coming from. Please let me know if there is any more info required for hunting this down. Cheers Beda p.s.- I also tried in Django 3.1 and the behavior is the same. p.p.s.- just to make sure, I am not interested in ideas on how to rewrite the query - the above is a very simplified version of what I use, so it would probably not be applicable anyway.

self.symmetrical = symmetrical         self.db_constraint = db_constraint <ins>    @property    def identity(self):        return super().identity + (            self.through,            self.through_fields,            self.db_constraint,        )</ins>     def get_related_field(self):         """         Return the field in the 'to' object to which this relationship is tied.


Question: Raise a descriptive error on get() with filters operation following (), intersection(), and difference(). Description The QuerySet resulting from q1.difference(q2) does not allow to retrieve one specific object of that resulting QuerySet. Minimal example: from import models class Test(models.Model): name = models.CharField(max_length=30) python shell from import Test qs1 = () qs1 <QuerySet [<Test: Test object (1), <Test: Test object (2), <Test: Test object (3)]qs2 = (id=1) qs2 <QuerySet [<Test: Test object (1)]qs3 = qs1.difference(qs2) qs3 <QuerySet <Test: Test object (2), <Test: Test object (3) Traceback (most recent call last): File "<console", line 1, in <moduleFile "/lib/python3.8/site-", line 433, in get raise ( get() returned more than one Test -- it returned 2! Django version: 3.1.2 Python version: 3.8.5 OS: Arch Linux I also experienced this in the regular request/view-context, with other Django versions (2.2) and other python versions (3.7). Sorry if this is the expected behavior, a known bug or if I missed something which changes the behavior only on my system. If you need more information, I'll be happy to assist.

Perform the query and return a single object matching the given         keyword arguments.         """<ins>        if self.query.combinator and (args or kwargs):            raise NotSupportedError(                'Calling QuerySet.get(...) with filters after %s() is not '                'supported.' % self.query.combinator            )</ins>         clone = self._chain() if self.query.combinator else self.filter(*args, **kwargs)         if self.query.can_filter() and not self.query.distinct_fields:             clone = clone.order_by()


Question: Admin doesn't display properly unicode chars in JSONFields. Description (last modified by ZhaoQi99) import json print ('') "\u4e2d\u56fd" use ASCII encoding by default when serializing Chinese. So when we edit a JsonField which contains Chinese character in Django admin,it will appear in ASCII characters. I have try to fix this this problem in  it works prefectly.

import datetime import decimal<ins>import json</ins> from collections import defaultdict  from django.core.exceptions import FieldDoesNotExist


Question: Admin doesn't display properly unicode chars in JSONFields. Description (last modified by ZhaoQi99) import json print ('') "\u4e2d\u56fd" use ASCII encoding by default when serializing Chinese. So when we edit a JsonField which contains Chinese character in Django admin,it will appear in ASCII characters. I have try to fix this this problem in  it works prefectly.

return format_html('{}', value.url, value)     elif isinstance(field, models.JSONField) and value:         try:            return field.get_prep_value(value)            return json.dumps(value, ensure_ascii=False, cls=field.encoder)         except TypeError:             return display_for_value(value, empty_value_display)     else:diff --git a/django/forms/fields.py b/django/forms/fields.py-- a/django/forms/fields.py++ b/django/forms/fields.py


Question: Admin doesn't display properly unicode chars in JSONFields. Description (last modified by ZhaoQi99) import json print ('') "\u4e2d\u56fd" use ASCII encoding by default when serializing Chinese. So when we edit a JsonField which contains Chinese character in Django admin,it will appear in ASCII characters. I have try to fix this this problem in  it works prefectly.

def prepare_value(self, value):         if isinstance(value, InvalidJSONInput):             return value        return json.dumps(value, cls=self.encoder)        return json.dumps(value, ensure_ascii=False, cls=self.encoder)      def has_changed(self, initial, data):         if super().has_changed(initial, data):


Question: debug error view doesn't respect exc.suppresscontext (PEP 415) Description Consider the following view that raises an exception: class TestView(View): def get(self, request, args, *kwargs): try: raise RuntimeError('my error') except Exception as exc: raise ValueError('my new error') from None Even though the raise is from None, unlike the traceback Python shows, the debug error view still shows the RuntimeError. This is because the explicitorimplicitcause() function inside gettracebackframes() doesn't respect exc.suppresscontext, which was introduced in Python 3.3's PEP 415:  def gettracebackframes(self): def explicitorimplicitcause(excvalue): explicit = getattr(excvalue, 'cause', None) implicit = getattr(excvalue, 'context', None) return explicit or implicit Instead, it should be something more like (simplifying also for Python 3): def explicitorimplicitcause(excvalue): return ( or (None if excvalue.suppresscontext else exc_value.context) )

return None, [], None, []         return lower_bound, pre_context, context_line, post_context     def get_traceback_frames(self):        def explicit_or_implicit_cause(exc_value):            explicit = getattr(exc_value, '__cause__', None)            suppress_context = getattr(exc_value, '__suppress_context__', None)            implicit = getattr(exc_value, '__context__', None)            return explicit or (None if suppress_context else implicit)    def _get_explicit_or_implicit_cause(self, exc_value):        explicit = getattr(exc_value, '__cause__', None)        suppress_context = getattr(exc_value, '__suppress_context__', None)        implicit = getattr(exc_value, '__context__', None)        return explicit or (None if suppress_context else implicit)     def get_traceback_frames(self):         # Get the exception and all its causes         exceptions = []         exc_value = self.exc_value         while exc_value:             exceptions.append(exc_value)            exc_value = explicit_or_implicit_cause(exc_value)            exc_value = self._get_explicit_or_implicit_cause(exc_value)             if exc_value in exceptions:                 warnings.warn(                     "Cycle in the exception chain detected: exception '%s' "


Question: debug error view doesn't respect exc.suppresscontext (PEP 415) Description Consider the following view that raises an exception: class TestView(View): def get(self, request, args, *kwargs): try: raise RuntimeError('my error') except Exception as exc: raise ValueError('my new error') from None Even though the raise is from None, unlike the traceback Python shows, the debug error view still shows the RuntimeError. This is because the explicitorimplicitcause() function inside gettracebackframes() doesn't respect exc.suppresscontext, which was introduced in Python 3.3's PEP 415:  def gettracebackframes(self): def explicitorimplicitcause(excvalue): explicit = getattr(excvalue, 'cause', None) implicit = getattr(excvalue, 'context', None) return explicit or implicit Instead, it should be something more like (simplifying also for Python 3): def explicitorimplicitcause(excvalue): return ( or (None if excvalue.suppresscontext else exc_value.context) )

# In case there's just one exception, take the traceback from self.tb         exc_value = exceptions.pop()         tb = self.tb if not exceptions else exc_value.__traceback__<ins>        frames.extend(self.get_exception_traceback_frames(exc_value, tb))        while exceptions:            exc_value = exceptions.pop()            frames.extend(                self.get_exception_traceback_frames(exc_value, exc_value.__traceback__),            )        return frames    def get_exception_traceback_frames(self, exc_value, tb):        exc_cause = self._get_explicit_or_implicit_cause(exc_value)        exc_cause_explicit = getattr(exc_value, '__cause__', True)</ins>          while tb is not None:             # Support for __traceback_hide__ which is used by a few libraries


Question: debug error view doesn't respect exc.suppresscontext (PEP 415) Description Consider the following view that raises an exception: class TestView(View): def get(self, request, args, *kwargs): try: raise RuntimeError('my error') except Exception as exc: raise ValueError('my new error') from None Even though the raise is from None, unlike the traceback Python shows, the debug error view still shows the RuntimeError. This is because the explicitorimplicitcause() function inside gettracebackframes() doesn't respect exc.suppresscontext, which was introduced in Python 3.3's PEP 415:  def gettracebackframes(self): def explicitorimplicitcause(excvalue): explicit = getattr(excvalue, 'cause', None) implicit = getattr(excvalue, 'context', None) return explicit or implicit Instead, it should be something more like (simplifying also for Python 3): def explicitorimplicitcause(excvalue): return ( or (None if excvalue.suppresscontext else exc_value.context) )

pre_context = []                 context_line = '<source code not available>'                 post_context = []            frames.append({                'exc_cause': explicit_or_implicit_cause(exc_value),                'exc_cause_explicit': getattr(exc_value, '__cause__', True),            yield {                'exc_cause': exc_cause,                'exc_cause_explicit': exc_cause_explicit,                 'tb': tb,                 'type': 'django' if module_name.startswith('django.') else 'user',                 'filename': filename,


Question: debug error view doesn't respect exc.suppresscontext (PEP 415) Description Consider the following view that raises an exception: class TestView(View): def get(self, request, args, *kwargs): try: raise RuntimeError('my error') except Exception as exc: raise ValueError('my new error') from None Even though the raise is from None, unlike the traceback Python shows, the debug error view still shows the RuntimeError. This is because the explicitorimplicitcause() function inside gettracebackframes() doesn't respect exc.suppresscontext, which was introduced in Python 3.3's PEP 415:  def gettracebackframes(self): def explicitorimplicitcause(excvalue): explicit = getattr(excvalue, 'cause', None) implicit = getattr(excvalue, 'context', None) return explicit or implicit Instead, it should be something more like (simplifying also for Python 3): def explicitorimplicitcause(excvalue): return ( or (None if excvalue.suppresscontext else exc_value.context) )

'context_line': context_line,                 'post_context': post_context,                 'pre_context_lineno': pre_context_lineno + 1,            })            # If the traceback for current exception is consumed, try the            # other exception.            if not tb.tb_next and exceptions:                exc_value = exceptions.pop()                tb = exc_value.__traceback__            else:                tb = tb.tb_next        return frames            }            tb = tb.tb_next   def technical_404_response(request, exception):


Question: flush() on management commands doesn't work. Description flush() is notably called during migrate command; it doesn't work, and a long migration effectively prints to stderr no relevant information up until the end: Operations to perform: Apply all migrations: myapp Running migrations: Then nothing more, but the migration is being done. Then at the end of the real migration, the rest is flushed: Applying myapp.0002auto202008171030... OK Expected behavior: Operations to perform: Apply all migrations: myapp Running migrations: Applying myapp.0002auto202008171030... then work then OK

def __getattr__(self, name):         return getattr(self._out, name) <ins>    def flush(self):        if hasattr(self._out, 'flush'):            self._out.flush()</ins>     def isatty(self):         return hasattr(self._out, 'isatty') and self._out.isatty()


Question: Incorrect separators when chaining floatformat to intcomma in some locales Description When u use floatvalue "2" and intcomma together in a template the output of intcomma won't be internationalized. Since intcomma wont work with decimals in django 1.5.1 i tried to convert a decimal to a float in a template, but it wont give me the excepted output. When i have the value of 1000.11 it should be 1000,11 in germany, with intcomma(float(1000,11)) i get 1.000,11. But when i use Decimal(1000,11)intcomma, i will get 1,000,11. Thats a bug or maybe an unwanted behavior.

* {{ num2|floatformat:"-3" }} displays "34"     * {{ num3|floatformat:"-3" }} displays "34.260"     If arg has the 'g' suffix, force the result to be grouped by the    THOUSAND_SEPARATOR for the active locale. When the active locale is    en (English):    * {{ 6666.6666|floatformat:"2g" }} displays "6,666.67"    * {{ 10000|floatformat:"g" }} displays "10,000"     If the input float is infinity or NaN, display the string representation     of that value.     """    force_grouping = False    if isinstance(arg, str) and arg.endswith('g'):        force_grouping = True        arg = arg[:-1] or -1     try:         input_val = repr(text)         d = Decimal(input_val)


Question: Incorrect separators when chaining floatformat to intcomma in some locales Description When u use floatvalue "2" and intcomma together in a template the output of intcomma won't be internationalized. Since intcomma wont work with decimals in django 1.5.1 i tried to convert a decimal to a float in a template, but it wont give me the excepted output. When i have the value of 1000.11 it should be 1000,11 in germany, with intcomma(float(1000,11)) i get 1.000,11. But when i use Decimal(1000,11)intcomma, i will get 1,000,11. Thats a bug or maybe an unwanted behavior.

return input_val      if not m and p < 0:        return mark_safe(formats.number_format('%d' % (int(d)), 0))        return mark_safe(            formats.number_format('%d' % (int(d)), 0, force_grouping=force_grouping),        )      exp = Decimal(1).scaleb(-abs(p))     # Set the precision high enough to avoid an exception (#15789).


Question: Incorrect separators when chaining floatformat to intcomma in some locales Description When u use floatvalue "2" and intcomma together in a template the output of intcomma won't be internationalized. Since intcomma wont work with decimals in django 1.5.1 i tried to convert a decimal to a float in a template, but it wont give me the excepted output. When i have the value of 1000.11 it should be 1000,11 in germany, with intcomma(float(1000,11)) i get 1.000,11. But when i use Decimal(1000,11)intcomma, i will get 1,000,11. Thats a bug or maybe an unwanted behavior.

if sign and rounded_d:         digits.append('-')     number = ''.join(reversed(digits))    return mark_safe(formats.number_format(number, abs(p)))    return mark_safe(        formats.number_format(number, abs(p), force_grouping=force_grouping),    )   @register.filter(is_safe=True)


Question: Using KeyTransform for JSONField produces invalid SQL in various places. Description (last modified by Igor Jerosimić) Using KeyTransform in ordering attribute of ArrayAgg function produces invalid SQL. I don't know if it matters but I'm using Postgres for DB. from import models class Parent(models.Model): name = models.CharField(default='test') class Child(models.Model): parent = models.ForeignKey( Parent, ondelete=models.SETNULL, relatedname='children', ) data = models.JSONField(default=dict) parent = () (parent=parent, data={'en': 'English', 'fr': 'French'}) Parent.objects.annotate( childrenarray=ArrayAgg( KeyTextTransform('en', 'childrendata'), distinct=True, ordering=[KeyTransform('en', 'childrendata')], ), ).all() Produces invalid SQL in the ORDER BY section: ARRAY_AGG(DISTINCT ("children"."data" -'default') ORDER BY None("children"."data")) NOTE: This was working fine before Django 3.1.

ordering_params = []             ordering_expr_sql = []             for expr in self.ordering:                expr_sql, expr_params = expr.as_sql(compiler, connection)                expr_sql, expr_params = compiler.compile(expr)                 ordering_expr_sql.append(expr_sql)                 ordering_params.extend(expr_params)             sql, sql_params = super().as_sql(compiler, connection, ordering=(diff --git a/django/contrib/postgres/constraints.py b/django/contrib/postgres/constraints.py-- a/django/contrib/postgres/constraints.py++ b/django/contrib/postgres/constraints.py


Question: Using KeyTransform for JSONField produces invalid SQL in various places. Description (last modified by Igor Jerosimić) Using KeyTransform in ordering attribute of ArrayAgg function produces invalid SQL. I don't know if it matters but I'm using Postgres for DB. from import models class Parent(models.Model): name = models.CharField(default='test') class Child(models.Model): parent = models.ForeignKey( Parent, ondelete=models.SETNULL, relatedname='children', ) data = models.JSONField(default=dict) parent = () (parent=parent, data={'en': 'English', 'fr': 'French'}) Parent.objects.annotate( childrenarray=ArrayAgg( KeyTextTransform('en', 'childrendata'), distinct=True, ordering=[KeyTransform('en', 'childrendata')], ), ).all() Produces invalid SQL in the ORDER BY section: ARRAY_AGG(DISTINCT ("children"."data" -'default') ORDER BY None("children"."data")) NOTE: This was working fine before Django 3.1.

if isinstance(expression, str):                 expression = F(expression)             expression = expression.resolve_expression(query=query)            sql, params = expression.as_sql(compiler, schema_editor.connection)            sql, params = compiler.compile(expression)             try:                 opclass = self.opclasses[idx]                 if opclass:diff --git a/django/db/models/expressions.py b/django/db/models/expressions.py-- a/django/db/models/expressions.py++ b/django/db/models/expressions.py


Question: Using KeyTransform for JSONField produces invalid SQL in various places. Description (last modified by Igor Jerosimić) Using KeyTransform in ordering attribute of ArrayAgg function produces invalid SQL. I don't know if it matters but I'm using Postgres for DB. from import models class Parent(models.Model): name = models.CharField(default='test') class Child(models.Model): parent = models.ForeignKey( Parent, ondelete=models.SETNULL, relatedname='children', ) data = models.JSONField(default=dict) parent = () (parent=parent, data={'en': 'English', 'fr': 'French'}) Parent.objects.annotate( childrenarray=ArrayAgg( KeyTextTransform('en', 'childrendata'), distinct=True, ordering=[KeyTransform('en', 'childrendata')], ), ).all() Produces invalid SQL in the ORDER BY section: ARRAY_AGG(DISTINCT ("children"."data" -'default') ORDER BY None("children"."data")) NOTE: This was working fine before Django 3.1.

return expression.get_group_by_cols(alias=alias)      def as_sql(self, compiler, connection):        return self.expression.as_sql(compiler, connection)        return compiler.compile(self.expression)      def __repr__(self):         return "{}({})".format(self.__class__.__name__, self.expression)diff --git a/django/db/models/fields/json.py b/django/db/models/fields/json.py-- a/django/db/models/fields/json.py++ b/django/db/models/fields/json.py


Question: Using KeyTransform for JSONField produces invalid SQL in various places. Description (last modified by Igor Jerosimić) Using KeyTransform in ordering attribute of ArrayAgg function produces invalid SQL. I don't know if it matters but I'm using Postgres for DB. from import models class Parent(models.Model): name = models.CharField(default='test') class Child(models.Model): parent = models.ForeignKey( Parent, ondelete=models.SETNULL, relatedname='children', ) data = models.JSONField(default=dict) parent = () (parent=parent, data={'en': 'English', 'fr': 'French'}) Parent.objects.annotate( childrenarray=ArrayAgg( KeyTextTransform('en', 'childrendata'), distinct=True, ordering=[KeyTransform('en', 'childrendata')], ), ).all() Produces invalid SQL in the ORDER BY section: ARRAY_AGG(DISTINCT ("children"."data" -'default') ORDER BY None("children"."data")) NOTE: This was working fine before Django 3.1.

class KeyTransformIn(lookups.In):    def process_rhs(self, compiler, connection):        rhs, rhs_params = super().process_rhs(compiler, connection)        if not connection.features.has_native_json_field:            func = ()    def resolve_expression_parameter(self, compiler, connection, sql, param):        sql, params = super().resolve_expression_parameter(            compiler, connection, sql, param,        )        if (            not hasattr(param, 'as_sql') and            not connection.features.has_native_json_field        ):             if connection.vendor == 'oracle':                func = []                for value in rhs_params:                    value = json.loads(value)                    function = 'JSON_QUERY' if isinstance(value, (list, dict)) else 'JSON_VALUE'                    func.append("%s('%s', '$.value')" % (                        function,                        json.dumps({'value': value}),                    ))                func = tuple(func)                rhs_params = ()            elif connection.vendor == 'mysql' and connection.mysql_is_mariadb:                func = ("JSON_UNQUOTE(JSON_EXTRACT(%s, '$'))",) * len(rhs_params)                value = json.loads(param)                if isinstance(value, (list, dict)):                    sql = "JSON_QUERY(%s, '$.value')"                else:                    sql = "JSON_VALUE(%s, '$.value')"                params = (json.dumps({'value': value}),)             elif connection.vendor in {'sqlite', 'mysql'}:                func = ("JSON_EXTRACT(%s, '$')",) * len(rhs_params)            rhs = rhs % func        return rhs, rhs_params                sql = "JSON_EXTRACT(%s, '$')"        if connection.vendor == 'mysql' and connection.mysql_is_mariadb:            sql = 'JSON_UNQUOTE(%s)' % sql        return sql, params   class KeyTransformExact(JSONExact):diff --git a/django/db/models/lookups.py b/django/db/models/lookups.py-- a/django/db/models/lookups.py++ b/django/db/models/lookups.py


Question: Using KeyTransform for JSONField produces invalid SQL in various places. Description (last modified by Igor Jerosimić) Using KeyTransform in ordering attribute of ArrayAgg function produces invalid SQL. I don't know if it matters but I'm using Postgres for DB. from import models class Parent(models.Model): name = models.CharField(default='test') class Child(models.Model): parent = models.ForeignKey( Parent, ondelete=models.SETNULL, relatedname='children', ) data = models.JSONField(default=dict) parent = () (parent=parent, data={'en': 'English', 'fr': 'French'}) Parent.objects.annotate( childrenarray=ArrayAgg( KeyTextTransform('en', 'childrendata'), distinct=True, ordering=[KeyTransform('en', 'childrendata')], ), ).all() Produces invalid SQL in the ORDER BY section: ARRAY_AGG(DISTINCT ("children"."data" -'default') ORDER BY None("children"."data")) NOTE: This was working fine before Django 3.1.

if hasattr(param, 'resolve_expression'):             param = param.resolve_expression(compiler.query)         if hasattr(param, 'as_sql'):            sql, params = param.as_sql(compiler, connection)            sql, params = compiler.compile(param)         return sql, params      def batch_process_rhs(self, compiler, connection, rhs=None):


Question: MySQL: dbshell does not get charset from DATABASES setting Description I noticed that dbshell doesn't respect the databaseoptions. I ran into an issue with an application we are creating that needs to support mysql and postgre at least, we execute some sql scripts that get piped to dbshell (to avoid hardcoding psql -U xxx or mysql -u xxx and creating 2 scripts every time). When running an utf8 database with utf8 as our charset in databaseoptions, we ran into some weird encoding issues. The solution for us was to learn to respect the encoding settings in Are you opposed to something like this? Attaching small patch that fixes our problem. Let me know if it needs extending to support other backends or database_options.

client_cert = settings_dict['OPTIONS'].get('ssl', {}).get('cert')         client_key = settings_dict['OPTIONS'].get('ssl', {}).get('key')         defaults_file = settings_dict['OPTIONS'].get('read_default_file')<ins>        charset = settings_dict['OPTIONS'].get('charset')</ins>         # Seems to be no good way to set sql_mode with CLI.          if defaults_file:


Question: MySQL: dbshell does not get charset from DATABASES setting Description I noticed that dbshell doesn't respect the databaseoptions. I ran into an issue with an application we are creating that needs to support mysql and postgre at least, we execute some sql scripts that get piped to dbshell (to avoid hardcoding psql -U xxx or mysql -u xxx and creating 2 scripts every time). When running an utf8 database with utf8 as our charset in databaseoptions, we ran into some weird encoding issues. The solution for us was to learn to respect the encoding settings in Are you opposed to something like this? Attaching small patch that fixes our problem. Let me know if it needs extending to support other backends or database_options.

args += ["--ssl-cert=%s" % client_cert]         if client_key:             args += ["--ssl-key=%s" % client_key]<ins>        if charset:            args += ['--default-character-set=%s' % charset]</ins>         if db:             args += [db]         args.extend(parameters)


Question: Changing user's email could invalidate password reset tokens Description Sequence: Have account with email address foo Password reset request for that email (unused) foo account changes their email address Password reset email is used The password reset email's token should be rejected at that point, but in fact it is allowed. The fix is to add the user's email address into PasswordResetTokenGenerator.makehashvalue() Nothing forces a user to even have an email as per AbstractBaseUser. Perhaps the token generation method could be factored out onto the model, ala getsessionauthhash().

def _make_hash_value(self, user, timestamp):         """        Hash the user's primary key and some user state that's sure to change        after a password reset to produce a token that invalidated when it's        used:        Hash the user's primary key, email (if available), and some user state        that's sure to change after a password reset to produce a token that is        invalidated when it's used:         1. The password field will change upon a password reset (even if the            same password is chosen, due to password salting).         2. The last_login field will usually be updated very shortly after


Question: Changing user's email could invalidate password reset tokens Description Sequence: Have account with email address foo Password reset request for that email (unused) foo account changes their email address Password reset email is used The password reset email's token should be rejected at that point, but in fact it is allowed. The fix is to add the user's email address into PasswordResetTokenGenerator.makehashvalue() Nothing forces a user to even have an email as per AbstractBaseUser. Perhaps the token generation method could be factored out onto the model, ala getsessionauthhash().

# Truncate microseconds so that tokens are consistent even if the         # database doesn't support microseconds.         login_timestamp = '' if user.last_login is None else user.last_login.replace(microsecond=0, tzinfo=None)        return str(user.pk) + user.password + str(login_timestamp) + str(timestamp)        email_field = user.get_email_field_name()        email = getattr(user, email_field, '') or ''        return f'{user.pk}{user.password}{login_timestamp}{timestamp}{email}'      def _num_seconds(self, dt):         return int((dt - datetime(2001, 1, 1)).total_seconds())


Question: QuerySet.orderby() crashes on union() queryset with a single non-empty query. Description (last modified by Hannes Ljungberg) I noticed this error while implementing a dynamic union like this: union = reduce( lambda qs1, qs2: (qs2), querysets, (), ) If len(querysets) == 1 it will result in a pretty weird query throwing a database error when both querysets are ordered, another example recreating this in the test suite: qs1 = ().orderby(‘pk’) qs2 = ().union(qs1).orderby(‘pk’) Executing qs2 result in the following query: (SELECT "queriesnumber"."id", "queriesnumber"."num", "queriesnumber"."othernum", "queriesnumber"."anothernum" FROM "queriesnumber" ORDER BY "queriesnumber"."id" ASC) ORDER BY (1) ASC Result in the following error on PostgresSQL: multiple ORDER BY clauses not allowed LINE 1: " ORDER BY "queriesnumber"."id" DESC) ORDER BY (1) ASC And sqlite: ORDER BY not allowed in subqueries of compound statements. MySQL seems to accept this query but it's most likely not the query the user expects the be executed. My proposal is to simply return the non-empty query and make the union a no-op set in this case.

# If the query is an EmptyQuerySet, combine all nonempty querysets.         if isinstance(self, EmptyQuerySet):             qs = [q for q in other_qs if not isinstance(q, EmptyQuerySet)]            return qs[0]._combinator_query('union', *qs[1:], all=all) if qs else self            if not qs:                return self            if len(qs) == 1:                return qs[0]            return qs[0]._combinator_query('union', *qs[1:], all=all)         return self._combinator_query('union', *other_qs, all=all)      def intersection(self, *other_qs):


Question: Allowing null values while registering function on transaction oncommit Description So if I were to try this command:- transaction.oncommit(None) and run any sample tests which has simple django db post_save signal in the flow and it has functions which run on transaction commit, the code would crash with error TypeError: 'NoneType' object is not callable. So how about if we do not allow it to register as none beforehand or allow it to exit gracefully?

return self.SchemaEditorClass(self, *args, **kwargs)      def on_commit(self, func):<ins>        if not callable(func):            raise TypeError("on_commit()'s callback must be a callable.")</ins>         if self.in_atomic_block:             # Transaction in progress; save for execution on commit.             self.run_on_commit.append((set(self.savepoint_ids), func))


Question: ProtectedError/RestrictedError raised from Collector.collect contains iterators. Description (last modified by Kwist) ("Cannot delete some instances of model 'A' because they are referenced through protected foreign keys: 'A.protect', 'B.protect'.", object at 0x7f96b5cb6c40) As a result, the repr of exception in Sentry is not informative.

model.__name__,                     ', '.join(protected_objects),                 ),                chain.from_iterable(protected_objects.values()),                set(chain.from_iterable(protected_objects.values())),             )         for related_model, related_fields in model_fast_deletes.items():             batches = self.get_del_batches(new_objs, related_fields)


Question: ProtectedError/RestrictedError raised from Collector.collect contains iterators. Description (last modified by Kwist) ("Cannot delete some instances of model 'A' because they are referenced through protected foreign keys: 'A.protect', 'B.protect'.", object at 0x7f96b5cb6c40) As a result, the repr of exception in Sentry is not informative.

model.__name__,                             ', '.join(restricted_objects),                         ),                        chain.from_iterable(restricted_objects.values()),                        set(chain.from_iterable(restricted_objects.values())),                     )      def related_objects(self, related_model, related_fields, objs):


Question: Skip auth.E003 system check for USERNAMEFIELD with total UniqueConstraints. Description Defining a user model like this: class User(AbstractBaseUser): username = models.CharField(maxlength=30) USERNAMEFIELD = "username" class Meta: constraints = [UniqueConstraint(fields=["username"], name="userusernameunq")] Will trigger auth.E003: auth.User: (auth.E003) 'User.username' must be unique because it is named as the 'USERNAMEFIELD'. Sometimes it’s not preferable to set the field as unique with unique=True as it will create an extra implicit *like index for CharField and TextField on PostgresSQL. The system check should be extended to check for the presence of USERNAMEFIELD in Model._meta.constraints. Not really sure if this classifies as a bug.

)      # Check that the username field is unique    if not cls._meta.get_field(cls.USERNAME_FIELD).unique:    if not cls._meta.get_field(cls.USERNAME_FIELD).unique and not any(        constraint.fields == (cls.USERNAME_FIELD,)        for constraint in cls._meta.total_unique_constraints    ):         if (settings.AUTHENTICATION_BACKENDS ==                 ['django.contrib.auth.backends.ModelBackend']):             errors.append(


Question: orderby('?') unexpectedly breaking queryset aggregation Description Steps to reproduce: class Thing(models.Model): pass class Related(models.Model): models.ForeignKey(Thing) With data t = () rs = [(thing=t) for in range(2)] The following query works as expected. The aggregation with Count produces a GROUP BY clause on Thing.objects.annotate(rc=Count('related')).orderby('rc').values('id', 'rc') <QuerySet [{'id': 1, 'rc': 2}]This also works as expected (at least to me). Although there is an aggregation, ordering by related means that the grouping will be broken down. Thing.objects.annotate(rc=Count('related')).orderby('related').values('id', 'rc') <QuerySet [{'id': 1, 'rc': 1}, {'id': 1, 'rc': 1}]But the following seems wrong to me. Thing.objects.annotate(rc=Count('related')).orderby('?').values('id', 'rc') <QuerySet [{'id': 1, 'rc': 1}, {'id': 1, 'rc': 1}]The random function call has nothing to do with the aggregation, and I see no reason it should break it. Dumping the query seems that indeed the random call breaks the group by call: (I simpilfied the table names a little) print(Thing.objects.annotate(rc=Count('related')).orderby('?').values('id', 'rc').query) SELECT "thing"."id", COUNT("related"."id") AS "rc" FROM "thing" LEFT OUTER JOIN "related" ON ("thing"."id" = "related"."thingid") GROUP BY "thing"."id", RANDOM() ORDER BY RANDOM() ASC I dug into the SQL compiler, and it seems to me the problem is inside , where the compiler combines all non-aggregate, non-ref orderby expressions into groupby. I patched it like this for expr, (sql, params, isref) in orderby: if expr.containsaggregate: continue if isref: continue ([ exp for exp in expr.getsource_expressions() if not isinstance(exp, Random) ]) and things seem to work correctly. No failed tests against SQLite3 with default settings.

def as_sqlite(self, compiler, connection, **extra_context):         return super().as_sql(compiler, connection, function='RAND', **extra_context) <ins>    def get_group_by_cols(self, alias=None):        return []</ins>  class Round(Transform):     function = 'ROUND'


Question: ManagementForm exception in case of bad prefix should be easier to understand Description If user adds multiple formsets with prefixes, and specifies the prefix incorrectly when binding the form and validating: someformset = SomeFormSet(request.POST, 'articles') instead of: someformset = SomeFormSet(request.POST, prefix='articles') Django "suppresses" the original exception and raises only relatively unhelpful "ManagementForm data is missing or has been tampered with". In file , line 57: if or form = ManagementForm(, , ) if not form.isvalid(): raise ValidationError('ManagementForm data is missing or has been tampered with') Suggestion: include in output, because for such a small bug in code, it can take a really long time find it. {'INITIALFORMS': [u'This field is required.'], 'MAXNUMFORMS': [u'This field is required.'], 'TOTALFORMS': [u'This field is required.']}

form = ManagementForm(self.data, auto_id=self.auto_id, prefix=self.prefix)             if not form.is_valid():                 raise ValidationError(                    _('ManagementForm data is missing or has been tampered with'),                    _(                        'ManagementForm data is missing or has been tampered '                        'with. Missing fields: %(field_names)s'                    ) % {                        'field_names': ', '.join(                            form.add_prefix(field_name) for field_name in form.errors                        ),                    },                     code='missing_management_form',                 )         else:


Question: Password reset token incompatibility. Description As noted here  the hashing for password reset tokens has changed between 3.0 and 3.1 and work has been done to ensure existing tokens will still work (at least until 4.0). However the encoding of the token creation time has also changed. Specifically from days since 1/1/01 to seconds since 1/1/01. And it appears no work has been done to support tokens with the older values. So a token generated on Oct 1, 2020 will come through as 7213 days which will then get interpreted as 7213 seconds, aka 2am Jan 1, 2001. So while exiting tokens in the wild will pass crypto validation they will all show as expired if your PASSWORDRESETTIMEOUT is less than ~20 years. The code base I'm working on uses these tokens (perhaps unwisely) in some email links that are expected to have a 3 month lifetime and an upgrade from 3.0 to 3.1 looks likely to render all the tokens in the wild expired which is suboptimal.

from datetime import datetimefrom datetime import datetime, time  from django.conf import settings from django.utils.crypto import constant_time_compare, salted_hmac


Question: Password reset token incompatibility. Description As noted here  the hashing for password reset tokens has changed between 3.0 and 3.1 and work has been done to ensure existing tokens will still work (at least until 4.0). However the encoding of the token creation time has also changed. Specifically from days since 1/1/01 to seconds since 1/1/01. And it appears no work has been done to support tokens with the older values. So a token generated on Oct 1, 2020 will come through as 7213 days which will then get interpreted as 7213 seconds, aka 2am Jan 1, 2001. So while exiting tokens in the wild will pass crypto validation they will all show as expired if your PASSWORDRESETTIMEOUT is less than ~20 years. The code base I'm working on uses these tokens (perhaps unwisely) in some email links that are expected to have a 3 month lifetime and an upgrade from 3.0 to 3.1 looks likely to render all the tokens in the wild expired which is suboptimal.

# Parse the token         try:             ts_b36, _ = token.split("-")<ins>            # RemovedInDjango40Warning.            legacy_token = len(ts_b36) < 4</ins>         except ValueError:             return False


Question: Password reset token incompatibility. Description As noted here  the hashing for password reset tokens has changed between 3.0 and 3.1 and work has been done to ensure existing tokens will still work (at least until 4.0). However the encoding of the token creation time has also changed. Specifically from days since 1/1/01 to seconds since 1/1/01. And it appears no work has been done to support tokens with the older values. So a token generated on Oct 1, 2020 will come through as 7213 days which will then get interpreted as 7213 seconds, aka 2am Jan 1, 2001. So while exiting tokens in the wild will pass crypto validation they will all show as expired if your PASSWORDRESETTIMEOUT is less than ~20 years. The code base I'm working on uses these tokens (perhaps unwisely) in some email links that are expected to have a 3 month lifetime and an upgrade from 3.0 to 3.1 looks likely to render all the tokens in the wild expired which is suboptimal.

):                 return False         # RemovedInDjango40Warning: convert days to seconds and round to        # midnight (server time) for pre-Django 3.1 tokens.        now = self._now()        if legacy_token:            ts *= 24 * 60 * 60            ts += int((now - datetime.combine(now.date(), time.min)).total_seconds())         # Check the timestamp is within limit.        if (self._num_seconds(self._now()) - ts) > settings.PASSWORD_RESET_TIMEOUT:        if (self._num_seconds(now) - ts) > settings.PASSWORD_RESET_TIMEOUT:             return False          return True


Question: Saving parent object after setting on child leads to unexpected data loss in bulkcreate(). Description Example: class Country(models.Model): name = models.CharField(maxlength=255) isotwoletter = models.CharField(maxlength=2) description = models.TextField() class City(models.Model): name = models.CharField(maxlength=255) country = models.ForeignKey(Country, ondelete=models.CASCADE) class BulkCreateTests(TestCase): def testfkbug(self): countrynl = Country(name='Netherlands', isotwoletter='NL') countrybe = Country(name='Belgium', isotwoletter='BE') city = City(country=countrybe, name='Brussels') # (1) () # (2) () This results in an integrity error: ERROR: testfkbug () Traceback (most recent call last): File "", line 85, in execute return (sql, params) psycopg2.IntegrityError: null value in column "countryid" violates not-null constraint DETAIL: Failing row contains (1, Brussels, null). The above exception was the direct cause of the following exception: ... I wonder wether there's a reason that this doesn't work. If setting a related object on a model instance automatically sets instance.relatedobjectid, you'd expect this behavior to continue working if the related object receives its primary key after the referencing instance was initialized. Of course, switching lines (1) and (2) makes it all work, but this behavior means that bulk creating with related objects (with postgres returning the primary keys, bless it's heart) becomes more complex than it should be, forcing the user to use mappings or even set the foreign keys themselves: for countrydata, citydata in data: country = Country(countrydata) (country) city = City(country=country, citydata) (city) Country.objects.bulkcreate(countries) for city in cities: city.countryid = City.objects.bulk_create(cities) Ideally, the main instance would, when saved, 'reach into' its related objects and set the foreign key field like the loop does. Is there a reason this can't work?

that the "save" must be an SQL insert or update (or equivalent for         non-SQL backends), respectively. Normally, they should not be set.         """        # Ensure that a model instance without a PK hasn't been assigned to        # a ForeignKey or OneToOneField on this model. If the field is        # nullable, allowing the save() would result in silent data loss.        for field in self._meta.concrete_fields:            # If the related field isn't cached, then an instance hasn't            # been assigned and there's no need to worry about this check.            if field.is_relation and field.is_cached(self):                obj = getattr(self, field.name, None)                if not obj:                    continue                # A pk may have been assigned manually to a model instance not                # saved to the database (or auto-generated in a case like                # UUIDField), but we allow the save to proceed and rely on the                # database to raise an IntegrityError if applicable. If                # constraints aren't supported by the database, there's the                # unavoidable risk of data corruption.                if obj.pk is None:                    # Remove the object from a related instance cache.                    if not field.remote_field.multiple:                        field.remote_field.delete_cached_value(obj)                    raise ValueError(                        "save() prohibited to prevent data loss due to "                        "unsaved related object '%s'." % field.name                    )                elif getattr(self, field.attname) is None:                    # Use pk from related object if it has been saved after                    # an assignment.                    setattr(self, field.attname, obj.pk)                # If the relationship's pk/to_field was changed, clear the                # cached relationship.                if getattr(obj, field.target_field.attname) != getattr(self, field.attname):                    field.delete_cached_value(self)        self._prepare_related_fields_for_save(operation_name='save')          using = using or router.db_for_write(self.__class__, instance=self)         if force_insert and (force_update or update_fields):


Question: Saving parent object after setting on child leads to unexpected data loss in bulkcreate(). Description Example: class Country(models.Model): name = models.CharField(maxlength=255) isotwoletter = models.CharField(maxlength=2) description = models.TextField() class City(models.Model): name = models.CharField(maxlength=255) country = models.ForeignKey(Country, ondelete=models.CASCADE) class BulkCreateTests(TestCase): def testfkbug(self): countrynl = Country(name='Netherlands', isotwoletter='NL') countrybe = Country(name='Belgium', isotwoletter='BE') city = City(country=countrybe, name='Brussels') # (1) () # (2) () This results in an integrity error: ERROR: testfkbug () Traceback (most recent call last): File "", line 85, in execute return (sql, params) psycopg2.IntegrityError: null value in column "countryid" violates not-null constraint DETAIL: Failing row contains (1, Brussels, null). The above exception was the direct cause of the following exception: ... I wonder wether there's a reason that this doesn't work. If setting a related object on a model instance automatically sets instance.relatedobjectid, you'd expect this behavior to continue working if the related object receives its primary key after the referencing instance was initialized. Of course, switching lines (1) and (2) makes it all work, but this behavior means that bulk creating with related objects (with postgres returning the primary keys, bless it's heart) becomes more complex than it should be, forcing the user to use mappings or even set the foreign keys themselves: for countrydata, citydata in data: country = Country(countrydata) (country) city = City(country=country, citydata) (city) Country.objects.bulkcreate(countries) for city in cities: city.countryid = City.objects.bulk_create(cities) Ideally, the main instance would, when saved, 'reach into' its related objects and set the foreign key field like the loop does. Is there a reason this can't work?

using=using, raw=raw,         )     def _prepare_related_fields_for_save(self, operation_name):        # Ensure that a model instance without a PK hasn't been assigned to        # a ForeignKey or OneToOneField on this model. If the field is        # nullable, allowing the save would result in silent data loss.        for field in self._meta.concrete_fields:            # If the related field isn't cached, then an instance hasn't been            # assigned and there's no need to worry about this check.            if field.is_relation and field.is_cached(self):                obj = getattr(self, field.name, None)                if not obj:                    continue                # A pk may have been assigned manually to a model instance not                # saved to the database (or auto-generated in a case like                # UUIDField), but we allow the save to proceed and rely on the                # database to raise an IntegrityError if applicable. If                # constraints aren't supported by the database, there's the                # unavoidable risk of data corruption.                if obj.pk is None:                    # Remove the object from a related instance cache.                    if not field.remote_field.multiple:                        field.remote_field.delete_cached_value(obj)                    raise ValueError(                        "%s() prohibited to prevent data loss due to unsaved "                        "related object '%s'." % (operation_name, field.name)                    )                elif getattr(self, field.attname) is None:                    # Use pk from related object if it has been saved after                    # an assignment.                    setattr(self, field.attname, obj.pk)                # If the relationship's pk/to_field was changed, clear the                # cached relationship.                if getattr(obj, field.target_field.attname) != getattr(self, field.attname):                    field.delete_cached_value(self)     def delete(self, using=None, keep_parents=False):         using = using or router.db_for_write(self.__class__, instance=self)         assert self.pk is not None, (diff --git a/django/db/models/query.py b/django/db/models/query.py-- a/django/db/models/query.py++ b/django/db/models/query.py


Question: Saving parent object after setting on child leads to unexpected data loss in bulkcreate(). Description Example: class Country(models.Model): name = models.CharField(maxlength=255) isotwoletter = models.CharField(maxlength=2) description = models.TextField() class City(models.Model): name = models.CharField(maxlength=255) country = models.ForeignKey(Country, ondelete=models.CASCADE) class BulkCreateTests(TestCase): def testfkbug(self): countrynl = Country(name='Netherlands', isotwoletter='NL') countrybe = Country(name='Belgium', isotwoletter='BE') city = City(country=countrybe, name='Brussels') # (1) () # (2) () This results in an integrity error: ERROR: testfkbug () Traceback (most recent call last): File "", line 85, in execute return (sql, params) psycopg2.IntegrityError: null value in column "countryid" violates not-null constraint DETAIL: Failing row contains (1, Brussels, null). The above exception was the direct cause of the following exception: ... I wonder wether there's a reason that this doesn't work. If setting a related object on a model instance automatically sets instance.relatedobjectid, you'd expect this behavior to continue working if the related object receives its primary key after the referencing instance was initialized. Of course, switching lines (1) and (2) makes it all work, but this behavior means that bulk creating with related objects (with postgres returning the primary keys, bless it's heart) becomes more complex than it should be, forcing the user to use mappings or even set the foreign keys themselves: for countrydata, citydata in data: country = Country(countrydata) (country) city = City(country=country, citydata) (city) Country.objects.bulkcreate(countries) for city in cities: city.countryid = City.objects.bulk_create(cities) Ideally, the main instance would, when saved, 'reach into' its related objects and set the foreign key field like the loop does. Is there a reason this can't work?

obj.save(force_insert=True, using=self.db)         return obj     def _populate_pk_values(self, objs):    def _prepare_for_bulk_create(self, objs):         for obj in objs:             if obj.pk is None:                # Populate new PK values.                 obj.pk = obj._meta.pk.get_pk_value_on_save(obj)            obj._prepare_related_fields_for_save(operation_name='bulk_create')      def bulk_create(self, objs, batch_size=None, ignore_conflicts=False):         """


Question: Saving parent object after setting on child leads to unexpected data loss in bulkcreate(). Description Example: class Country(models.Model): name = models.CharField(maxlength=255) isotwoletter = models.CharField(maxlength=2) description = models.TextField() class City(models.Model): name = models.CharField(maxlength=255) country = models.ForeignKey(Country, ondelete=models.CASCADE) class BulkCreateTests(TestCase): def testfkbug(self): countrynl = Country(name='Netherlands', isotwoletter='NL') countrybe = Country(name='Belgium', isotwoletter='BE') city = City(country=countrybe, name='Brussels') # (1) () # (2) () This results in an integrity error: ERROR: testfkbug () Traceback (most recent call last): File "", line 85, in execute return (sql, params) psycopg2.IntegrityError: null value in column "countryid" violates not-null constraint DETAIL: Failing row contains (1, Brussels, null). The above exception was the direct cause of the following exception: ... I wonder wether there's a reason that this doesn't work. If setting a related object on a model instance automatically sets instance.relatedobjectid, you'd expect this behavior to continue working if the related object receives its primary key after the referencing instance was initialized. Of course, switching lines (1) and (2) makes it all work, but this behavior means that bulk creating with related objects (with postgres returning the primary keys, bless it's heart) becomes more complex than it should be, forcing the user to use mappings or even set the foreign keys themselves: for countrydata, citydata in data: country = Country(countrydata) (country) city = City(country=country, citydata) (city) Country.objects.bulkcreate(countries) for city in cities: city.countryid = City.objects.bulk_create(cities) Ideally, the main instance would, when saved, 'reach into' its related objects and set the foreign key field like the loop does. Is there a reason this can't work?

opts = self.model._meta         fields = opts.concrete_fields         objs = list(objs)        self._populate_pk_values(objs)        self._prepare_for_bulk_create(objs)         with transaction.atomic(using=self.db, savepoint=False):             objs_with_pk, objs_without_pk = partition(lambda o: o.pk is None, objs)             if objs_with_pk:


Question: Upgrading 2.23.0 causes named tuples used as arguments to range to error. Description I noticed this while upgrading a project from 2.2 to 3.0. This project passes named 2-tuples as arguments to range queryset filters. This works fine on 2.2. On 3.0 it causes the following error: TypeError: new_() missing 1 required positional argument: 'far'. This happens because goes into the tuple elements to resolve lookups and then attempts to reconstitute the tuple with the resolved elements. When it attempts to construct the new tuple it preserves the type (the named tuple) but it passes a iterator to it's constructor. NamedTuples don't have the code path for copying an iterator, and so it errors on insufficient arguments. The fix is to * expand the contents of the iterator into the constructor.

elif isinstance(value, (list, tuple)):             # The items of the iterable may be expressions and therefore need             # to be resolved independently.            return type(value)(            values = (                 self.resolve_lookup_value(sub_value, can_reuse, allow_joins)                 for sub_value in value             )            type_ = type(value)            if hasattr(type_, '_make'):  # namedtuple                return type_(*values)            return type_(values)         return value      def solve_lookup_type(self, lookup):


Question: ManyToManyField does not respect the PositiveBigIntegerField in m2m intermediate table. Description When creating a relation between 2 models using PositiveBigIntegerField on Postgresql, the relation table is created using regular ints as the column type. This in turn leads to out of bound error when using large numbers for ids. from import models class Node(models.Model): id = models.PositiveBigIntegerField(primarykey=True) point = models.PointField() class Relation(models.Model): id = models.PositiveBigIntegerField(primarykey=True) nodes = models.ManyToManyField(Node) The generated table will look like this: Column Description -------------+---------+-----------+----------+------------------------------------------------+---------+--------------+------------- id relationid nodeid As you can see, the PositiveBigInteger is not respected and a regular int is set

})  <ins>class SmallIntegerField(IntegerField):    description = _('Small integer')    def get_internal_type(self):        return 'SmallIntegerField'</ins> class IPAddressField(Field):     empty_strings_allowed = False     description = _("IPv4 address")


Question: ManyToManyField does not respect the PositiveBigIntegerField in m2m intermediate table. Description When creating a relation between 2 models using PositiveBigIntegerField on Postgresql, the relation table is created using regular ints as the column type. This in turn leads to out of bound error when using large numbers for ids. from import models class Node(models.Model): id = models.PositiveBigIntegerField(primarykey=True) point = models.PointField() class Relation(models.Model): id = models.PositiveBigIntegerField(primarykey=True) nodes = models.ManyToManyField(Node) The generated table will look like this: Column Description -------------+---------+-----------+----------+------------------------------------------------+---------+--------------+------------- id relationid nodeid As you can see, the PositiveBigInteger is not respected and a regular int is set

class PositiveIntegerRelDbTypeMixin:<ins>    def __init_subclass__(cls, **kwargs):        super().__init_subclass__(**kwargs)        if not hasattr(cls, 'integer_field_class'):            cls.integer_field_class = next(                (                    parent                    for parent in cls.__mro__[1:]                    if issubclass(parent, IntegerField)                ),                None,            )</ins>      def rel_db_type(self, connection):         """


Question: ManyToManyField does not respect the PositiveBigIntegerField in m2m intermediate table. Description When creating a relation between 2 models using PositiveBigIntegerField on Postgresql, the relation table is created using regular ints as the column type. This in turn leads to out of bound error when using large numbers for ids. from import models class Node(models.Model): id = models.PositiveBigIntegerField(primarykey=True) point = models.PointField() class Relation(models.Model): id = models.PositiveBigIntegerField(primarykey=True) nodes = models.ManyToManyField(Node) The generated table will look like this: Column Description -------------+---------+-----------+----------+------------------------------------------------+---------+--------------+------------- id relationid nodeid As you can see, the PositiveBigInteger is not respected and a regular int is set

if connection.features.related_fields_match_type:             return self.db_type(connection)         else:            return IntegerField().db_type(connection=connection)            return self.integer_field_class().db_type(connection=connection)  class PositiveBigIntegerField(PositiveIntegerRelDbTypeMixin, IntegerField):class PositiveBigIntegerField(PositiveIntegerRelDbTypeMixin, BigIntegerField):     description = _('Positive big integer')      def get_internal_type(self):