Question: else:             self.cookies[key]['expires'] = ''         if max_age is not None:            self.cookies[key]['max-age'] = max_age            self.cookies[key]['max-age'] = int(max_age)             # IE requires expires, so set it if hasn't been already.             if not expires:                 self.cookies[key]['expires'] = http_date(time.time() + max_age)

Convert maxage to an int in setcookie() Description The max-age attribute of cookie is supposed to be an integer  I think it would be helpful to convert the maxage parameter of setcookie() to an integer for the user. The benefit is simply that there are some cookie parsers that don't handle decimals gracefully. It's pretty easy to pass in a float without understanding the consequences. I spent a good chunk of time today trying to track down the problem. Things to consider: Do we only convert floats where the decimal part is 0? Or do we round or truncate? If we can't successfully convert to an int, do we throw an exception, or just pass in the original value?


Question: def __eq__(self, other):         # Needed for @total_ordering         if isinstance(other, Field):            return self.creation_counter == other.creation_counter            return (                self.creation_counter == other.creation_counter and                getattr(self, 'model', None) == getattr(other, 'model', None)            )         return NotImplemented      def __lt__(self, other):         # This is needed because bisect does not take a comparison function.        # Order by creation_counter first for backward compatibility.         if isinstance(other, Field):            return self.creation_counter < other.creation_counter            if (                self.creation_counter != other.creation_counter or                not hasattr(self, 'model') and not hasattr(other, 'model')            ):                return self.creation_counter < other.creation_counter            elif hasattr(self, 'model') != hasattr(other, 'model'):                return not hasattr(self, 'model')  # Order no-model fields first            else:                # creation_counter's are equal, compare only models.                return (                    (self.model._meta.app_label, self.model._meta.model_name) <                    (other.model._meta.app_label, other.model._meta.model_name)                )         return NotImplemented      def __hash__(self):        return hash(self.creation_counter)        return hash((            self.creation_counter,            self.model._meta.app_label if hasattr(self, 'model') else None,            self.model._meta.model_name if hasattr(self, 'model') else None,        ))      def __deepcopy__(self, memodict):         # We don't have to deepcopy very much here, since most things are not

Abstract model field should not be equal across models Description Consider the following models: class A(models.Model): class Meta: abstract = True myfield = IntegerField() class B(A): pass class C(A): pass If I pull the fields of B and C into a shared set, one will be de-duplicated away, because they compare as equal. I found this surprising, though in practice using a list was sufficient for my need. The root of the issue is that they compare equal, as fields only consider self.creationcounter when comparing for equality. len({B.meta.getfield('myfield'), C.meta.getfield('myfield')}) == 1 B.meta.getfield('myfield') == C.meta.getfield('myfield') We should adjust eq so that if the is different, they will compare unequal. Similarly, it is probably wise to adjust hash and lt to match. When adjusting lt, it may be wise to order first by self.creationcounter so that cases not affected by this equality collision won't be re-ordered. In my experimental branch, there was one test that broke if I ordered them by model first. I brought this up on IRC django-dev to check my intuitions, and those conversing with me there seemed to agree that the current behavior is not intuitive.


Question: @query.setter     def query(self, value):<ins>        if value.values_select:            self._iterable_class = ValuesIterable</ins>         self._query = value      def as_manager(cls):

Queryset with values()/valueslist() crashes when recreated from a pickled query. Description I am pickling query objects () for later re-evaluation as per  However, when I tried to rerun a query that combines values and annotate for a GROUP BY functionality, the result is broken. Normally, the result of the query is and should be a list of dicts, but in this case instances of the model are returned, but their internal state is broken and it is impossible to even access their .id because of a AttributeError: 'NoneType' object has no attribute 'attname' error. I created a minimum reproducible example. from import models class Toy(models.Model): name = models.CharField(maxlength=16) material = models.CharField(maxlength=16) price = models.PositiveIntegerField() crashing code import pickle from import Sum from import Toy (name='foo', price=10, material='wood') (name='bar', price=20, material='plastic') (name='baz', price=100, material='wood') prices = ('material').annotate(totalprice=Sum('price')) print(prices) print(type(prices[0])) prices2 = () = (()) print(type(prices2[0])) print(prices2) The type of prices[0] is reported as 'dict', which is ok, the type of prices2[0] is reported as '<class "models.Toy"', which is wrong.


Question: @query.setter     def query(self, value):<ins>        if value.values_select:            self._iterable_class = ValuesIterable</ins>         self._query = value      def as_manager(cls):

The code then crashes when trying to print the evaluated queryset with the following: Traceback (most recent call last): File "", line 19, in <moduleprint(prices2) File "/home/beda/virtualenvs/celus/lib/python3.6/site-", line 253, in repr return '<%s %r' % (, data) File "/home/beda/virtualenvs/celus/lib/python3.6/site-", line 519, in repr return '<%s: %s' % (, self) File "/home/beda/virtualenvs/celus/lib/python3.6/site-", line 522, in str return '%s object (%s)' % (, ) File "/home/beda/virtualenvs/celus/lib/python3.6/site-", line 569, in getpkval return getattr(self, ) File "/home/beda/virtualenvs/celus/lib/python3.6/site-", line 133, in get val = self.checkparentchain(instance, self.fieldname) File "/home/beda/virtualenvs/celus/lib/python3.6/site-", line 150, in checkparentchain return getattr(instance, link_field.attname) AttributeError: 'NoneType' object has no attribute 'attname' From my point of view it seems as though Django retrieves the correct data from the database, but instead of returning them as a dict, it tries to create model instances from them, which does not work as the data has wrong structure.


Question: return True     else:         def lock(f, flags):            ret = fcntl.flock(_fd(f), flags)            return ret == 0            try:                fcntl.flock(_fd(f), flags)                return True            except BlockingIOError:                return False          def unlock(f):            ret = fcntl.flock(_fd(f), fcntl.LOCK_UN)            return ret == 0            fcntl.flock(_fd(f), fcntl.LOCK_UN)            return True

Bug in posix implementation of Description The posix version of locks (the version which supports import fcntl) has a bug. The code attempts to return True to indicate success or failure acquiring a lock, but instead it always returns False. The reason is that cpython fcntl module returns None if successful, and raises an OSError to indicate failure (see ). Anyone interested in using the non-blocking (i.e. locks.LOCKSNB) requires a valid return value to know if they have successfully acquired the lock. I believe the correct implementation should be the following: diff --git index .. --- +++ -107,9 +107,15 else: return True else: def lock(f, flags): ret = (fd(f), flags) return ret == 0 try: (fd(f), flags) return True except OSError: return False def unlock(f): ret = (fd(f), fcntl.LOCKUN) return ret == 0 try: (fd(f), fcntl.LOCK_UN) return True except OSError: return False


Question: return [self.lookup_kwarg, self.lookup_kwarg2]      def choices(self, changelist):        field_choices = dict(self.field.flatchoices)         for lookup, title in (                (None, _('All')),                ('1', _('Yes')),                ('0', _('No'))):            (None, _('All')),            ('1', field_choices.get(True, _('Yes'))),            ('0', field_choices.get(False, _('No'))),        ):             yield {                 'selected': self.lookup_val == lookup and not self.lookup_val2,                 'query_string': changelist.get_query_string({self.lookup_kwarg: lookup}, [self.lookup_kwarg2]),

BooleanFieldListFilter doesn't respect field choices. Description If I have such construction: class BoolTest(models.Model): BOUGHTCHOICES = ( (False, 'Pending'), (True, 'Bought') ) bought = models.BooleanField( verbosename="Fancy Boolean", default=False, choices=BOUGHTCHOICES) class BoolTestAdmin(admin.ModelAdmin): listfilter = ('bought',) (BoolTest, BoolTestAdmin) The boolean Filter text is not modified to fit choices param Example (in FR): FILTRE Par Fancy Boolean Tout Oui Non Should be : FILTRE Par Fancy Boolean Tout Bought Pending


Question: yield {                 'selected': self.lookup_val2 == 'True',                 'query_string': changelist.get_query_string({self.lookup_kwarg2: 'True'}, [self.lookup_kwarg]),                'display': _('Unknown'),                'display': field_choices.get(None, _('Unknown')),             }


Question: return True         if self.query.extra_order_by or self.query.order_by:             return True        elif self.query.default_ordering and self.query.get_meta().ordering:        elif (            self.query.default_ordering and            self.query.get_meta().ordering and            # A default ordering doesn't affect GROUP BY queries.            not self.query.group_by        ):             return True         else:             return False

QuerySet.ordered property is incorrect for GROUP BY queries on models with Meta.ordering. Description Using the annotate function on a queryset doesn't keep the default ordering set in model's meta class. A property should say whether the queryset will be ordered or not. I wanted to use the qs.ordered property for this but it seems to stay truthy, even if the resulting SQL query will not have an ORDER BY clause. Example: qs = ()   qs.ordered # =True # =True  qs2 = Foo.objects.annotate(Count("pk")).all()   qs2.ordered # =True # =True If it can help : I'm using PostgreSQL


Question: for expr, is_ref in order_by:             resolved = expr.resolve_expression(self.query, allow_joins=True, reuse=None)            if self.query.combinator:            if self.query.combinator and self.select:                 src = resolved.get_source_expressions()[0]                 expr_src = expr.get_source_expressions()[0]                 # Relabel order by columns to raw numbers if this is a combined

Combined queryset crash on combined querysets with ordering. Description I have the following models. from import models class M1(models.Model): id = models.AutoField(primarykey=True,blank=True, null=True) f1 = models.TextField(blank=True, null=True) class Meta: managed = False dbtable = 'm1' class M2(models.Model): id = models.AutoField(primarykey=True,blank=True, null=True) f2 = models.TextField(blank=True, null=True) class Meta: managed = False dbtable = 'm2' class M3(models.Model): id = models.AutoField(primarykey=True,blank=True, null=True) f3 = models.TextField(blank=True, null=True) class Meta: managed = False dbtable = 'm3' Based on these models, I perform the following query. o1 = ('default') o2 = ('default') u1 = (o2) q = u1.orderby('-f2') o3 = ('default') res = q.union(o3) print(()) Unfortunately, this query crashes with a TypeError exception. The track trace is Traceback (most recent call last): File "", line 19, in <moduleprint(()) File "", line 411, in count return () File "", line 517, in getcount number = obj.getaggregation(using, ['count'])['count'] File "", line 485, in getaggregation outerquery.addsubquery(innerquery, using) File "", line 162, in addsubquery self.subquery, self.subparams = query.getcompiler(using).assql(withcolaliases=True) File "", line 507, in assql result, params = self.getcombinatorsql(combinator, ) File "", line 443, in getcombinatorsql if compiler.getorderby(): File "", line 368, in getorderby for idx, (selexpr, , col_alias) in enumerate(): TypeError: 'NoneType' object is not iterable


Question: from django.db.models import AutoField, DateField, DateTimeField, sql from django.db.models.constants import LOOKUP_SEP from django.db.models.deletion import Collectorfrom django.db.models.expressions import Case, Expression, F, Value, Whenfrom django.db.models.expressions import Case, Expression, F, Ref, Value, When from django.db.models.functions import Cast, Trunc from django.db.models.query_utils import FilteredRelation, Q from django.db.models.sql.constants import CURSOR, GET_ITERATOR_CHUNK_SIZE

QuerySet.aggregate() mixes annotated fields names. Description I have the following model (simplified for opening the issue). class T(models.Model): id = models.AutoField(primary=True) foo = models.IntegerField() and I perform the following query on MySQL (simplified for opening the issue) T.objects.annotate(anon=F('foo')).aggregate(foo=Max(F('anon')), sum=Sum(F('foo'))) this produces the following SQL query SELECT MAX(), SUM() FROM (SELECT . AS FROM ) subquery that causes the following exception (1054, "Unknown column 'foo' in 'field list'") I would expect django to generate SELECT MAX(), SUM() FROM (SELECT . AS FROM ) subquery


Question: query = self.query.chain()         for (alias, aggregate_expr) in kwargs.items():             query.add_annotation(aggregate_expr, alias, is_summary=True)            if not query.annotations[alias].contains_aggregate:            annotation = query.annotations[alias]            if not annotation.contains_aggregate:                 raise TypeError("%s is not an aggregate expression" % alias)            for expr in annotation.get_source_expressions():                if expr.contains_aggregate and isinstance(expr, Ref) and expr.refs in kwargs:                    name = expr.refs                    raise exceptions.FieldError(                        "Cannot compute %s('%s'): '%s' is an aggregate"                        % (annotation.name, name, name)                    )         return query.get_aggregation(self.db, kwargs)      def count(self):


Question: info = (app_label, model._meta.model_name)             model_dict = {<ins>                'model': model,</ins>                 'name': capfirst(model._meta.verbose_name_plural),                 'object_name': model._meta.object_name,                 'perms': perms,

Added model class to applist context Description (last modified by Raffaele Salmaso) I need to manipulate the applist in my custom admin view, and the easiest way to get the result is to have access to the model class (currently the dictionary is a serialized model). In addition I would make the buildappdict method public, as it is used by the two views index and appindex.


Question: settings.DATABASES[self.connection.alias]["NAME"] = test_database_name         self.connection.settings_dict["NAME"] = test_database_name         if self.connection.settings_dict['TEST']['MIGRATE']:        try:            if self.connection.settings_dict['TEST']['MIGRATE'] is False:                # Disable migrations for all apps.                old_migration_modules = settings.MIGRATION_MODULES                settings.MIGRATION_MODULES = {                    app.label: None                    for app in apps.get_app_configs()                }             # We report migrate messages at one level lower than that             # requested. This ensures we don't get flooded with messages during             # testing (unless you really ask to be flooded).

Test runner setupdatabases crashes with "TEST": {"MIGRATE": False}. Description I'm trying to upgrade a project from Django 3.0 to Django 3.1 and wanted to try out the new "TEST": {"MIGRATE": False} database setting. Sadly I'm running into an issue immediately when running test. Removing the "TEST": {"MIGRATE": False} line allows the tests to run. So this is not blocking the upgrade for us, but it would be nice if we were able to use the new feature to skip migrations during testing. For reference, this project was recently upgraded from Django 1.4 all the way to 3.0 so there might be some legacy cruft somewhere that triggers this. Here's the trackeback. I'll try to debug this some more. Traceback (most recent call last): File "/usr/local/lib/python3.6/site-", line 84, in execute return (sql, params) relation "djangoadminlog" does not exist LINE 1: ", "djangoadminlog"."changemessage" FROM "djangoad... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.6/site-", line 1156, in executesql cursor.execute(sql, params) File "/usr/local/lib/python3.6/site-", line 66, in execute return self.executewithwrappers(sql, params, many=False, executor=self.execute) File "/usr/local/lib/python3.6/site-", line 75, in executewithwrappers return executor(sql, params, many, context) File "/usr/local/lib/python3.6/site-", line 84, in execute return (sql, params) File "/usr/local/lib/python3.6/site-", line 90, in exit raise djexcvalue.withtraceback(traceback) from excvalue File "/usr/local/lib/python3.6/site-", line 84, in execute return (sql, params) relation "djangoadminlog" does not exist LINE 1: ", "djangoadminlog"."changemessage" FROM "djangoad. ^ During handling of the above exception, another exception occurred: Traceback (most recent call last): File "", line 15, in <modulemain() File "", line 11,


Question: settings.DATABASES[self.connection.alias]["NAME"] = test_database_name         self.connection.settings_dict["NAME"] = test_database_name         if self.connection.settings_dict['TEST']['MIGRATE']:        try:            if self.connection.settings_dict['TEST']['MIGRATE'] is False:                # Disable migrations for all apps.                old_migration_modules = settings.MIGRATION_MODULES                settings.MIGRATION_MODULES = {                    app.label: None                    for app in apps.get_app_configs()                }             # We report migrate messages at one level lower than that             # requested. This ensures we don't get flooded with messages during             # testing (unless you really ask to be flooded).

in main executefromcommandline() File "/usr/local/lib/python3.6/site-", line 401, in executefromcommandline utility.execute() File "/usr/local/lib/python3.6/site-", line 395, in execute self.fetchcommand(subcommand).runfromargv() File "/usr/local/lib/python3.6/site-", line 23, in runfromargv super().runfromargv(argv) File "/usr/local/lib/python3.6/site-", line 330, in runfromargv self.execute(args, cmdoptions) File "/usr/local/lib/python3.6/site-", line 371, in execute output = (args, options) File "/usr/local/lib/python3.6/site-", line 53, in handle failures = testrunner.runtests(testlabels) File "/usr/local/lib/python3.6/site-", line 695, in runtests oldconfig = self.setupdatabases(aliases=databases) File "/usr/local/lib/python3.6/site-", line 616, in setupdatabases self.parallel, kwargs File "/usr/local/lib/python3.6/site-", line 174, in setupdatabases serialize=connection.settingsdict['TEST'].get('SERIALIZE', True), File "/usr/local/lib/python3.6/site-", line 78, in createtestdb self.connection.testserializedcontents = self.serializedbtostring() File "/usr/local/lib/python3.6/site-", line 121, in serializedbtostring serializers.serialize("json", getobjects(), indent=None, stream=out) File "/usr/local/lib/python3.6/site-", line 128, in serialize s.serialize(queryset, options) File "/usr/local/lib/python3.6/site-", line 90, in serialize for count, obj in enumerate(queryset, start=1): File "/usr/local/lib/python3.6/site-", line 118, in getobjects yield from queryset.iterator() File "/usr/local/lib/python3.6/site-", line 360, in iterator yield from self.iterableclass(self, chunkedfetch=usechunkedfetch, chunksize=chunksize) File "/usr/local/lib/python3.6/site-", line 53, in iter results = compiler.executesql(chunkedfetch=self.chunkedfetch, chunksize=self.chunksize) File "/usr/local/lib/python3.6/site-", line 1159,


Question: settings.DATABASES[self.connection.alias]["NAME"] = test_database_name         self.connection.settings_dict["NAME"] = test_database_name         if self.connection.settings_dict['TEST']['MIGRATE']:        try:            if self.connection.settings_dict['TEST']['MIGRATE'] is False:                # Disable migrations for all apps.                old_migration_modules = settings.MIGRATION_MODULES                settings.MIGRATION_MODULES = {                    app.label: None                    for app in apps.get_app_configs()                }             # We report migrate messages at one level lower than that             # requested. This ensures we don't get flooded with messages during             # testing (unless you really ask to be flooded).

in executesql () cursor "djangocurs139860821038912sync_1" does not exist


Question: database=self.connection.alias,                 run_syncdb=True,             )<ins>        finally:            if self.connection.settings_dict['TEST']['MIGRATE'] is False:                settings.MIGRATION_MODULES = old_migration_modules</ins>          # We then serialize the current state of the database into a string         # and store it on the connection. This slightly horrific process is so people


Question: database=self.connection.alias,                 run_syncdb=True,             )<ins>        finally:            if self.connection.settings_dict['TEST']['MIGRATE'] is False:                settings.MIGRATION_MODULES = old_migration_modules</ins>          # We then serialize the current state of the database into a string         # and store it on the connection. This slightly horrific process is so people


Question: database=self.connection.alias,                 run_syncdb=True,             )<ins>        finally:            if self.connection.settings_dict['TEST']['MIGRATE'] is False:                settings.MIGRATION_MODULES = old_migration_modules</ins>          # We then serialize the current state of the database into a string         # and store it on the connection. This slightly horrific process is so people


Question: self.descending = True  class Window(Expression):class Window(SQLiteNumericMixin, Expression):     template = '%(expression)s OVER (%(window)s)'     # Although the main expression may either be an aggregate or an     # expression with an aggregate function, the GROUP BY that will

Lag() with DecimalField crashes on SQLite. Description On Django 3.0.7 with a SQLite database using the following model: from import models class LagTest(models.Model): modified = models.DateField() data = models.FloatField() amount = models.DecimalField(decimalplaces=4, maxdigits=7) and the following query from import F from import Lag from import Window from import LagTest w = Window(expression=Lag('amount',7), partitionby=[F('modified')], orderby=F('modified').asc()) q = ().annotate(w=w) generates the following error: In [12]: print(q) OperationalError Traceback (most recent call last) C:\ProgramData\Anaconda3\envs\djbase\lib\site-packages\django\db\backends\ in execute(self, sql, params, ignoredwrapperargs) 85 else: ---86 return (sql, params) 87 C:\ProgramData\Anaconda3\envs\djbase\lib\site-packages\django\db\backends\sqlite3\ in execute(self, query, params) 395 query = self.convertquery(query) --396 return Database.Cursor.execute(self, query, params) 397 OperationalError: near "OVER": syntax error The above exception was the direct cause of the following exception: OperationalError Traceback (most recent call last) <ipython-input-12-996617e96a38in <module----1 print(q) C:\ProgramData\Anaconda3\envs\djbase\lib\site-packages\django\db\models\ in repr(self) 250 251 def repr(self): --252 data = list(self[:REPROUTPUTSIZE + 1]) 253 if len(data) REPROUTPUTSIZE: 254 data[-1] = "...(remaining elements truncated)..." C:\ProgramData\Anaconda3\envs\djbase\lib\site-packages\django\db\models\ in iter(self) 274 - Responsible for turning the rows into model objects.


Question: self.descending = True  class Window(Expression):class Window(SQLiteNumericMixin, Expression):     template = '%(expression)s OVER (%(window)s)'     # Although the main expression may either be an aggregate or an     # expression with an aggregate function, the GROUP BY that will

275 """ --276 self.fetchall() 277 return iter(self.resultcache) 278 C:\ProgramData\Anaconda3\envs\djbase\lib\site-packages\django\db\models\ in fetchall(self) 1259 def fetchall(self): 1260 if self.resultcache is None: -1261 self.resultcache = list(self.iterableclass(self)) 1262 if self.prefetchrelatedlookups and not self.prefetchdone: 1263 self.prefetchrelatedobjects() C:\ProgramData\Anaconda3\envs\djbase\lib\site-packages\django\db\models\ in iter(self) 55 # Execute the query. This will also fill , klassinfo, 56 # and annotations. ---57 results = compiler.executesql(chunkedfetch=self.chunkedfetch, chunksize=self.chunksize) 58 select, klassinfo, annotationcolmap = (, compiler.klassinfo, 59 compiler.annotationcolmap) C:\ProgramData\Anaconda3\envs\djbase\lib\site-packages\django\db\models\sql\ in executesql(self, resulttype, chunkedfetch, chunksize) 1150 cursor = () 1151 try: -1152 cursor.execute(sql, params) 1153 except Exception: 1154 # Might fail for server-side cursors (e.g. connection closed) C:\ProgramData\Anaconda3\envs\djbase\lib\site-packages\django\db\backends\ in execute(self, sql, params) 98 def execute(self, sql, params=None): 99 with self.debugsql(sql, params, uselastexecutedquery=True): --100 return super().execute(sql, params) 101 102 def executemany(self, sql, paramlist): C:\ProgramData\Anaconda3\envs\djbase\lib\site-packages\django\db\backends\ in execute(self, sql, params) 66 67 def execute(self, sql, params=None): ---68 return self.executewithwrappers(sql, params, many=False, executor=self.execute) 69 70 def executemany(self, sql, paramlist): C:\ProgramData\Anaconda3\envs\djbase\lib\site-packages\django\db\backends\ in executewithwrappers(self, sql, params, many,


Question: self.descending = True  class Window(Expression):class Window(SQLiteNumericMixin, Expression):     template = '%(expression)s OVER (%(window)s)'     # Although the main expression may either be an aggregate or an     # expression with an aggregate function, the GROUP BY that will

executor) 75 for wrapper in reversed(): 76 executor = functools.partial(wrapper, executor) ---77 return executor(sql, params, many, context) 78 79 def execute(self, sql, params, ignoredwrapperargs): C:\ProgramData\Anaconda3\envs\djbase\lib\site-packages\django\db\backends\ in execute(self, sql, params, ignoredwrapperargs) 84 return (sql) 85 else: ---86 return (sql, params) 87 88 def executemany(self, sql, paramlist, ignoredwrapperargs): C:\ProgramData\Anaconda3\envs\djbase\lib\site-packages\django\db\ in exit(self, exctype, excvalue, traceback) 88 if djexctype not in (DataError, IntegrityError): 89 self.wrapper.errorsoccurred = True ---90 raise djexcvalue.withtraceback(traceback) from excvalue 91 92 def call(self, func): C:\ProgramData\Anaconda3\envs\djbase\lib\site-packages\django\db\backends\ in execute(self, sql, params, ignoredwrapperargs) 84 return (sql) 85 else: ---86 return (sql, params) 87 88 def executemany(self, sql, paramlist, ignoredwrapperargs): C:\ProgramData\Anaconda3\envs\djbase\lib\site-packages\django\db\backends\sqlite3\ in execute(self, query, params) 394 return Database.Cursor.execute(self, query) 395 query = self.convertquery(query) --396 return Database.Cursor.execute(self, query, params) 397 398 def executemany(self, query, paramlist): OperationalError: near "OVER": syntax error The generated SQL query is: SELECT "test1lagtest"."id", "test1lagtest"."modified", "test1lagtest"."data", "test1lagtest"."amount", CAST(LAG("test1lagtest"."amount", 7) AS NUMERIC) OVER (PARTITION BY "test1lagtest"."modified" ORDER BY "test1lagtest".


Question: self.descending = True  class Window(Expression):class Window(SQLiteNumericMixin, Expression):     template = '%(expression)s OVER (%(window)s)'     # Although the main expression may either be an aggregate or an     # expression with an aggregate function, the GROUP BY that will

"modified" ASC) AS "w" FROM "test1lagtest" I believe this fails as the CAST() statement ends after LAG whereas it should be around the whole statement up until "w" This only applies where the lagged field is a DecimalField e.g. w = Window(expression=Lag('data',7), partitionby=[F('modified')], orderby=F('modified').asc()) works correctly. I can override it by adding outputfield=FloatField() to the Lag function e.g. w = Window(expression=Lag('amount',7,outputfield=FloatField()), partitionby=[F('modified')], orderby=F('modified').asc())


Question: 'window': ''.join(window_sql).strip()         }, params <ins>    def as_sqlite(self, compiler, connection):        if isinstance(self.output_field, fields.DecimalField):            # Casting to numeric must be outside of the window expression.            copy = self.copy()            source_expressions = copy.get_source_expressions()            source_expressions[0].output_field = fields.FloatField()            copy.set_source_expressions(source_expressions)            return super(Window, copy).as_sqlite(compiler, connection)        return self.as_sql(compiler, connection)</ins>     def __str__(self):         return '{} OVER ({}{}{})'.format(             str(self.source_expression),


Question: 'window': ''.join(window_sql).strip()         }, params <ins>    def as_sqlite(self, compiler, connection):        if isinstance(self.output_field, fields.DecimalField):            # Casting to numeric must be outside of the window expression.            copy = self.copy()            source_expressions = copy.get_source_expressions()            source_expressions[0].output_field = fields.FloatField()            copy.set_source_expressions(source_expressions)            return super(Window, copy).as_sqlite(compiler, connection)        return self.as_sql(compiler, connection)</ins>     def __str__(self):         return '{} OVER ({}{}{})'.format(             str(self.source_expression),


Question: 'window': ''.join(window_sql).strip()         }, params <ins>    def as_sqlite(self, compiler, connection):        if isinstance(self.output_field, fields.DecimalField):            # Casting to numeric must be outside of the window expression.            copy = self.copy()            source_expressions = copy.get_source_expressions()            source_expressions[0].output_field = fields.FloatField()            copy.set_source_expressions(source_expressions)            return super(Window, copy).as_sqlite(compiler, connection)        return self.as_sql(compiler, connection)</ins>     def __str__(self):         return '{} OVER ({}{}{})'.format(             str(self.source_expression),


Question: 'window': ''.join(window_sql).strip()         }, params <ins>    def as_sqlite(self, compiler, connection):        if isinstance(self.output_field, fields.DecimalField):            # Casting to numeric must be outside of the window expression.            copy = self.copy()            source_expressions = copy.get_source_expressions()            source_expressions[0].output_field = fields.FloatField()            copy.set_source_expressions(source_expressions)            return super(Window, copy).as_sqlite(compiler, connection)        return self.as_sql(compiler, connection)</ins>     def __str__(self):         return '{} OVER ({}{}{})'.format(             str(self.source_expression),


Question: # Field flags     auto_created = False<ins>    empty_strings_allowed = False</ins>      many_to_many = False     many_to_one = False

Using EmptyFieldListFilter with GenericForeignKey and GenericRelation crashes. Description I have a GenericForeignKey in my model: contenttyperesourcecontract = models.ForeignKey( ContentType, ondelete=models.CASCADE, blank=True, null=True, ) resourcecontractid = models.PositiveIntegerField(blank=True, null=True) resourcecontract = GenericForeignKey('contenttyperesource', 'resourcecontractid') and I want to use the new admin.EmptyFieldListFilter in my model admin: class myAdmin(admin.ModelAdmin): listfilter = (('resourcecontract', admin.EmptyFieldListFilter),) But when I try to run it I get a "'GenericForeignKey' object has no attribute 'emptystringsallowed'". It will work fine if I use the resourcecontractid field. Would it make sense to extend the GenericForeignKey to use those emptystrings_allowed attributes from fields that are used in this generic relation?


Question: limit_choices_to=limit_choices_to,         ) <ins>        # Reverse relations are always nullable (Django can't enforce that a        # foreign key on the related model points to this model).        kwargs['null'] = True</ins>         kwargs['blank'] = True         kwargs['on_delete'] = models.CASCADE         kwargs['editable'] = False


Question: self.total_form_count() - len(self.deleted_forms) > self.max_num) or \                     self.management_form.cleaned_data[TOTAL_FORM_COUNT] > self.absolute_max:                 raise ValidationError(ngettext(                    "Please submit %d or fewer forms.",                    "Please submit %d or fewer forms.", self.max_num) % self.max_num,                    "Please submit at most %d form.",                    "Please submit at most %d forms.", self.max_num) % self.max_num,                     code='too_many_forms',                 )             if (self.validate_min and                     self.total_form_count() - len(self.deleted_forms) - empty_forms_count < self.min_num):                 raise ValidationError(ngettext(                    "Please submit %d or more forms.",                    "Please submit %d or more forms.", self.min_num) % self.min_num,                    "Please submit at least %d form.",                    "Please submit at least %d forms.", self.min_num) % self.min_num,                     code='too_few_forms')             # Give self.clean() a chance to do cross-form validation.             self.clean()

Incorrect messaging when validatemin/validatemax and minnum == maxnum. Description When using Django Formset, the error messaging isn't checking for the condition where both minnum and maxnum are equal and both validatemin and validatemax are set to true. Code highlighting: class TestForm(forms.Form): msg = forms.CharField() testformset = formsetfactory( TestForm, minnum=2, maxnum=2, validatemin=True, validatemax=True) When formset is created in the following way and both validate flags are set True the following error messages show up If the supplied forms are less than two - please submit 2 or more forms expected please submit 2 forms similarly the reverse is also true when the forms are more than two it gives the error message saying please submit 2 or fewer forms expected please submit 2 forms This was a  bug reported on Wagtail and after investigating a little I noticed the incorrect messaging was coming from  this part in the validation


Question: <ins>from decimal import Decimal</ins> from django.conf import settings from django.template import Library, Node, TemplateSyntaxError, Variable from django.template.base import TokenType, render_value_in_context

String formatting error when passing floats as values in {% blocktrans %} tags Description With the following template code: {% blocktrans count counter=person.distanceinmiles|floatformat:0 %}{{ counter }} mile away{% plural %}{{ counter }} miles away{% endblocktrans %} And a russian translation: #, python-format msgid "%(counter)s mile away" msgidplural "%(counter)s miles away" msgstr[0] "На расстоянии %(counter)s мили" msgstr[1] "На расстоянии %(counter)s миль" msgstr[2] "На расстоянии %(counter)s миль" Rendering the template fails with a String formatting error: "TypeError: not all arguments converted during string formatting". This happens because gettext string formatting fails when a float is passed. Removing the floatformat and casting the value as int works fine. This could be improved by either: Swallow TypeError and throw a friendlier message saying the type might not be compatible with gettext's string formatting (not a fan of swallowing TypeError though, we need to make sure nothing else in dotranslate throws such an exception) Actively checking the type's compatibility with the string format


Question: singular, vars = self.render_token_list(self.singular)         if self.plural and self.countervar and self.counter:             count = self.counter.resolve(context)<ins>            if not isinstance(count, (Decimal, float, int)):                raise TemplateSyntaxError(                    "%r argument to %r tag must be a number."                    % (self.countervar, self.tag_name)                )</ins>             context[self.countervar] = count             plural, plural_vars = self.render_token_list(self.plural)             if message_context:


Question: import os import pkgutil import sysfrom argparse import _SubParsersActionfrom argparse import (    _AppendConstAction, _CountAction, _StoreConstAction, _SubParsersAction,) from collections import defaultdict from difflib import get_close_matches from importlib import import_module

Required mutually exclusive groups don't work with boolean arguments. Description I have the following management command (called ): from import BaseCommand class Command(BaseCommand): def addarguments(self, parser): group = parser.addmutuallyexclusivegroup(required=True) group.addargument('--value', type=str) group.addargument('--flag', action='storetrue') def handle(self, args, *options): pass Running test --flag or --value foo, the command works as expected. Using callcommand to call the command from my code fails if I pass --flag, e.g.: from import callcommand callcommand('test', flag=True) This gets me the following traceback: argparse.ArgumentError: argument --flag: ignored explicit argument 'True' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "<input", line 1, in <moduleFile "", line 147, in callcommand defaults = parser.parseargs(args=parseargs) File "", line 57, in parseargs return super().parseargs(args, namespace) File "", line 1780, in parseargs args, argv = self.parseknownargs(args, namespace) File "", line 1819, in parseknownargs (str(err)) File "", line 63, in error raise CommandError("Error: %s" % message) Error: argument --flag: ignored explicit argument 'True'


Question: # Any required arguments which are passed in via **options must be passed     # to parse_args().     parse_args += [        '{}={}'.format(min(opt.option_strings), arg_options[opt.dest])        min(opt.option_strings)        if isinstance(opt, (_AppendConstAction, _CountAction, _StoreConstAction))        else '{}={}'.format(min(opt.option_strings), arg_options[opt.dest])         for opt in parser_actions if (             opt.dest in options and             (opt.required or opt in mutually_exclusive_required_options)


Question: from django.core import exceptions from django.utils.functional import cached_property<ins>from django.utils.hashable import make_hashable</ins>  from . import BLANK_CHOICE_DASH from .mixins import FieldCacheMixin

Queryset crashes when recreated from a pickled query with FilteredRelation used in aggregation. Description I am pickling query objects () for later re-evaluation as per  However, when I tried to rerun a query that contains a FilteredRelation inside a filter, I get an missing FROM-clause entry for table "t3" error. I created a minimum reproducible example. from import models class Publication(models.Model): title = models.CharField(maxlength=64) class Session(models.Model): TYPECHOICES = (('A', 'A'), ('B', 'B')) publication = models.ForeignKey(Publication, ondelete=models.CASCADE) sessiontype = models.CharField(choices=TYPECHOICES, default='A', maxlength=1) place = models.CharField(maxlength=16) value = models.PositiveIntegerField(default=1) The actual code to cause the crash: import pickle from import FilteredRelation, Q, Sum from import Publication, Session p1 = (title='Foo') p2 = (title='Bar') (publication=p1, sessiontype='A', place='X', value=1) (publication=p1, sessiontype='B', place='X', value=2) (publication=p2, sessiontype='A', place='X', value=4) (publication=p2, sessiontype='B', place='X', value=8) (publication=p1, sessiontype='A', place='Y', value=1) (publication=p1, sessiontype='B', place='Y', value=2) (publication=p2, sessiontype='A', place='Y', value=4) (publication=p2, sessiontype='B', place='Y', value=8) qs = ().annotate( relevantsessions=FilteredRelation('session', condition=Q(sessionsessiontype='A')) ).annotate(x=Sum('relevantsessionsvalue')) print(list(qs)) qs2 = () = (()) print(list(qs2)) In the crashing query, there seems to be a difference in the tablemap attribute - this is probably where the t3 table is coming from. Please let me know if there is any more info required for hunting this down. Cheers Beda p.s.- I also tried in Django 3.1 and the behavior is the same.


Question: from django.core import exceptions from django.utils.functional import cached_property<ins>from django.utils.hashable import make_hashable</ins>  from . import BLANK_CHOICE_DASH from .mixins import FieldCacheMixin

p.p.s.- just to make sure, I am not interested in ideas on how to rewrite the query - the above is a very simplified version of what I use, so it would probably not be applicable anyway.


Question: self.related_model._meta.model_name,         ) <ins>    @property    def identity(self):        return (            self.field,            self.model,            self.related_name,            self.related_query_name,            tuple(sorted(make_hashable(self.limit_choices_to))),            self.parent_link,            self.on_delete,            self.symmetrical,            self.multiple,        )    def __eq__(self, other):        if not isinstance(other, self.__class__):            return NotImplemented        return self.identity == other.identity    def __hash__(self):        return hash(self.identity)</ins>     def get_choices(         self, include_blank=True, blank_choice=BLANK_CHOICE_DASH,         limit_choices_to=None, ordering=(),


Question: self.related_model._meta.model_name,         ) <ins>    @property    def identity(self):        return (            self.field,            self.model,            self.related_name,            self.related_query_name,            tuple(sorted(make_hashable(self.limit_choices_to))),            self.parent_link,            self.on_delete,            self.symmetrical,            self.multiple,        )    def __eq__(self, other):        if not isinstance(other, self.__class__):            return NotImplemented        return self.identity == other.identity    def __hash__(self):        return hash(self.identity)</ins>     def get_choices(         self, include_blank=True, blank_choice=BLANK_CHOICE_DASH,         limit_choices_to=None, ordering=(),


Question: state.pop('related_model', None)         return state <ins>    @property    def identity(self):        return super().identity + (self.field_name,)</ins>     def get_related_field(self):         """         Return the Field in the 'to' object to which this relationship is tied.


Question: state.pop('related_model', None)         return state <ins>    @property    def identity(self):        return super().identity + (self.field_name,)</ins>     def get_related_field(self):         """         Return the Field in the 'to' object to which this relationship is tied.


Question: self.symmetrical = symmetrical         self.db_constraint = db_constraint <ins>    @property    def identity(self):        return super().identity + (            self.through,            self.through_fields,            self.db_constraint,        )</ins>     def get_related_field(self):         """         Return the field in the 'to' object to which this relationship is tied.


Question: self.symmetrical = symmetrical         self.db_constraint = db_constraint <ins>    @property    def identity(self):        return super().identity + (            self.through,            self.through_fields,            self.db_constraint,        )</ins>     def get_related_field(self):         """         Return the field in the 'to' object to which this relationship is tied.


Question: Perform the query and return a single object matching the given         keyword arguments.         """<ins>        if self.query.combinator and (args or kwargs):            raise NotSupportedError(                'Calling QuerySet.get(...) with filters after %s() is not '                'supported.' % self.query.combinator            )</ins>         clone = self._chain() if self.query.combinator else self.filter(*args, **kwargs)         if self.query.can_filter() and not self.query.distinct_fields:             clone = clone.order_by()

Raise a descriptive error on get() with filters operation following (), intersection(), and difference(). Description The QuerySet resulting from q1.difference(q2) does not allow to retrieve one specific object of that resulting QuerySet. Minimal example: from import models class Test(models.Model): name = models.CharField(max_length=30) python shell from import Test qs1 = () qs1 <QuerySet [<Test: Test object (1), <Test: Test object (2), <Test: Test object (3)]qs2 = (id=1) qs2 <QuerySet [<Test: Test object (1)]qs3 = qs1.difference(qs2) qs3 <QuerySet <Test: Test object (2), <Test: Test object (3) Traceback (most recent call last): File "<console", line 1, in <moduleFile "/lib/python3.8/site-", line 433, in get raise ( get() returned more than one Test -- it returned 2! Django version: 3.1.2 Python version: 3.8.5 OS: Arch Linux I also experienced this in the regular request/view-context, with other Django versions (2.2) and other python versions (3.7). Sorry if this is the expected behavior, a known bug or if I missed something which changes the behavior only on my system. If you need more information, I'll be happy to assist.


Question: import datetime import decimal<ins>import json</ins> from collections import defaultdict  from django.core.exceptions import FieldDoesNotExist

Admin doesn't display properly unicode chars in JSONFields. Description (last modified by ZhaoQi99) import json print ('') "\u4e2d\u56fd" use ASCII encoding by default when serializing Chinese. So when we edit a JsonField which contains Chinese character in Django admin,it will appear in ASCII characters. I have try to fix this this problem in  it works prefectly.


Question: return format_html('{}', value.url, value)     elif isinstance(field, models.JSONField) and value:         try:            return field.get_prep_value(value)            return json.dumps(value, ensure_ascii=False, cls=field.encoder)         except TypeError:             return display_for_value(value, empty_value_display)     else:diff --git a/django/forms/fields.py b/django/forms/fields.py-- a/django/forms/fields.py++ b/django/forms/fields.py


Question: def prepare_value(self, value):         if isinstance(value, InvalidJSONInput):             return value        return json.dumps(value, cls=self.encoder)        return json.dumps(value, ensure_ascii=False, cls=self.encoder)      def has_changed(self, initial, data):         if super().has_changed(initial, data):


Question: return None, [], None, []         return lower_bound, pre_context, context_line, post_context     def get_traceback_frames(self):        def explicit_or_implicit_cause(exc_value):            explicit = getattr(exc_value, '__cause__', None)            suppress_context = getattr(exc_value, '__suppress_context__', None)            implicit = getattr(exc_value, '__context__', None)            return explicit or (None if suppress_context else implicit)    def _get_explicit_or_implicit_cause(self, exc_value):        explicit = getattr(exc_value, '__cause__', None)        suppress_context = getattr(exc_value, '__suppress_context__', None)        implicit = getattr(exc_value, '__context__', None)        return explicit or (None if suppress_context else implicit)     def get_traceback_frames(self):         # Get the exception and all its causes         exceptions = []         exc_value = self.exc_value         while exc_value:             exceptions.append(exc_value)            exc_value = explicit_or_implicit_cause(exc_value)            exc_value = self._get_explicit_or_implicit_cause(exc_value)             if exc_value in exceptions:                 warnings.warn(                     "Cycle in the exception chain detected: exception '%s' "

debug error view doesn't respect exc.suppresscontext (PEP 415) Description Consider the following view that raises an exception: class TestView(View): def get(self, request, args, *kwargs): try: raise RuntimeError('my error') except Exception as exc: raise ValueError('my new error') from None Even though the raise is from None, unlike the traceback Python shows, the debug error view still shows the RuntimeError. This is because the explicitorimplicitcause() function inside gettracebackframes() doesn't respect exc.suppresscontext, which was introduced in Python 3.3's PEP 415:  def gettracebackframes(self): def explicitorimplicitcause(excvalue): explicit = getattr(excvalue, 'cause', None) implicit = getattr(excvalue, 'context', None) return explicit or implicit Instead, it should be something more like (simplifying also for Python 3): def explicitorimplicitcause(excvalue): return ( or (None if excvalue.suppresscontext else exc_value.context) )


Question: # In case there's just one exception, take the traceback from self.tb         exc_value = exceptions.pop()         tb = self.tb if not exceptions else exc_value.__traceback__<ins>        frames.extend(self.get_exception_traceback_frames(exc_value, tb))        while exceptions:            exc_value = exceptions.pop()            frames.extend(                self.get_exception_traceback_frames(exc_value, exc_value.__traceback__),            )        return frames    def get_exception_traceback_frames(self, exc_value, tb):        exc_cause = self._get_explicit_or_implicit_cause(exc_value)        exc_cause_explicit = getattr(exc_value, '__cause__', True)</ins>          while tb is not None:             # Support for __traceback_hide__ which is used by a few libraries


Question: pre_context = []                 context_line = '<source code not available>'                 post_context = []            frames.append({                'exc_cause': explicit_or_implicit_cause(exc_value),                'exc_cause_explicit': getattr(exc_value, '__cause__', True),            yield {                'exc_cause': exc_cause,                'exc_cause_explicit': exc_cause_explicit,                 'tb': tb,                 'type': 'django' if module_name.startswith('django.') else 'user',                 'filename': filename,


Question: 'context_line': context_line,                 'post_context': post_context,                 'pre_context_lineno': pre_context_lineno + 1,            })            # If the traceback for current exception is consumed, try the            # other exception.            if not tb.tb_next and exceptions:                exc_value = exceptions.pop()                tb = exc_value.__traceback__            else:                tb = tb.tb_next        return frames            }            tb = tb.tb_next   def technical_404_response(request, exception):


Question: def __getattr__(self, name):         return getattr(self._out, name) <ins>    def flush(self):        if hasattr(self._out, 'flush'):            self._out.flush()</ins>     def isatty(self):         return hasattr(self._out, 'isatty') and self._out.isatty()

flush() on management commands doesn't work. Description flush() is notably called during migrate command; it doesn't work, and a long migration effectively prints to stderr no relevant information up until the end: Operations to perform: Apply all migrations: myapp Running migrations: Then nothing more, but the migration is being done. Then at the end of the real migration, the rest is flushed: Applying myapp.0002auto202008171030... OK Expected behavior: Operations to perform: Apply all migrations: myapp Running migrations: Applying myapp.0002auto202008171030... then work then OK


Question: * {{ num2|floatformat:"-3" }} displays "34"     * {{ num3|floatformat:"-3" }} displays "34.260"     If arg has the 'g' suffix, force the result to be grouped by the    THOUSAND_SEPARATOR for the active locale. When the active locale is    en (English):    * {{ 6666.6666|floatformat:"2g" }} displays "6,666.67"    * {{ 10000|floatformat:"g" }} displays "10,000"     If the input float is infinity or NaN, display the string representation     of that value.     """    force_grouping = False    if isinstance(arg, str) and arg.endswith('g'):        force_grouping = True        arg = arg[:-1] or -1     try:         input_val = repr(text)         d = Decimal(input_val)

Incorrect separators when chaining floatformat to intcomma in some locales Description When u use floatvalue "2" and intcomma together in a template the output of intcomma won't be internationalized. Since intcomma wont work with decimals in django 1.5.1 i tried to convert a decimal to a float in a template, but it wont give me the excepted output. When i have the value of 1000.11 it should be 1000,11 in germany, with intcomma(float(1000,11)) i get 1.000,11. But when i use Decimal(1000,11)intcomma, i will get 1,000,11. Thats a bug or maybe an unwanted behavior.


Question: return input_val      if not m and p < 0:        return mark_safe(formats.number_format('%d' % (int(d)), 0))        return mark_safe(            formats.number_format('%d' % (int(d)), 0, force_grouping=force_grouping),        )      exp = Decimal(1).scaleb(-abs(p))     # Set the precision high enough to avoid an exception (#15789).


Question: if sign and rounded_d:         digits.append('-')     number = ''.join(reversed(digits))    return mark_safe(formats.number_format(number, abs(p)))    return mark_safe(        formats.number_format(number, abs(p), force_grouping=force_grouping),    )   @register.filter(is_safe=True)


Question: ordering_params = []             ordering_expr_sql = []             for expr in self.ordering:                expr_sql, expr_params = expr.as_sql(compiler, connection)                expr_sql, expr_params = compiler.compile(expr)                 ordering_expr_sql.append(expr_sql)                 ordering_params.extend(expr_params)             sql, sql_params = super().as_sql(compiler, connection, ordering=(diff --git a/django/contrib/postgres/constraints.py b/django/contrib/postgres/constraints.py-- a/django/contrib/postgres/constraints.py++ b/django/contrib/postgres/constraints.py

Using KeyTransform for JSONField produces invalid SQL in various places. Description (last modified by Igor Jerosimić) Using KeyTransform in ordering attribute of ArrayAgg function produces invalid SQL. I don't know if it matters but I'm using Postgres for DB. from import models class Parent(models.Model): name = models.CharField(default='test') class Child(models.Model): parent = models.ForeignKey( Parent, ondelete=models.SETNULL, relatedname='children', ) data = models.JSONField(default=dict) parent = () (parent=parent, data={'en': 'English', 'fr': 'French'}) Parent.objects.annotate( childrenarray=ArrayAgg( KeyTextTransform('en', 'childrendata'), distinct=True, ordering=[KeyTransform('en', 'childrendata')], ), ).all() Produces invalid SQL in the ORDER BY section: ARRAY_AGG(DISTINCT ("children"."data" -'default') ORDER BY None("children"."data")) NOTE: This was working fine before Django 3.1.


Question: if isinstance(expression, str):                 expression = F(expression)             expression = expression.resolve_expression(query=query)            sql, params = expression.as_sql(compiler, schema_editor.connection)            sql, params = compiler.compile(expression)             try:                 opclass = self.opclasses[idx]                 if opclass:diff --git a/django/db/models/expressions.py b/django/db/models/expressions.py-- a/django/db/models/expressions.py++ b/django/db/models/expressions.py


Question: return expression.get_group_by_cols(alias=alias)      def as_sql(self, compiler, connection):        return self.expression.as_sql(compiler, connection)        return compiler.compile(self.expression)      def __repr__(self):         return "{}({})".format(self.__class__.__name__, self.expression)diff --git a/django/db/models/fields/json.py b/django/db/models/fields/json.py-- a/django/db/models/fields/json.py++ b/django/db/models/fields/json.py


Question: class KeyTransformIn(lookups.In):    def process_rhs(self, compiler, connection):        rhs, rhs_params = super().process_rhs(compiler, connection)        if not connection.features.has_native_json_field:            func = ()    def resolve_expression_parameter(self, compiler, connection, sql, param):        sql, params = super().resolve_expression_parameter(            compiler, connection, sql, param,        )        if (            not hasattr(param, 'as_sql') and            not connection.features.has_native_json_field        ):             if connection.vendor == 'oracle':                func = []                for value in rhs_params:                    value = json.loads(value)                    function = 'JSON_QUERY' if isinstance(value, (list, dict)) else 'JSON_VALUE'                    func.append("%s('%s', '$.value')" % (                        function,                        json.dumps({'value': value}),                    ))                func = tuple(func)                rhs_params = ()            elif connection.vendor == 'mysql' and connection.mysql_is_mariadb:                func = ("JSON_UNQUOTE(JSON_EXTRACT(%s, '$'))",) * len(rhs_params)                value = json.loads(param)                if isinstance(value, (list, dict)):                    sql = "JSON_QUERY(%s, '$.value')"                else:                    sql = "JSON_VALUE(%s, '$.value')"                params = (json.dumps({'value': value}),)             elif connection.vendor in {'sqlite', 'mysql'}:                func = ("JSON_EXTRACT(%s, '$')",) * len(rhs_params)            rhs = rhs % func        return rhs, rhs_params                sql = "JSON_EXTRACT(%s, '$')"        if connection.vendor == 'mysql' and connection.mysql_is_mariadb:            sql = 'JSON_UNQUOTE(%s)' % sql        return sql, params   class KeyTransformExact(JSONExact):diff --git a/django/db/models/lookups.py b/django/db/models/lookups.py-- a/django/db/models/lookups.py++ b/django/db/models/lookups.py


Question: if hasattr(param, 'resolve_expression'):             param = param.resolve_expression(compiler.query)         if hasattr(param, 'as_sql'):            sql, params = param.as_sql(compiler, connection)            sql, params = compiler.compile(param)         return sql, params      def batch_process_rhs(self, compiler, connection, rhs=None):


Question: client_cert = settings_dict['OPTIONS'].get('ssl', {}).get('cert')         client_key = settings_dict['OPTIONS'].get('ssl', {}).get('key')         defaults_file = settings_dict['OPTIONS'].get('read_default_file')<ins>        charset = settings_dict['OPTIONS'].get('charset')</ins>         # Seems to be no good way to set sql_mode with CLI.          if defaults_file:

MySQL: dbshell does not get charset from DATABASES setting Description I noticed that dbshell doesn't respect the databaseoptions. I ran into an issue with an application we are creating that needs to support mysql and postgre at least, we execute some sql scripts that get piped to dbshell (to avoid hardcoding psql -U xxx or mysql -u xxx and creating 2 scripts every time). When running an utf8 database with utf8 as our charset in databaseoptions, we ran into some weird encoding issues. The solution for us was to learn to respect the encoding settings in Are you opposed to something like this? Attaching small patch that fixes our problem. Let me know if it needs extending to support other backends or database_options.


Question: args += ["--ssl-cert=%s" % client_cert]         if client_key:             args += ["--ssl-key=%s" % client_key]<ins>        if charset:            args += ['--default-character-set=%s' % charset]</ins>         if db:             args += [db]         args.extend(parameters)


Question: def _make_hash_value(self, user, timestamp):         """        Hash the user's primary key and some user state that's sure to change        after a password reset to produce a token that invalidated when it's        used:        Hash the user's primary key, email (if available), and some user state        that's sure to change after a password reset to produce a token that is        invalidated when it's used:         1. The password field will change upon a password reset (even if the            same password is chosen, due to password salting).         2. The last_login field will usually be updated very shortly after

Changing user's email could invalidate password reset tokens Description Sequence: Have account with email address foo Password reset request for that email (unused) foo account changes their email address Password reset email is used The password reset email's token should be rejected at that point, but in fact it is allowed. The fix is to add the user's email address into PasswordResetTokenGenerator.makehashvalue() Nothing forces a user to even have an email as per AbstractBaseUser. Perhaps the token generation method could be factored out onto the model, ala getsessionauthhash().


Question: # Truncate microseconds so that tokens are consistent even if the         # database doesn't support microseconds.         login_timestamp = '' if user.last_login is None else user.last_login.replace(microsecond=0, tzinfo=None)        return str(user.pk) + user.password + str(login_timestamp) + str(timestamp)        email_field = user.get_email_field_name()        email = getattr(user, email_field, '') or ''        return f'{user.pk}{user.password}{login_timestamp}{timestamp}{email}'      def _num_seconds(self, dt):         return int((dt - datetime(2001, 1, 1)).total_seconds())


Question: # If the query is an EmptyQuerySet, combine all nonempty querysets.         if isinstance(self, EmptyQuerySet):             qs = [q for q in other_qs if not isinstance(q, EmptyQuerySet)]            return qs[0]._combinator_query('union', *qs[1:], all=all) if qs else self            if not qs:                return self            if len(qs) == 1:                return qs[0]            return qs[0]._combinator_query('union', *qs[1:], all=all)         return self._combinator_query('union', *other_qs, all=all)      def intersection(self, *other_qs):

QuerySet.orderby() crashes on union() queryset with a single non-empty query. Description (last modified by Hannes Ljungberg) I noticed this error while implementing a dynamic union like this: union = reduce( lambda qs1, qs2: (qs2), querysets, (), ) If len(querysets) == 1 it will result in a pretty weird query throwing a database error when both querysets are ordered, another example recreating this in the test suite: qs1 = ().orderby(‘pk’) qs2 = ().union(qs1).orderby(‘pk’) Executing qs2 result in the following query: (SELECT "queriesnumber"."id", "queriesnumber"."num", "queriesnumber"."othernum", "queriesnumber"."anothernum" FROM "queriesnumber" ORDER BY "queriesnumber"."id" ASC) ORDER BY (1) ASC Result in the following error on PostgresSQL: multiple ORDER BY clauses not allowed LINE 1: " ORDER BY "queriesnumber"."id" DESC) ORDER BY (1) ASC And sqlite: ORDER BY not allowed in subqueries of compound statements. MySQL seems to accept this query but it's most likely not the query the user expects the be executed. My proposal is to simply return the non-empty query and make the union a no-op set in this case.


Question: return self.SchemaEditorClass(self, *args, **kwargs)      def on_commit(self, func):<ins>        if not callable(func):            raise TypeError("on_commit()'s callback must be a callable.")</ins>         if self.in_atomic_block:             # Transaction in progress; save for execution on commit.             self.run_on_commit.append((set(self.savepoint_ids), func))

Allowing null values while registering function on transaction oncommit Description So if I were to try this command:- transaction.oncommit(None) and run any sample tests which has simple django db post_save signal in the flow and it has functions which run on transaction commit, the code would crash with error TypeError: 'NoneType' object is not callable. So how about if we do not allow it to register as none beforehand or allow it to exit gracefully?


Question: model.__name__,                     ', '.join(protected_objects),                 ),                chain.from_iterable(protected_objects.values()),                set(chain.from_iterable(protected_objects.values())),             )         for related_model, related_fields in model_fast_deletes.items():             batches = self.get_del_batches(new_objs, related_fields)

ProtectedError/RestrictedError raised from Collector.collect contains iterators. Description (last modified by Kwist) ("Cannot delete some instances of model 'A' because they are referenced through protected foreign keys: 'A.protect', 'B.protect'.", object at 0x7f96b5cb6c40) As a result, the repr of exception in Sentry is not informative.


Question: model.__name__,                             ', '.join(restricted_objects),                         ),                        chain.from_iterable(restricted_objects.values()),                        set(chain.from_iterable(restricted_objects.values())),                     )      def related_objects(self, related_model, related_fields, objs):


Question: )      # Check that the username field is unique    if not cls._meta.get_field(cls.USERNAME_FIELD).unique:    if not cls._meta.get_field(cls.USERNAME_FIELD).unique and not any(        constraint.fields == (cls.USERNAME_FIELD,)        for constraint in cls._meta.total_unique_constraints    ):         if (settings.AUTHENTICATION_BACKENDS ==                 ['django.contrib.auth.backends.ModelBackend']):             errors.append(

Skip auth.E003 system check for USERNAMEFIELD with total UniqueConstraints. Description Defining a user model like this: class User(AbstractBaseUser): username = models.CharField(maxlength=30) USERNAMEFIELD = "username" class Meta: constraints = [UniqueConstraint(fields=["username"], name="userusernameunq")] Will trigger auth.E003: auth.User: (auth.E003) 'User.username' must be unique because it is named as the 'USERNAMEFIELD'. Sometimes it’s not preferable to set the field as unique with unique=True as it will create an extra implicit *like index for CharField and TextField on PostgresSQL. The system check should be extended to check for the presence of USERNAMEFIELD in Model._meta.constraints. Not really sure if this classifies as a bug.


Question: def as_sqlite(self, compiler, connection, **extra_context):         return super().as_sql(compiler, connection, function='RAND', **extra_context) <ins>    def get_group_by_cols(self, alias=None):        return []</ins>  class Round(Transform):     function = 'ROUND'

orderby('?') unexpectedly breaking queryset aggregation Description Steps to reproduce: class Thing(models.Model): pass class Related(models.Model): models.ForeignKey(Thing) With data t = () rs = [(thing=t) for in range(2)] The following query works as expected. The aggregation with Count produces a GROUP BY clause on Thing.objects.annotate(rc=Count('related')).orderby('rc').values('id', 'rc') <QuerySet [{'id': 1, 'rc': 2}]This also works as expected (at least to me). Although there is an aggregation, ordering by related means that the grouping will be broken down. Thing.objects.annotate(rc=Count('related')).orderby('related').values('id', 'rc') <QuerySet [{'id': 1, 'rc': 1}, {'id': 1, 'rc': 1}]But the following seems wrong to me. Thing.objects.annotate(rc=Count('related')).orderby('?').values('id', 'rc') <QuerySet [{'id': 1, 'rc': 1}, {'id': 1, 'rc': 1}]The random function call has nothing to do with the aggregation, and I see no reason it should break it. Dumping the query seems that indeed the random call breaks the group by call: (I simpilfied the table names a little) print(Thing.objects.annotate(rc=Count('related')).orderby('?').values('id', 'rc').query) SELECT "thing"."id", COUNT("related"."id") AS "rc" FROM "thing" LEFT OUTER JOIN "related" ON ("thing"."id" = "related"."thingid") GROUP BY "thing"."id", RANDOM() ORDER BY RANDOM() ASC I dug into the SQL compiler, and it seems to me the problem is inside , where the compiler combines all non-aggregate, non-ref orderby expressions into groupby. I patched it like this for expr, (sql, params, isref) in orderby: if expr.containsaggregate: continue if isref: continue ([ exp for exp in expr.getsource_expressions() if not isinstance(exp, Random) ]) and things seem to work correctly. No failed tests against SQLite3 with default settings.


Question: form = ManagementForm(self.data, auto_id=self.auto_id, prefix=self.prefix)             if not form.is_valid():                 raise ValidationError(                    _('ManagementForm data is missing or has been tampered with'),                    _(                        'ManagementForm data is missing or has been tampered '                        'with. Missing fields: %(field_names)s'                    ) % {                        'field_names': ', '.join(                            form.add_prefix(field_name) for field_name in form.errors                        ),                    },                     code='missing_management_form',                 )         else:

ManagementForm exception in case of bad prefix should be easier to understand Description If user adds multiple formsets with prefixes, and specifies the prefix incorrectly when binding the form and validating: someformset = SomeFormSet(request.POST, 'articles') instead of: someformset = SomeFormSet(request.POST, prefix='articles') Django "suppresses" the original exception and raises only relatively unhelpful "ManagementForm data is missing or has been tampered with". In file , line 57: if or form = ManagementForm(, , ) if not form.isvalid(): raise ValidationError('ManagementForm data is missing or has been tampered with') Suggestion: include in output, because for such a small bug in code, it can take a really long time find it. {'INITIALFORMS': [u'This field is required.'], 'MAXNUMFORMS': [u'This field is required.'], 'TOTALFORMS': [u'This field is required.']}


Question: from datetime import datetimefrom datetime import datetime, time  from django.conf import settings from django.utils.crypto import constant_time_compare, salted_hmac

Password reset token incompatibility. Description As noted here  the hashing for password reset tokens has changed between 3.0 and 3.1 and work has been done to ensure existing tokens will still work (at least until 4.0). However the encoding of the token creation time has also changed. Specifically from days since 1/1/01 to seconds since 1/1/01. And it appears no work has been done to support tokens with the older values. So a token generated on Oct 1, 2020 will come through as 7213 days which will then get interpreted as 7213 seconds, aka 2am Jan 1, 2001. So while exiting tokens in the wild will pass crypto validation they will all show as expired if your PASSWORDRESETTIMEOUT is less than ~20 years. The code base I'm working on uses these tokens (perhaps unwisely) in some email links that are expected to have a 3 month lifetime and an upgrade from 3.0 to 3.1 looks likely to render all the tokens in the wild expired which is suboptimal.


Question: # Parse the token         try:             ts_b36, _ = token.split("-")<ins>            # RemovedInDjango40Warning.            legacy_token = len(ts_b36) < 4</ins>         except ValueError:             return False


Question: ):                 return False         # RemovedInDjango40Warning: convert days to seconds and round to        # midnight (server time) for pre-Django 3.1 tokens.        now = self._now()        if legacy_token:            ts *= 24 * 60 * 60            ts += int((now - datetime.combine(now.date(), time.min)).total_seconds())         # Check the timestamp is within limit.        if (self._num_seconds(self._now()) - ts) > settings.PASSWORD_RESET_TIMEOUT:        if (self._num_seconds(now) - ts) > settings.PASSWORD_RESET_TIMEOUT:             return False          return True


Question: that the "save" must be an SQL insert or update (or equivalent for         non-SQL backends), respectively. Normally, they should not be set.         """        # Ensure that a model instance without a PK hasn't been assigned to        # a ForeignKey or OneToOneField on this model. If the field is        # nullable, allowing the save() would result in silent data loss.        for field in self._meta.concrete_fields:            # If the related field isn't cached, then an instance hasn't            # been assigned and there's no need to worry about this check.            if field.is_relation and field.is_cached(self):                obj = getattr(self, field.name, None)                if not obj:                    continue                # A pk may have been assigned manually to a model instance not                # saved to the database (or auto-generated in a case like                # UUIDField), but we allow the save to proceed and rely on the                # database to raise an IntegrityError if applicable. If                # constraints aren't supported by the database, there's the                # unavoidable risk of data corruption.                if obj.pk is None:                    # Remove the object from a related instance cache.                    if not field.remote_field.multiple:                        field.remote_field.delete_cached_value(obj)                    raise ValueError(                        "save() prohibited to prevent data loss due to "                        "unsaved related object '%s'." % field.name                    )                elif getattr(self, field.attname) is None:                    # Use pk from related object if it has been saved after                    # an assignment.                    setattr(self, field.attname, obj.pk)                # If the relationship's pk/to_field was changed, clear the                # cached relationship.                if getattr(obj, field.target_field.attname) != getattr(self, field.attname):                    field.delete_cached_value(self)        self._prepare_related_fields_for_save(operation_name='save')          using = using or router.db_for_write(self.__class__, instance=self)         if force_insert and (force_update or update_fields):

Saving parent object after setting on child leads to unexpected data loss in bulkcreate(). Description Example: class Country(models.Model): name = models.CharField(maxlength=255) isotwoletter = models.CharField(maxlength=2) description = models.TextField() class City(models.Model): name = models.CharField(maxlength=255) country = models.ForeignKey(Country, ondelete=models.CASCADE) class BulkCreateTests(TestCase): def testfkbug(self): countrynl = Country(name='Netherlands', isotwoletter='NL') countrybe = Country(name='Belgium', isotwoletter='BE') city = City(country=countrybe, name='Brussels') # (1) () # (2) () This results in an integrity error: ERROR: testfkbug () Traceback (most recent call last): File "", line 85, in execute return (sql, params) psycopg2.IntegrityError: null value in column "countryid" violates not-null constraint DETAIL: Failing row contains (1, Brussels, null). The above exception was the direct cause of the following exception: ... I wonder wether there's a reason that this doesn't work. If setting a related object on a model instance automatically sets instance.relatedobjectid, you'd expect this behavior to continue working if the related object receives its primary key after the referencing instance was initialized. Of course, switching lines (1) and (2) makes it all work, but this behavior means that bulk creating with related objects (with postgres returning the primary keys, bless it's heart) becomes more complex than it should be, forcing the user to use mappings or even set the foreign keys themselves: for countrydata, citydata in data: country = Country(countrydata) (country) city = City(country=country, citydata) (city) Country.objects.bulkcreate(countries) for city in cities: city.countryid = City.objects.bulk_create(cities) Ideally, the main instance would, when saved, 'reach into' its related objects and set the foreign key field like the loop does. Is there a reason this can't work?


Question: using=using, raw=raw,         )     def _prepare_related_fields_for_save(self, operation_name):        # Ensure that a model instance without a PK hasn't been assigned to        # a ForeignKey or OneToOneField on this model. If the field is        # nullable, allowing the save would result in silent data loss.        for field in self._meta.concrete_fields:            # If the related field isn't cached, then an instance hasn't been            # assigned and there's no need to worry about this check.            if field.is_relation and field.is_cached(self):                obj = getattr(self, field.name, None)                if not obj:                    continue                # A pk may have been assigned manually to a model instance not                # saved to the database (or auto-generated in a case like                # UUIDField), but we allow the save to proceed and rely on the                # database to raise an IntegrityError if applicable. If                # constraints aren't supported by the database, there's the                # unavoidable risk of data corruption.                if obj.pk is None:                    # Remove the object from a related instance cache.                    if not field.remote_field.multiple:                        field.remote_field.delete_cached_value(obj)                    raise ValueError(                        "%s() prohibited to prevent data loss due to unsaved "                        "related object '%s'." % (operation_name, field.name)                    )                elif getattr(self, field.attname) is None:                    # Use pk from related object if it has been saved after                    # an assignment.                    setattr(self, field.attname, obj.pk)                # If the relationship's pk/to_field was changed, clear the                # cached relationship.                if getattr(obj, field.target_field.attname) != getattr(self, field.attname):                    field.delete_cached_value(self)     def delete(self, using=None, keep_parents=False):         using = using or router.db_for_write(self.__class__, instance=self)         assert self.pk is not None, (diff --git a/django/db/models/query.py b/django/db/models/query.py-- a/django/db/models/query.py++ b/django/db/models/query.py


Question: obj.save(force_insert=True, using=self.db)         return obj     def _populate_pk_values(self, objs):    def _prepare_for_bulk_create(self, objs):         for obj in objs:             if obj.pk is None:                # Populate new PK values.                 obj.pk = obj._meta.pk.get_pk_value_on_save(obj)            obj._prepare_related_fields_for_save(operation_name='bulk_create')      def bulk_create(self, objs, batch_size=None, ignore_conflicts=False):         """


Question: opts = self.model._meta         fields = opts.concrete_fields         objs = list(objs)        self._populate_pk_values(objs)        self._prepare_for_bulk_create(objs)         with transaction.atomic(using=self.db, savepoint=False):             objs_with_pk, objs_without_pk = partition(lambda o: o.pk is None, objs)             if objs_with_pk:


Question: elif isinstance(value, (list, tuple)):             # The items of the iterable may be expressions and therefore need             # to be resolved independently.            return type(value)(            values = (                 self.resolve_lookup_value(sub_value, can_reuse, allow_joins)                 for sub_value in value             )            type_ = type(value)            if hasattr(type_, '_make'):  # namedtuple                return type_(*values)            return type_(values)         return value      def solve_lookup_type(self, lookup):

Upgrading 2.23.0 causes named tuples used as arguments to range to error. Description I noticed this while upgrading a project from 2.2 to 3.0. This project passes named 2-tuples as arguments to range queryset filters. This works fine on 2.2. On 3.0 it causes the following error: TypeError: new_() missing 1 required positional argument: 'far'. This happens because goes into the tuple elements to resolve lookups and then attempts to reconstitute the tuple with the resolved elements. When it attempts to construct the new tuple it preserves the type (the named tuple) but it passes a iterator to it's constructor. NamedTuples don't have the code path for copying an iterator, and so it errors on insufficient arguments. The fix is to * expand the contents of the iterator into the constructor.


Question: })  <ins>class SmallIntegerField(IntegerField):    description = _('Small integer')    def get_internal_type(self):        return 'SmallIntegerField'</ins> class IPAddressField(Field):     empty_strings_allowed = False     description = _("IPv4 address")

ManyToManyField does not respect the PositiveBigIntegerField in m2m intermediate table. Description When creating a relation between 2 models using PositiveBigIntegerField on Postgresql, the relation table is created using regular ints as the column type. This in turn leads to out of bound error when using large numbers for ids. from import models class Node(models.Model): id = models.PositiveBigIntegerField(primarykey=True) point = models.PointField() class Relation(models.Model): id = models.PositiveBigIntegerField(primarykey=True) nodes = models.ManyToManyField(Node) The generated table will look like this: Column Description -------------+---------+-----------+----------+------------------------------------------------+---------+--------------+------------- id relationid nodeid As you can see, the PositiveBigInteger is not respected and a regular int is set


Question: class PositiveIntegerRelDbTypeMixin:<ins>    def __init_subclass__(cls, **kwargs):        super().__init_subclass__(**kwargs)        if not hasattr(cls, 'integer_field_class'):            cls.integer_field_class = next(                (                    parent                    for parent in cls.__mro__[1:]                    if issubclass(parent, IntegerField)                ),                None,            )</ins>      def rel_db_type(self, connection):         """


Question: if connection.features.related_fields_match_type:             return self.db_type(connection)         else:            return IntegerField().db_type(connection=connection)            return self.integer_field_class().db_type(connection=connection)  class PositiveBigIntegerField(PositiveIntegerRelDbTypeMixin, IntegerField):class PositiveBigIntegerField(PositiveIntegerRelDbTypeMixin, BigIntegerField):     description = _('Positive big integer')      def get_internal_type(self):


Question: })  class PositiveSmallIntegerField(PositiveIntegerRelDbTypeMixin, IntegerField):class PositiveSmallIntegerField(PositiveIntegerRelDbTypeMixin, SmallIntegerField):     description = _("Positive small integer")      def get_internal_type(self):


Question: })  <del>class SmallIntegerField(IntegerField):    description = _("Small integer")    def get_internal_type(self):        return "SmallIntegerField"</del> class TextField(Field):     description = _("Text")


Question: from django.utils.functional import cached_property from django.utils.html import html_safe from django.utils.safestring import mark_safefrom django.utils.translation import gettext as _, ngettextfrom django.utils.translation import gettext_lazy as _, ngettext  __all__ = ('BaseFormSet', 'formset_factory', 'all_valid')

BaseFormSet.isvalid() produces ValidationError when there is no management form Description I was torn between reporting this as a bug or a feature request, but then I thought if I make it a feature request it will most likely break a lot of Django apps and hence I guess that means it's more of a bug... Anyway so the line in question is (in Django version 1.6.1): for i in range(0, self.totalformcount()): the self.totalformcount() executes this line (in Django version 1.6.1): return min(self.managementform.cleaneddata[TOTALFORMCOUNT], self.absolutemax) ..which then raises this exception (in Django version 1.6.1): raise ValidationError( ('ManagementForm data is missing or has been tampered with'), code='missingmanagementform', ) That stack trace occurs if/when a user submits a formset after stripping out the management form hidden fields. I have been using Django for a few years now and have never come across an exception being raised by a form/formset isvalid() call before. So my point is that I believe this exception should never be allowed to leave the BaseFormSet.isvalid() call, because it is an oddball behaviour compared to the rest of the isvalid() implementations. I.e. I believe there should be a check in BaseFormSet.isvalid() which checks for the presence of a valid management form (first) and returns False if it is not present, as opposed to raising an exception. Yes I could wrap the isvalid() call in a try/catch, but I believe this is an unnecessary hack caused by a bad design deviation of the implementation of the BaseFormSet.is_valid() method. I didn't bother creating a patch and test cases, because I have a feeling this will get rejected or something like that, but I just thought I should bring this up, as I can't find mention of it anywhere and it seems important to me.


Question: self.base_fields[MAX_NUM_FORM_COUNT] = IntegerField(required=False, widget=HiddenInput)         super().__init__(*args, **kwargs) <ins>    def clean(self):        cleaned_data = super().clean()        # When the management form is invalid, we don't know how many forms        # were submitted.        cleaned_data.setdefault(TOTAL_FORM_COUNT, 0)        cleaned_data.setdefault(INITIAL_FORM_COUNT, 0)        return cleaned_data</ins>  @html_safe class BaseFormSet:


Question: A collection of instances of the same Form class.     """     ordering_widget = NumberInput    default_error_messages = {        'missing_management_form': _(            'ManagementForm data is missing or has been tampered with. Missing fields: '            '%(field_names)s. You may need to file a bug report if the issue persists.'        ),    }      def __init__(self, data=None, files=None, auto_id='id_%s', prefix=None,                 initial=None, error_class=ErrorList, form_kwargs=None):                 initial=None, error_class=ErrorList, form_kwargs=None,                 error_messages=None):         self.is_bound = data is not None or files is not None         self.prefix = prefix or self.get_default_prefix()         self.auto_id = auto_id


Question: self._errors = None         self._non_form_errors = None <ins>        messages = {}        for cls in reversed(type(self).__mro__):            messages.update(getattr(cls, 'default_error_messages', {}))        if error_messages is not None:            messages.update(error_messages)        self.error_messages = messages</ins>     def __str__(self):         return self.as_table()


Question: """Return the ManagementForm instance for this FormSet."""         if self.is_bound:             form = ManagementForm(self.data, auto_id=self.auto_id, prefix=self.prefix)            if not form.is_valid():                raise ValidationError(                    _(                        'ManagementForm data is missing or has been tampered '                        'with. Missing fields: %(field_names)s'                    ) % {                        'field_names': ', '.join(                            form.add_prefix(field_name) for field_name in form.errors                        ),                    },                    code='missing_management_form',                )            form.full_clean()         else:             form = ManagementForm(auto_id=self.auto_id, prefix=self.prefix, initial={                 TOTAL_FORM_COUNT: self.total_form_count(),


Question: if not self.is_bound:  # Stop further processing.             return<ins>        if not self.management_form.is_valid():            error = ValidationError(                self.error_messages['missing_management_form'],                params={                    'field_names': ', '.join(                        self.management_form.add_prefix(field_name)                        for field_name in self.management_form.errors                    ),                },                code='missing_management_form',            )            self._non_form_errors.append(error)</ins>         for i, form in enumerate(self.forms):             # Empty forms are unchanged forms beyond those with initial data.             if not form.has_changed() and i >= self.initial_form_count():


Question: # Build po files for each selected locale             for locale in locales:<ins>                if '-' in locale:                    self.stdout.write(                        'invalid locale %s, did you mean %s?' % (                            locale,                            locale.replace('-', '_'),                        ),                    )                    continue</ins>                 if self.verbosity > 0:                     self.stdout.write('processing locale %s' % locale)                 for potfile in potfiles:

Add warning in makemessages command if the localecode with flag is not correct Description (last modified by Sanyam Khurana) Hey Calude, What about normalizing the directory name to something that would just work. For example, No matter, if the developer is doing all these: python makemessages -l zhcn python makemessages -l zhCN python makemessages -l ZHCN python makemessages -l ZH-CN etc. we, just normalize the directory name to zhCN and it would work. I'm still about to read the code of makemessages command and probably if there are any more checks than just this, then we'll have to figure out another way all together.


Question: if locale_dir is NO_LOCALE_DIR:                 file_path = os.path.normpath(build_files[0].path)                 raise CommandError(                    'Unable to find a locale path to store translations for '                    'file %s' % file_path                    "Unable to find a locale path to store translations for "                    "file %s. Make sure the 'locale' directory exist in an "                    "app or LOCALE_PATHS setting is set." % file_path                 )             for build_file in build_files:                 msgs = build_file.postprocess_messages(msgs)

Unclear error raised by makemessages when /locale does not exist under app directory Description I got this error message: CommandError: Unable to find a locale path to store translations for file I had to google the error message and got to this:  It would be better if the error message was simple and obvious so you could solve the problem yourself directly instead. I thought the error message would be better if it said 'We are looking for a directory called "locale"" but after that change the command fails silently anyway. Maybe LOCALE_PATHS should also include the directory where is by default? Or some other error message should exist...


Question: field_names.append(f)             self.set_extra_mask(extra_names)             self.set_annotation_mask(annotation_names)            selected = frozenset(field_names + extra_names + annotation_names)         else:             field_names = [f.attname for f in self.model._meta.concrete_fields]            selected = frozenset(field_names)         # Selected annotations must be known before setting the GROUP BY         # clause.         if self.group_by is True:

Aggregation over subquery annotation GROUP BY produces wrong results Description (last modified by Christian Klus) Starting in django 3.0.7, specifically after patch I noticed some of my more complex queries returning incorrect results. I think I've narrowed it down to a simpler test case: Example query: ().annotate( pubyear=TruncYear('pubdate') ).orderby().values('pubyear').annotate( totalpages=Sum('pages'), toprating=Subquery( ( pubdateyear=OuterRef('pubyear') ).orderby('rating').values('rating')[:1] ) ).values('pubyear', 'totalpages', 'toprating') Generated SQL on 3.0.6: SELECT djangodatetrunc('year', "aggregationregressbook"."pubdate") AS "pubyear", SUM("aggregationregressbook"."pages") AS "totalpages", ( SELECT U0."rating" FROM "aggregationregressbook" U0 WHERE djangodateextract('year', U0."pubdate") = djangodatetrunc('year', "aggregationregressbook"."pubdate") ORDER BY U0."rating" ASC LIMIT 1 ) AS "toprating" FROM "aggregationregressbook" GROUP BY djangodatetrunc('year', "aggregationregressbook"."pubdate"), "toprating" Generated SQL on current master: SELECT djangodatetrunc('year', "aggregationregressbook"."pubdate", NULL, NULL) AS "pubyear", SUM("aggregationregressbook"."pages") AS "totalpages", ( SELECT U0."rating" FROM "aggregationregressbook" U0 WHERE djangodateextract('year', U0."pubdate") = djangodatetrunc('year', "aggregationregressbook"."pubdate", NULL, NULL) ORDER BY U0."rating" ASC LIMIT 1 ) AS "toprating" FROM "aggregationregressbook" GROUP BY djangodatetrunc('year', "aggregationregressbook"."pubdate", NULL, NULL), ( SELECT U0."rating" FROM "aggregationregressbook" U0 WHERE djangodateextract('year', U0.


Question: field_names.append(f)             self.set_extra_mask(extra_names)             self.set_annotation_mask(annotation_names)            selected = frozenset(field_names + extra_names + annotation_names)         else:             field_names = [f.attname for f in self.model._meta.concrete_fields]            selected = frozenset(field_names)         # Selected annotations must be known before setting the GROUP BY         # clause.         if self.group_by is True:

"pubdate") = djangodatetrunc('year', "aggregationregressbook"."pubdate", NULL, NULL) ORDER BY U0."rating" ASC LIMIT 1 ), "aggregationregressbook"."pubdate" I see two separate issues here: "aggregationregressbook"."pubdate" is being to the group by clause incorrectly (this issue probably predates the patch mentioned above) Even though the subquery is in the select statement, the alias is not being used and instead the subquery is reevaluated. This nearly doubles the cost of one of my queries that is experiencing this problem. I don't know much about the ORM internals, but here is my naive patch for the second issue: diff --git index .. --- +++ -2220,7 +2220,7 class Query(BaseExpression): # the selected fields anymore. groupby = [] for expr in self.groupby: if isinstance(expr, Ref) and not in fieldnames: if isinstance(expr, Ref) and not in fieldnames + annotationnames: expr = self.annotations[] (expr) self.groupby = tuple(groupby) I'd appreciate anyone with deeper knowlege of the ORM to chime in and let me know if I'm on the right track. Tests are passing locally. The resulting query on master: SELECT djangodatetrunc('year', "aggregationregressbook"."pubdate", NULL, NULL) AS "pubyear", SUM("aggregationregressbook"."pages") AS "totalpages", ( SELECT U0."rating" FROM "aggregationregressbook" U0 WHERE djangodateextract('year', U0."pubdate") = djangodatetrunc('year', "aggregationregressbook"."pubdate", NULL, NULL) ORDER BY U0."rating" ASC LIMIT 1 ) AS "toprating" FROM "aggregationregressbook" GROUP BY djangodatetrunc('year', "aggregationregressbook"."pubdate", NULL, NULL), "top_rating"


Question: # the selected fields anymore.             group_by = []             for expr in self.group_by:                if isinstance(expr, Ref) and expr.refs not in field_names:                if isinstance(expr, Ref) and expr.refs not in selected:                     expr = self.annotations[expr.refs]                 group_by.append(expr)             self.group_by = tuple(group_by)


Question: # the selected fields anymore.             group_by = []             for expr in self.group_by:                if isinstance(expr, Ref) and expr.refs not in field_names:                if isinstance(expr, Ref) and expr.refs not in selected:                     expr = self.annotations[expr.refs]                 group_by.append(expr)             self.group_by = tuple(group_by)


Question: for s_opt in parser._actions if s_opt.option_strings     }     arg_options = {opt_mapping.get(key, key): value for key, value in options.items()}    parse_args = [str(a) for a in args]    parse_args = []    for arg in args:        if isinstance(arg, (list, tuple)):            parse_args += map(str, arg)        else:            parse_args.append(str(arg))      def get_actions(parser):         # Parser actions and actions from sub-parser choices.

Support for list arguments inside exclusive required groups Description (last modified by Mark Gajdosik) Are there any plans to add support for the following? from import BaseCommand class Command(BaseCommand): def addarguments(self, parser): group = parser.addmutuallyexclusivegroup(required=True) group.addargument('--foo', nargs='+', type=int) def handle(self, args, *options): pass When calling the above using callcommand: callcommand('callcommandtest', foo=[1, 2, 3]) callcommand('callcommandtest', '--foo=1', '--foo=2', '--foo=3') I can't think of any workarounds other than setting type=str (somehow, that works fine) and coercing manually. Thank you!


Question: }     # Any required arguments which are passed in via **options must be passed     # to parse_args().    parse_args += [        min(opt.option_strings)        if isinstance(opt, (_AppendConstAction, _CountAction, _StoreConstAction))        else '{}={}'.format(min(opt.option_strings), arg_options[opt.dest])        for opt in parser_actions if (    for opt in parser_actions:        if (             opt.dest in options and             (opt.required or opt in mutually_exclusive_required_options)        )    ]        ):            parse_args.append(min(opt.option_strings))            if isinstance(opt, (_AppendConstAction, _CountAction, _StoreConstAction)):                continue            value = arg_options[opt.dest]            if isinstance(value, (list, tuple)):                parse_args += map(str, value)            else:                parse_args.append(str(value))     defaults = parser.parse_args(args=parse_args)     defaults = dict(defaults._get_kwargs(), **arg_options)     # Raise an error if any unknown options were passed.