Question: Template Cache "maketemplatefragmentkey" function speed up + simplify (also discussing switch to alternate hashes) Description (last modified by Daniel) The maketemplatefragmentkey function in has the following (minor) issues: Using for varyon args, is not needed any more - it was originally to make the unhashed strings safe to send to memcached and similar restricted systems. But since the value is hashed, this is now adding nothing. (See ). Use of the MD5 hashing function is disallowed on certain (odd) systems, not being FIPS compliant. See (). Creates a string of all joined varyon args to send to the hashing function, rather than using the hashlib .update() method. Here is a version solving these, switching to SHA256, and speeding up the function quite a bit:  And PR:  And here's the repo showing performance improvement:  Which seems to be faster in every case. The downside of this is that the cache key is now different from before. The tests have been updated to the new values. There are other cache key generating functions used in other places which use MD5 still - if switching to SHA256 it would make sense to me to change those at the same time, meaning only one time invalidating keys on upgrade. Thoughts?

import hashlibfrom urllib.parse import quote  TEMPLATE_FRAGMENT_KEY_TEMPLATE = 'template.cache.%s.%s'   def make_template_fragment_key(fragment_name, vary_on=None):    if vary_on is None:        vary_on = ()    key = ':'.join(quote(str(var)) for var in vary_on)    args = hashlib.md5(key.encode())    return TEMPLATE_FRAGMENT_KEY_TEMPLATE % (fragment_name, args.hexdigest())    hasher = hashlib.md5()    if vary_on is not None:        for arg in vary_on:            hasher.update(str(arg).encode())            hasher.update(b':')    return TEMPLATE_FRAGMENT_KEY_TEMPLATE % (fragment_name, hasher.hexdigest())


Question: AuthenticationForm's username field doesn't set maxlength HTML attribute. Description AuthenticationForm's username field doesn't render with maxlength HTML attribute anymore. Regression introduced in and . !topic/django-developers/qnfSqro0DlA

# Set the max length and label for the "username" field.         self.username_field = UserModel._meta.get_field(UserModel.USERNAME_FIELD)        self.fields['username'].max_length = self.username_field.max_length or 254        username_max_length = self.username_field.max_length or 254        self.fields['username'].max_length = username_max_length        self.fields['username'].widget.attrs['maxlength'] = username_max_length         if self.fields['username'].label is None:             self.fields['username'].label = capfirst(self.username_field.verbose_name)


Question: Filtering on query result overrides GROUP BY of internal query Description from import models a = (email__isnull=True).values('email').annotate(m=Max('id')).values('m') print(a.query) # good print(a[:1].query) # good b = (id=a[:1]) print(b.query) # GROUP BY U0."id" should be GROUP BY U0."email"

from django.db.models.sql.query import Query         if isinstance(self.rhs, Query):             if self.rhs.has_limit_one():                # The subquery must select only the pk.                self.rhs.clear_select_clause()                self.rhs.add_fields(['pk'])                if not self.rhs.has_select_fields:                    self.rhs.clear_select_clause()                    self.rhs.add_fields(['pk'])             else:                 raise ValueError(                     'The QuerySet value for an exact lookup must be limited to '


Question: Chaining selectrelated mutates original QuerySet. Description (last modified by Darren Maki) When creating a new QuerySet from an existing QuerySet that has had 'selectrelated' applied, if you apply another 'selectrelated' to the new QuerySet it will mutate the original QuerySet to also have the extra 'selectrelated'. from import models class ModelA(models.Model): pass class ModelB(models.Model): pass class ModelC(models.Model): modela = models.ForeignKey('foobar.ModelA', ondelete=models.CASCADE) modelb = models.ForeignKey('foobar.ModelB', ondelete=models.CASCADE) query1 = ModelC.objects.selectrelated('modela') print('QUERY 1:', str()) query2 = query1.selectrelated('modelb') print('QUERY 2:', str()) print('QUERY 1:', str()) if str() == str(): print('\n!!! The two queries are the same !!!\n') output QUERY 1: SELECT "foobarmodelc"."id", "foobarmodelc"."modelaid", "foobarmodelc"."modelbid", "foobarmodela"."id" FROM "foobarmodelc" INNER JOIN "foobarmodela" ON ("foobarmodelc"."modelaid" = "foobarmodela"."id") QUERY 2: SELECT "foobarmodelc"."id", "foobarmodelc"."modelaid", "foobarmodelc"."modelbid", "foobarmodela"."id", "foobarmodelb"."id" FROM "foobarmodelc" INNER JOIN "foobarmodela" ON ("foobarmodelc"."modelaid" = "foobarmodela"."id") INNER JOIN "foobarmodelb" ON ("foobarmodelc"."modelbid" = "foobarmodelb"."id") QUERY 1: SELECT "foobarmodelc"."id", "foobarmodelc"."modelaid", "foobarmodelc"."modelbid", "foobarmodela"."id", "foobarmodelb"."id" FROM "foobarmodelc" INNER JOIN "foobarmodela" ON ("foobarmodelc"."modelaid" = "foobarmodela"."id") INNER JOIN "foobarmodelb" ON ("foobarmodelc"."modelbid" = "foobarmodelb"."id") !!! The two queries are the same !!! The expectation is that the original QuerySet is not mutated, and the two queries are different. This behavior also happens with 'prefetchrelated'. Since the QuerySet methods call '()', and state that they return 'a new QuerySet instance', this behavior does not seem correct.

databases). The abstraction barrier only works one way: this module has to know all about the internals of models in order to get the information it needs. """<ins>import copy</ins> import difflib import functools import inspect


Question: Chaining selectrelated mutates original QuerySet. Description (last modified by Darren Maki) When creating a new QuerySet from an existing QuerySet that has had 'selectrelated' applied, if you apply another 'selectrelated' to the new QuerySet it will mutate the original QuerySet to also have the extra 'selectrelated'. from import models class ModelA(models.Model): pass class ModelB(models.Model): pass class ModelC(models.Model): modela = models.ForeignKey('foobar.ModelA', ondelete=models.CASCADE) modelb = models.ForeignKey('foobar.ModelB', ondelete=models.CASCADE) query1 = ModelC.objects.selectrelated('modela') print('QUERY 1:', str()) query2 = query1.selectrelated('modelb') print('QUERY 2:', str()) print('QUERY 1:', str()) if str() == str(): print('\n!!! The two queries are the same !!!\n') output QUERY 1: SELECT "foobarmodelc"."id", "foobarmodelc"."modelaid", "foobarmodelc"."modelbid", "foobarmodela"."id" FROM "foobarmodelc" INNER JOIN "foobarmodela" ON ("foobarmodelc"."modelaid" = "foobarmodela"."id") QUERY 2: SELECT "foobarmodelc"."id", "foobarmodelc"."modelaid", "foobarmodelc"."modelbid", "foobarmodela"."id", "foobarmodelb"."id" FROM "foobarmodelc" INNER JOIN "foobarmodela" ON ("foobarmodelc"."modelaid" = "foobarmodela"."id") INNER JOIN "foobarmodelb" ON ("foobarmodelc"."modelbid" = "foobarmodelb"."id") QUERY 1: SELECT "foobarmodelc"."id", "foobarmodelc"."modelaid", "foobarmodelc"."modelbid", "foobarmodela"."id", "foobarmodelb"."id" FROM "foobarmodelc" INNER JOIN "foobarmodela" ON ("foobarmodelc"."modelaid" = "foobarmodela"."id") INNER JOIN "foobarmodelb" ON ("foobarmodelc"."modelbid" = "foobarmodelb"."id") !!! The two queries are the same !!! The expectation is that the original QuerySet is not mutated, and the two queries are different. This behavior also happens with 'prefetchrelated'. Since the QuerySet methods call '()', and state that they return 'a new QuerySet instance', this behavior does not seem correct.

obj._extra_select_cache = None         else:             obj._extra_select_cache = self._extra_select_cache.copy()<ins>        if self.select_related is not False:            # Use deepcopy because select_related stores fields in nested            # dicts.            obj.select_related = copy.deepcopy(obj.select_related)</ins>         if 'subq_aliases' in self.__dict__:             obj.subq_aliases = self.subq_aliases.copy()         obj.used_aliases = self.used_aliases.copy()


Question: Migrations uses value of enum object instead of its name. Description (last modified by oasl) When using Enum object as a default value for a CharField, the generated migration file uses the value of the Enum object instead of the its name. This causes a problem when using Django translation on the value of the Enum object. The problem is that, when the Enum object value get translated to the users language, the old migration files raise an error stating that the Enum does not have the corresponding value. (because the Enum value is translated to another language) Example: Let say we have this code in from enum import Enum from import gettextlazy as from import models class Status(Enum): GOOD = ('Good') # 'Good' will be translated BAD = ('Bad') # 'Bad' will be translated def str(self): return class Item(models.Model): status = models.CharField(default=Status.GOOD, maxlength=128) In the generated migration file, the code will be: ... ('status', models.CharField(default=Status('Good'), maxlength=128)) ... After the translation, 'Good' will be translated to another word and it will not be part of the Status Enum class any more, so the migration file will raise the error on the previous line: ValueError: 'Good' is not a valid Status Shouldn't the code generated by the migration uses the name of the Status Enum 'GOOD', not the value of it, since it is changeable? It should be: ('status', models.CharField(default=Status['GOOD'], max_length=128)) This will be correct regardless of the translated word

def serialize(self):         enum_class = self.value.__class__         module = enum_class.__module__        v_string, v_imports = serializer_factory(self.value.value).serialize()        imports = {'import %s' % module, *v_imports}        return "%s.%s(%s)" % (module, enum_class.__name__, v_string), imports        return (            '%s.%s[%r]' % (module, enum_class.__name__, self.value.name),            {'import %s' % module},        )   class FloatSerializer(BaseSimpleSerializer):


Question: models.E015 is raised when Meta.ordering contains "pk" of a related field. Description models.E015 is raised when Meta.ordering contains pk of a related field, e.g.: test_app.SomeModel: (models.E015) 'ordering' refers to the nonexistent field, related field, or lookup 'optionpk'. Regression in .

fld = None             for part in field.split(LOOKUP_SEP):                 try:                    fld = _cls._meta.get_field(part)                    # pk is an alias that won't be found by opts.get_field.                    if part == 'pk':                        fld = _cls._meta.pk                    else:                        fld = _cls._meta.get_field(part)                     if fld.is_relation:                         _cls = fld.get_path_info()[-1].to_opts.model                    else:                        _cls = None                 except (FieldDoesNotExist, AttributeError):                     if fld is None or fld.get_transform(part) is None:                         errors.append(


Question: cachecontrol() "maxage" overrides cachepage() "timeout" Description If you decorate a view with both cachecontrol(maxage=3600) and cachepage(timeout=), the server side cache uses the maxage value instead of the timeout value. The comments in UpdateCacheMiddleware.processresponse() indicate it's trying to set the timeout by first looking for the max-age header before reverting to the default cachetimeout : Try to get the timeout from the "max-age" section of the "Cache- Control" header before reverting to using the default cachetimeout length. However, cachepage(timeout=) is explicitly setting the cachetimeout so that is what should be used. In summary, if a user wants the client-side cache to be shorter-lived than the server-side cache, it is currently not possible using these two decorators.

"""     def __init__(self, get_response=None):         self.cache_timeout = settings.CACHE_MIDDLEWARE_SECONDS<ins>        self.page_timeout = None</ins>         self.key_prefix = settings.CACHE_MIDDLEWARE_KEY_PREFIX         self.cache_alias = settings.CACHE_MIDDLEWARE_ALIAS         self.cache = caches[self.cache_alias]


Question: cachecontrol() "maxage" overrides cachepage() "timeout" Description If you decorate a view with both cachecontrol(maxage=3600) and cachepage(timeout=), the server side cache uses the maxage value instead of the timeout value. The comments in UpdateCacheMiddleware.processresponse() indicate it's trying to set the timeout by first looking for the max-age header before reverting to the default cachetimeout : Try to get the timeout from the "max-age" section of the "Cache- Control" header before reverting to using the default cachetimeout length. However, cachepage(timeout=) is explicitly setting the cachetimeout so that is what should be used. In summary, if a user wants the client-side cache to be shorter-lived than the server-side cache, it is currently not possible using these two decorators.

if 'private' in response.get('Cache-Control', ()):             return response         # Try to get the timeout from the "max-age" section of the "Cache-        # Control" header before reverting to using the default cache_timeout        # length.        timeout = get_max_age(response)        # Page timeout takes precedence over the "max-age" and the default        # cache timeout.        timeout = self.page_timeout         if timeout is None:            timeout = self.cache_timeout        elif timeout == 0:            # max-age was set to 0, don't bother caching.            return response            # The timeout from the "max-age" section of the "Cache-Control"            # header takes precedence over the default cache timeout.            timeout = get_max_age(response)            if timeout is None:                timeout = self.cache_timeout            elif timeout == 0:                # max-age was set to 0, don't cache.                return response         patch_response_headers(response, timeout)         if timeout and response.status_code == 200:             cache_key = learn_cache_key(request, response, timeout, self.key_prefix, cache=self.cache)


Question: cachecontrol() "maxage" overrides cachepage() "timeout" Description If you decorate a view with both cachecontrol(maxage=3600) and cachepage(timeout=), the server side cache uses the maxage value instead of the timeout value. The comments in UpdateCacheMiddleware.processresponse() indicate it's trying to set the timeout by first looking for the max-age header before reverting to the default cachetimeout : Try to get the timeout from the "max-age" section of the "Cache- Control" header before reverting to using the default cachetimeout length. However, cachepage(timeout=) is explicitly setting the cachetimeout so that is what should be used. In summary, if a user wants the client-side cache to be shorter-lived than the server-side cache, it is currently not possible using these two decorators.

Also used as the hook point for the cache decorator, which is generated     using the decorator-from-middleware utility.     """    def __init__(self, get_response=None, cache_timeout=None, **kwargs):    def __init__(self, get_response=None, cache_timeout=None, page_timeout=None, **kwargs):         self.get_response = get_response         # We need to differentiate between "provided, but using default value",         # and "not provided". If the value is provided using a default, then


Question: cachecontrol() "maxage" overrides cachepage() "timeout" Description If you decorate a view with both cachecontrol(maxage=3600) and cachepage(timeout=), the server side cache uses the maxage value instead of the timeout value. The comments in UpdateCacheMiddleware.processresponse() indicate it's trying to set the timeout by first looking for the max-age header before reverting to the default cachetimeout : Try to get the timeout from the "max-age" section of the "Cache- Control" header before reverting to using the default cachetimeout length. However, cachepage(timeout=) is explicitly setting the cachetimeout so that is what should be used. In summary, if a user wants the client-side cache to be shorter-lived than the server-side cache, it is currently not possible using these two decorators.

if cache_timeout is None:             cache_timeout = settings.CACHE_MIDDLEWARE_SECONDS         self.cache_timeout = cache_timeout        self.page_timeout = page_timeout         self.cache = caches[self.cache_alias]diff --git a/django/views/decorators/cache.py b/django/views/decorators/cache.py-- a/django/views/decorators/cache.py++ b/django/views/decorators/cache.py


Question: cachecontrol() "maxage" overrides cachepage() "timeout" Description If you decorate a view with both cachecontrol(maxage=3600) and cachepage(timeout=), the server side cache uses the maxage value instead of the timeout value. The comments in UpdateCacheMiddleware.processresponse() indicate it's trying to set the timeout by first looking for the max-age header before reverting to the default cachetimeout : Try to get the timeout from the "max-age" section of the "Cache- Control" header before reverting to using the default cachetimeout length. However, cachepage(timeout=) is explicitly setting the cachetimeout so that is what should be used. In summary, if a user wants the client-side cache to be shorter-lived than the server-side cache, it is currently not possible using these two decorators.

into account on caching -- just like the middleware does.     """     return decorator_from_middleware_with_args(CacheMiddleware)(        cache_timeout=timeout, cache_alias=cache, key_prefix=key_prefix        page_timeout=timeout, cache_alias=cache, key_prefix=key_prefix,     )


Question: patchcachecontrol should special case "no-cache" Description (last modified by Tim Graham) From my cursory reading of , it looks like patchcachecontrol needs to special case "no-cache". no-cache If the no-cache directive does not specify a field-name, then a cache MUST NOT use the response to satisfy a subsequent request without successful revalidation with the origin server. This allows an origin server to prevent caching even by caches that have been configured to return stale responses to client requests. If the no-cache directive does specify one or more field-names, then a cache MAY use the response to satisfy a subsequent request, subject to any other restrictions on caching. However, the specified field-name(s) MUST NOT be sent in the response to a subsequent request without successful revalidation with the origin server. This allows an origin server to prevent the re-use of certain header fields in a response, while still allowing caching of the rest of the response. For example, to integrate a site that uses "Vary: Cookie" with AWS CloudFront, one must use 'Cache-Control: no-cache="Set-Cookie"' if a response does not vary by cookie. (I've confirmed this with AWS support as of 10/31/2014). patchcachecontrol does not treat "no-cache" as a list. If you call patchcachecontrol(response, nocache="Set-Cookie") and then patchcachecontrol(response, nocache="foo"), you end up with 'Cache-Control: no-cache="foo"' Also, nocache=True should take precedence over nocache="foo" regardless of the order it is applied. I found Ticket which proposes to add "no-cache" to Just wanted to link it here since they are related.

import hashlib import re import time<ins>from collections import defaultdict</ins>  from django.conf import settings from django.core.cache import caches


Question: patchcachecontrol should special case "no-cache" Description (last modified by Tim Graham) From my cursory reading of , it looks like patchcachecontrol needs to special case "no-cache". no-cache If the no-cache directive does not specify a field-name, then a cache MUST NOT use the response to satisfy a subsequent request without successful revalidation with the origin server. This allows an origin server to prevent caching even by caches that have been configured to return stale responses to client requests. If the no-cache directive does specify one or more field-names, then a cache MAY use the response to satisfy a subsequent request, subject to any other restrictions on caching. However, the specified field-name(s) MUST NOT be sent in the response to a subsequent request without successful revalidation with the origin server. This allows an origin server to prevent the re-use of certain header fields in a response, while still allowing caching of the rest of the response. For example, to integrate a site that uses "Vary: Cookie" with AWS CloudFront, one must use 'Cache-Control: no-cache="Set-Cookie"' if a response does not vary by cookie. (I've confirmed this with AWS support as of 10/31/2014). patchcachecontrol does not treat "no-cache" as a list. If you call patchcachecontrol(response, nocache="Set-Cookie") and then patchcachecontrol(response, nocache="foo"), you end up with 'Cache-Control: no-cache="foo"' Also, nocache=True should take precedence over nocache="foo" regardless of the order it is applied. I found Ticket which proposes to add "no-cache" to Just wanted to link it here since they are related.

else:             return (t[0].lower(), True)     def dictvalue(t):    def dictvalue(*t):         if t[1] is True:             return t[0]         else:             return '%s=%s' % (t[0], t[1])     cc = defaultdict(set)     if response.get('Cache-Control'):        cc = cc_delim_re.split(response['Cache-Control'])        cc = dict(dictitem(el) for el in cc)    else:        cc = {}        for field in cc_delim_re.split(response['Cache-Control']):            directive, value = dictitem(field)            if directive == 'no-cache':                # no-cache supports multiple field names.                cc[directive].add(value)            else:                cc[directive] = value      # If there's already a max-age header but we're being asked to set a new     # max-age, use the minimum of the two ages. In practice this happens when


Question: patchcachecontrol should special case "no-cache" Description (last modified by Tim Graham) From my cursory reading of , it looks like patchcachecontrol needs to special case "no-cache". no-cache If the no-cache directive does not specify a field-name, then a cache MUST NOT use the response to satisfy a subsequent request without successful revalidation with the origin server. This allows an origin server to prevent caching even by caches that have been configured to return stale responses to client requests. If the no-cache directive does specify one or more field-names, then a cache MAY use the response to satisfy a subsequent request, subject to any other restrictions on caching. However, the specified field-name(s) MUST NOT be sent in the response to a subsequent request without successful revalidation with the origin server. This allows an origin server to prevent the re-use of certain header fields in a response, while still allowing caching of the rest of the response. For example, to integrate a site that uses "Vary: Cookie" with AWS CloudFront, one must use 'Cache-Control: no-cache="Set-Cookie"' if a response does not vary by cookie. (I've confirmed this with AWS support as of 10/31/2014). patchcachecontrol does not treat "no-cache" as a list. If you call patchcachecontrol(response, nocache="Set-Cookie") and then patchcachecontrol(response, nocache="foo"), you end up with 'Cache-Control: no-cache="foo"' Also, nocache=True should take precedence over nocache="foo" regardless of the order it is applied. I found Ticket which proposes to add "no-cache" to Just wanted to link it here since they are related.

del cc['public']      for (k, v) in kwargs.items():        cc[k.replace('_', '-')] = v    cc = ', '.join(dictvalue(el) for el in cc.items())        directive = k.replace('_', '-')        if directive == 'no-cache':            # no-cache supports multiple field names.            cc[directive].add(v)        else:            cc[directive] = v    directives = []    for directive, values in cc.items():        if isinstance(values, set):            if True in values:                # True takes precedence.                values = {True}            directives.extend([dictvalue(directive, value) for value in values])        else:            directives.append(dictvalue(directive, values))    cc = ', '.join(directives)     response['Cache-Control'] = cc


Question: two digit year check is incorrect Description (last modified by Ad Timmering) RFC 850 does not mention this, but in RFC 7231 (and there's something similar in RFC 2822), there's the following quote: Recipients of a timestamp value in rfc850-date format, which uses a two-digit year, MUST interpret a timestamp that appears to be more than 50 years in the future as representing the most recent year in the past that had the same last two digits. Current logic is hard coded to consider 0-69 to be in 2000-2069, and 70-99 to be 1970-1999, instead of comparing versus the current year.

try:         year = int(m.group('year'))         if year < 100:            if year < 70:                year += 2000            current_year = datetime.datetime.utcnow().year            current_century = current_year - (current_year % 100)            if year - (current_year % 100) > 50:                # year that appears to be more than 50 years in the future are                # interpreted as representing the past.                year += current_century - 100             else:                year += 1900                year += current_century         month = MONTHS.index(m.group('mon').lower()) + 1         day = int(m.group('day'))         hour = int(m.group('hour'))


Question: Form Field’s deepcopy does not (deep)copy the error messages. Description The deepcopy method defined for the formfields () performs a shallow copy of self and does not include additional treatment for the error_messages dictionary. As a result, all copies of the same field share the same dictionary and any modification of either the dictionary or the error message itself for one formfield is immediately reflected on all other formfiels. This is relevant for Forms and ModelForms that modify the error messages of their fields dynamically: while each instance of the specific form (e.g., ProfileForm) is expected to have a set of fields “sealed” away from other instances of the same ProfileForm (), in fact all these instances share the same error messages, resulting in incorrectly raised errors. Confirmed for versions of Django going back to 1.11.

result = copy.copy(self)         memo[id(self)] = result         result.widget = copy.deepcopy(self.widget, memo)<ins>        result.error_messages = self.error_messages.copy()</ins>         result.validators = self.validators[:]         return result


Question: Make () return whether or not it suceeded. Description It can be quite useful when dealing with complex caching/locking systems or simply for logging purposes. Memcache clients already returns this value and it should be straigtforward to implement for file, inmemory, and database backend based on the number of returned rows. Redis del operation also returns the number of keys it successfully deleted so it should be implementable if it's eventually as discussed on the mailing list.

def delete(self, key, version=None):         """        Delete a key from the cache, failing silently.        Delete a key from the cache and return whether it succeeded, failing        silently.         """         raise NotImplementedError('subclasses of BaseCache must provide a delete() method') diff --git a/django/core/cache/backends/db.py b/django/core/cache/backends/db.py-- a/django/core/cache/backends/db.py++ b/django/core/cache/backends/db.py


Question: Make () return whether or not it suceeded. Description It can be quite useful when dealing with complex caching/locking systems or simply for logging purposes. Memcache clients already returns this value and it should be straigtforward to implement for file, inmemory, and database backend based on the number of returned rows. Redis del operation also returns the number of keys it successfully deleted so it should be implementable if it's eventually as discussed on the mailing list.

return True      def delete(self, key, version=None):        self.delete_many([key], version)        self.validate_key(key)        return self._base_delete_many([self.make_key(key, version)])      def delete_many(self, keys, version=None):         key_list = []


Question: Make () return whether or not it suceeded. Description It can be quite useful when dealing with complex caching/locking systems or simply for logging purposes. Memcache clients already returns this value and it should be straigtforward to implement for file, inmemory, and database backend based on the number of returned rows. Redis del operation also returns the number of keys it successfully deleted so it should be implementable if it's eventually as discussed on the mailing list.

def _base_delete_many(self, keys):         if not keys:            return            return False          db = router.db_for_write(self.cache_model_class)         connection = connections[db]


Question: Make () return whether or not it suceeded. Description It can be quite useful when dealing with complex caching/locking systems or simply for logging purposes. Memcache clients already returns this value and it should be straigtforward to implement for file, inmemory, and database backend based on the number of returned rows. Redis del operation also returns the number of keys it successfully deleted so it should be implementable if it's eventually as discussed on the mailing list.

),                 keys,             )        return bool(cursor.rowcount)      def has_key(self, key, version=None):         key = self.make_key(key, version=version)diff --git a/django/core/cache/backends/dummy.py b/django/core/cache/backends/dummy.py-- a/django/core/cache/backends/dummy.py++ b/django/core/cache/backends/dummy.py


Question: Make () return whether or not it suceeded. Description It can be quite useful when dealing with complex caching/locking systems or simply for logging purposes. Memcache clients already returns this value and it should be straigtforward to implement for file, inmemory, and database backend based on the number of returned rows. Redis del operation also returns the number of keys it successfully deleted so it should be implementable if it's eventually as discussed on the mailing list.

def delete(self, key, version=None):         key = self.make_key(key, version=version)         self.validate_key(key)        return False      def has_key(self, key, version=None):         key = self.make_key(key, version=version)diff --git a/django/core/cache/backends/filebased.py b/django/core/cache/backends/filebased.py-- a/django/core/cache/backends/filebased.py++ b/django/core/cache/backends/filebased.py


Question: Make () return whether or not it suceeded. Description It can be quite useful when dealing with complex caching/locking systems or simply for logging purposes. Memcache clients already returns this value and it should be straigtforward to implement for file, inmemory, and database backend based on the number of returned rows. Redis del operation also returns the number of keys it successfully deleted so it should be implementable if it's eventually as discussed on the mailing list.

return False      def delete(self, key, version=None):        self._delete(self._key_to_file(key, version))        return self._delete(self._key_to_file(key, version))      def _delete(self, fname):         if not fname.startswith(self._dir) or not os.path.exists(fname):            return            return False         try:             os.remove(fname)         except FileNotFoundError:             # The file may have been removed by another process.            pass            return False        return True      def has_key(self, key, version=None):         fname = self._key_to_file(key, version)diff --git a/django/core/cache/backends/locmem.py b/django/core/cache/backends/locmem.py-- a/django/core/cache/backends/locmem.py++ b/django/core/cache/backends/locmem.py


Question: Make () return whether or not it suceeded. Description It can be quite useful when dealing with complex caching/locking systems or simply for logging purposes. Memcache clients already returns this value and it should be straigtforward to implement for file, inmemory, and database backend based on the number of returned rows. Redis del operation also returns the number of keys it successfully deleted so it should be implementable if it's eventually as discussed on the mailing list.

del self._cache[key]             del self._expire_info[key]         except KeyError:            pass            return False        return True      def delete(self, key, version=None):         key = self.make_key(key, version=version)         self.validate_key(key)         with self._lock:            self._delete(key)            return self._delete(key)      def clear(self):         with self._lock:diff --git a/django/core/cache/backends/memcached.py b/django/core/cache/backends/memcached.py-- a/django/core/cache/backends/memcached.py++ b/django/core/cache/backends/memcached.py


Question: Make () return whether or not it suceeded. Description It can be quite useful when dealing with complex caching/locking systems or simply for logging purposes. Memcache clients already returns this value and it should be straigtforward to implement for file, inmemory, and database backend based on the number of returned rows. Redis del operation also returns the number of keys it successfully deleted so it should be implementable if it's eventually as discussed on the mailing list.

def delete(self, key, version=None):         key = self.make_key(key, version=version)        self._cache.delete(key)        return bool(self._cache.delete(key))      def get_many(self, keys, version=None):         key_map = {self.make_key(key, version=version): key for key in keys}


Question: Make () return whether or not it suceeded. Description It can be quite useful when dealing with complex caching/locking systems or simply for logging purposes. Memcache clients already returns this value and it should be straigtforward to implement for file, inmemory, and database backend based on the number of returned rows. Redis del operation also returns the number of keys it successfully deleted so it should be implementable if it's eventually as discussed on the mailing list.

return default         return val <ins>    def delete(self, key, version=None):        # python-memcached's delete() returns True when key doesn't exist.        # https://github.com/linsomniac/python-memcached/issues/170        # Call _deletetouch() without the NOT_FOUND in expected results.        key = self.make_key(key, version=version)        return bool(self._cache._deletetouch([b'DELETED'], 'delete', key))</ins>  class PyLibMCCache(BaseMemcachedCache):     "An implementation of a cache binding using pylibmc"


Question: Combine fast delete queries Description When emulating ON DELETE CASCADE via ondelete=models.CASCADE the deletion.Collector will try to perform fast queries which are DELETE FROM table WHERE IN .... There's a few conditions required for this fast path to be taken but when this happens the collection logic should combine such queries by table to reduce the number of roundtrips to the database. For example, given the following models class Person(models.Model): friends = models.ManyToManyField('self') class User(models.Model): pass class Entry(models.Model): createdby = models.ForeignKey(User) updatedby = models.ForeignKey(User) Issuing a () or () will result in 3 queries of the form DELETE FROM personfriends WHERE fromid = :id DELETE FROM personfriends WHERE toid = :id DELETE FROM person WHERE id = :id DELETE FROM entry WHERE createdbyid = :id DELETE FROM entry WHERE updatedby = :id DELETRE FROM user WHERE id = :id But both queries (or N queries depending on the number of foreign relationships) can be combined into a single one by using OR DELETE FROM personfriends WHERE fromid = :id OR toid = :id DELETE FROM person WHERE id = :id DELETE FROM entry WHERE createdbyid = :id OR updatedby = :id DELETE FROM user WHERE id = :id

except models.ProtectedError as e:             self.protected.update(e.protected_objects)     def related_objects(self, related, objs):        qs = super().related_objects(related, objs)        return qs.select_related(related.field.name)    def related_objects(self, related_model, related_fields, objs):        qs = super().related_objects(related_model, related_fields, objs)        return qs.select_related(*[related_field.name for related_field in related_fields])      def _nested(self, obj, seen, format_callback):         if obj in seen:diff --git a/django/db/models/deletion.py b/django/db/models/deletion.py-- a/django/db/models/deletion.py++ b/django/db/models/deletion.py


Question: Combine fast delete queries Description When emulating ON DELETE CASCADE via ondelete=models.CASCADE the deletion.Collector will try to perform fast queries which are DELETE FROM table WHERE IN .... There's a few conditions required for this fast path to be taken but when this happens the collection logic should combine such queries by table to reduce the number of roundtrips to the database. For example, given the following models class Person(models.Model): friends = models.ManyToManyField('self') class User(models.Model): pass class Entry(models.Model): createdby = models.ForeignKey(User) updatedby = models.ForeignKey(User) Issuing a () or () will result in 3 queries of the form DELETE FROM personfriends WHERE fromid = :id DELETE FROM personfriends WHERE toid = :id DELETE FROM person WHERE id = :id DELETE FROM entry WHERE createdbyid = :id DELETE FROM entry WHERE updatedby = :id DELETRE FROM user WHERE id = :id But both queries (or N queries depending on the number of foreign relationships) can be combined into a single one by using OR DELETE FROM personfriends WHERE fromid = :id OR toid = :id DELETE FROM person WHERE id = :id DELETE FROM entry WHERE createdbyid = :id OR updatedby = :id DELETE FROM user WHERE id = :id

from collections import Counterimport operatorfrom collections import Counter, defaultdictfrom functools import partial, reduce from itertools import chain from operator import attrgetter  from django.db import IntegrityError, connections, transactionfrom django.db.models import signals, sqlfrom django.db.models import query_utils, signals, sql   class ProtectedError(IntegrityError):


Question: Combine fast delete queries Description When emulating ON DELETE CASCADE via ondelete=models.CASCADE the deletion.Collector will try to perform fast queries which are DELETE FROM table WHERE IN .... There's a few conditions required for this fast path to be taken but when this happens the collection logic should combine such queries by table to reduce the number of roundtrips to the database. For example, given the following models class Person(models.Model): friends = models.ManyToManyField('self') class User(models.Model): pass class Entry(models.Model): createdby = models.ForeignKey(User) updatedby = models.ForeignKey(User) Issuing a () or () will result in 3 queries of the form DELETE FROM personfriends WHERE fromid = :id DELETE FROM personfriends WHERE toid = :id DELETE FROM person WHERE id = :id DELETE FROM entry WHERE createdbyid = :id DELETE FROM entry WHERE updatedby = :id DELETRE FROM user WHERE id = :id But both queries (or N queries depending on the number of foreign relationships) can be combined into a single one by using OR DELETE FROM personfriends WHERE fromid = :id OR toid = :id DELETE FROM person WHERE id = :id DELETE FROM entry WHERE createdbyid = :id OR updatedby = :id DELETE FROM user WHERE id = :id

def __init__(self, using):         self.using = using         # Initially, {model: {instances}}, later values become lists.        self.data = {}        self.field_updates = {}  # {model: {(field, value): {instances}}}        self.data = defaultdict(set)        # {model: {(field, value): {instances}}}        self.field_updates = defaultdict(partial(defaultdict, set))         # fast_deletes is a list of queryset-likes that can be deleted without         # fetching the objects into memory.         self.fast_deletes = []


Question: Combine fast delete queries Description When emulating ON DELETE CASCADE via ondelete=models.CASCADE the deletion.Collector will try to perform fast queries which are DELETE FROM table WHERE IN .... There's a few conditions required for this fast path to be taken but when this happens the collection logic should combine such queries by table to reduce the number of roundtrips to the database. For example, given the following models class Person(models.Model): friends = models.ManyToManyField('self') class User(models.Model): pass class Entry(models.Model): createdby = models.ForeignKey(User) updatedby = models.ForeignKey(User) Issuing a () or () will result in 3 queries of the form DELETE FROM personfriends WHERE fromid = :id DELETE FROM personfriends WHERE toid = :id DELETE FROM person WHERE id = :id DELETE FROM entry WHERE createdbyid = :id DELETE FROM entry WHERE updatedby = :id DELETRE FROM user WHERE id = :id But both queries (or N queries depending on the number of foreign relationships) can be combined into a single one by using OR DELETE FROM personfriends WHERE fromid = :id OR toid = :id DELETE FROM person WHERE id = :id DELETE FROM entry WHERE createdbyid = :id OR updatedby = :id DELETE FROM user WHERE id = :id

# should be included, as the dependencies exist only between actual         # database tables; proxy models are represented here by their concrete         # parent.        self.dependencies = {}  # {model: {models}}        self.dependencies = defaultdict(set)  # {model: {models}}      def add(self, objs, source=None, nullable=False, reverse_dependency=False):         """


Question: Combine fast delete queries Description When emulating ON DELETE CASCADE via ondelete=models.CASCADE the deletion.Collector will try to perform fast queries which are DELETE FROM table WHERE IN .... There's a few conditions required for this fast path to be taken but when this happens the collection logic should combine such queries by table to reduce the number of roundtrips to the database. For example, given the following models class Person(models.Model): friends = models.ManyToManyField('self') class User(models.Model): pass class Entry(models.Model): createdby = models.ForeignKey(User) updatedby = models.ForeignKey(User) Issuing a () or () will result in 3 queries of the form DELETE FROM personfriends WHERE fromid = :id DELETE FROM personfriends WHERE toid = :id DELETE FROM person WHERE id = :id DELETE FROM entry WHERE createdbyid = :id DELETE FROM entry WHERE updatedby = :id DELETRE FROM user WHERE id = :id But both queries (or N queries depending on the number of foreign relationships) can be combined into a single one by using OR DELETE FROM personfriends WHERE fromid = :id OR toid = :id DELETE FROM person WHERE id = :id DELETE FROM entry WHERE createdbyid = :id OR updatedby = :id DELETE FROM user WHERE id = :id

return []         new_objs = []         model = objs[0].__class__        instances = self.data.setdefault(model, set())        instances = self.data[model]         for obj in objs:             if obj not in instances:                 new_objs.append(obj)


Question: Combine fast delete queries Description When emulating ON DELETE CASCADE via ondelete=models.CASCADE the deletion.Collector will try to perform fast queries which are DELETE FROM table WHERE IN .... There's a few conditions required for this fast path to be taken but when this happens the collection logic should combine such queries by table to reduce the number of roundtrips to the database. For example, given the following models class Person(models.Model): friends = models.ManyToManyField('self') class User(models.Model): pass class Entry(models.Model): createdby = models.ForeignKey(User) updatedby = models.ForeignKey(User) Issuing a () or () will result in 3 queries of the form DELETE FROM personfriends WHERE fromid = :id DELETE FROM personfriends WHERE toid = :id DELETE FROM person WHERE id = :id DELETE FROM entry WHERE createdbyid = :id DELETE FROM entry WHERE updatedby = :id DELETRE FROM user WHERE id = :id But both queries (or N queries depending on the number of foreign relationships) can be combined into a single one by using OR DELETE FROM personfriends WHERE fromid = :id OR toid = :id DELETE FROM person WHERE id = :id DELETE FROM entry WHERE createdbyid = :id OR updatedby = :id DELETE FROM user WHERE id = :id

if source is not None and not nullable:             if reverse_dependency:                 source, model = model, source            self.dependencies.setdefault(                source._meta.concrete_model, set()).add(model._meta.concrete_model)            self.dependencies[source._meta.concrete_model].add(model._meta.concrete_model)         return new_objs      def add_field_update(self, field, value, objs):


Question: Combine fast delete queries Description When emulating ON DELETE CASCADE via ondelete=models.CASCADE the deletion.Collector will try to perform fast queries which are DELETE FROM table WHERE IN .... There's a few conditions required for this fast path to be taken but when this happens the collection logic should combine such queries by table to reduce the number of roundtrips to the database. For example, given the following models class Person(models.Model): friends = models.ManyToManyField('self') class User(models.Model): pass class Entry(models.Model): createdby = models.ForeignKey(User) updatedby = models.ForeignKey(User) Issuing a () or () will result in 3 queries of the form DELETE FROM personfriends WHERE fromid = :id DELETE FROM personfriends WHERE toid = :id DELETE FROM person WHERE id = :id DELETE FROM entry WHERE createdbyid = :id DELETE FROM entry WHERE updatedby = :id DELETRE FROM user WHERE id = :id But both queries (or N queries depending on the number of foreign relationships) can be combined into a single one by using OR DELETE FROM personfriends WHERE fromid = :id OR toid = :id DELETE FROM person WHERE id = :id DELETE FROM entry WHERE createdbyid = :id OR updatedby = :id DELETE FROM user WHERE id = :id

if not objs:             return         model = objs[0].__class__        self.field_updates.setdefault(            model, {}).setdefault(            (field, value), set()).update(objs)        self.field_updates[model][field, value].update(objs)      def _has_signal_listeners(self, model):         return (


Question: Combine fast delete queries Description When emulating ON DELETE CASCADE via ondelete=models.CASCADE the deletion.Collector will try to perform fast queries which are DELETE FROM table WHERE IN .... There's a few conditions required for this fast path to be taken but when this happens the collection logic should combine such queries by table to reduce the number of roundtrips to the database. For example, given the following models class Person(models.Model): friends = models.ManyToManyField('self') class User(models.Model): pass class Entry(models.Model): createdby = models.ForeignKey(User) updatedby = models.ForeignKey(User) Issuing a () or () will result in 3 queries of the form DELETE FROM personfriends WHERE fromid = :id DELETE FROM personfriends WHERE toid = :id DELETE FROM person WHERE id = :id DELETE FROM entry WHERE createdbyid = :id DELETE FROM entry WHERE updatedby = :id DELETRE FROM user WHERE id = :id But both queries (or N queries depending on the number of foreign relationships) can be combined into a single one by using OR DELETE FROM personfriends WHERE fromid = :id OR toid = :id DELETE FROM person WHERE id = :id DELETE FROM entry WHERE createdbyid = :id OR updatedby = :id DELETE FROM user WHERE id = :id

if from_field and from_field.remote_field.on_delete is not CASCADE:             return False         if hasattr(objs, '_meta'):            model = type(objs)            model = objs._meta.model         elif hasattr(objs, 'model') and hasattr(objs, '_raw_delete'):             model = objs.model         else:


Question: Combine fast delete queries Description When emulating ON DELETE CASCADE via ondelete=models.CASCADE the deletion.Collector will try to perform fast queries which are DELETE FROM table WHERE IN .... There's a few conditions required for this fast path to be taken but when this happens the collection logic should combine such queries by table to reduce the number of roundtrips to the database. For example, given the following models class Person(models.Model): friends = models.ManyToManyField('self') class User(models.Model): pass class Entry(models.Model): createdby = models.ForeignKey(User) updatedby = models.ForeignKey(User) Issuing a () or () will result in 3 queries of the form DELETE FROM personfriends WHERE fromid = :id DELETE FROM personfriends WHERE toid = :id DELETE FROM person WHERE id = :id DELETE FROM entry WHERE createdbyid = :id DELETE FROM entry WHERE updatedby = :id DELETRE FROM user WHERE id = :id But both queries (or N queries depending on the number of foreign relationships) can be combined into a single one by using OR DELETE FROM personfriends WHERE fromid = :id OR toid = :id DELETE FROM person WHERE id = :id DELETE FROM entry WHERE createdbyid = :id OR updatedby = :id DELETE FROM user WHERE id = :id

)         )     def get_del_batches(self, objs, field):    def get_del_batches(self, objs, fields):         """         Return the objs in suitably sized batches for the used connection.         """        field_names = [field.name for field in fields]         conn_batch_size = max(            connections[self.using].ops.bulk_batch_size([field.name], objs), 1)            connections[self.using].ops.bulk_batch_size(field_names, objs), 1)         if len(objs) > conn_batch_size:             return [objs[i:i + conn_batch_size]                     for i in range(0, len(objs), conn_batch_size)]


Question: Combine fast delete queries Description When emulating ON DELETE CASCADE via ondelete=models.CASCADE the deletion.Collector will try to perform fast queries which are DELETE FROM table WHERE IN .... There's a few conditions required for this fast path to be taken but when this happens the collection logic should combine such queries by table to reduce the number of roundtrips to the database. For example, given the following models class Person(models.Model): friends = models.ManyToManyField('self') class User(models.Model): pass class Entry(models.Model): createdby = models.ForeignKey(User) updatedby = models.ForeignKey(User) Issuing a () or () will result in 3 queries of the form DELETE FROM personfriends WHERE fromid = :id DELETE FROM personfriends WHERE toid = :id DELETE FROM person WHERE id = :id DELETE FROM entry WHERE createdbyid = :id DELETE FROM entry WHERE updatedby = :id DELETRE FROM user WHERE id = :id But both queries (or N queries depending on the number of foreign relationships) can be combined into a single one by using OR DELETE FROM personfriends WHERE fromid = :id OR toid = :id DELETE FROM person WHERE id = :id DELETE FROM entry WHERE createdbyid = :id OR updatedby = :id DELETE FROM user WHERE id = :id

source_attr=ptr.remote_field.related_name,                                  collect_related=False,                                  reverse_dependency=True)        if collect_related:            if keep_parents:                parents = set(model._meta.get_parent_list())            for related in get_candidate_relations_to_delete(model._meta):                # Preserve parent reverse relationships if keep_parents=True.                if keep_parents and related.model in parents:                    continue                field = related.field                if field.remote_field.on_delete == DO_NOTHING:                    continue                batches = self.get_del_batches(new_objs, field)                for batch in batches:                    sub_objs = self.related_objects(related, batch)                    if self.can_fast_delete(sub_objs, from_field=field):                        self.fast_deletes.append(sub_objs)                    else:                        related_model = related.related_model                        # Non-referenced fields can be deferred if no signal                        # receivers are connected for the related model as                        # they'll never be exposed to the user. Skip field                        # deferring when some relationships are select_related                        # as interactions between both features are hard to                        # get right. This should only happen in the rare                        # cases where .related_objects is overridden anyway.                        if not (sub_objs.query.select_related or self._has_signal_listeners(related_model)):                            referenced_fields = set(chain.from_iterable(                                (rf.attname for rf in rel.field.foreign_related_fields)                                for rel in get_candidate_relations_to_delete(related_model._meta)                            ))                            sub_objs = sub_objs.only(*tuple(referenced_fields))                        if sub_objs:                            field.remote_field.on_delete(self, field, sub_objs, self.using)            for field in model._meta.private_fields:                if hasattr(field, 'bulk_related_objects'):                    # It's something like generic foreign key.                    sub_objs = field.bulk_related_objects(new_objs, self.using)                    self.collect(sub_objs, source=model, nullable=True)    def related_objects(self, related, objs):        if not collect_related:            return        if keep_parents:            parents = set(model._meta.get_parent_list())        model_fast_deletes = defaultdict(list)        for related in get_candidate_relations_to_delete(model._meta):            # Preserve parent reverse relationships if keep_parents=True.            if keep_parents and related.model in parents:                continue            field = related.field            if field.remote_field.on_delete == DO_NOTHING:                continue            related_model = related.related_model            if self.can_fast_delete(related_model, from_field=field):                model_fast_deletes[related_model].append(field)                continue            batches = self.get_del_batches(new_objs, [field])            for batch in batches:                sub_objs = self.related_objects(related_model, [field], batch)                # Non-referenced fields can be deferred if no signal receivers                # are connected for the related model as they'll never be                # exposed to the user. Skip field deferring when some                # relationships are select_related as interactions between both                # features are hard to get right. This should only happen in                # the rare cases where .related_objects is overridden anyway.                if not (sub_objs.query.select_related or self._has_signal_listeners(related_model)):                    referenced_fields = set(chain.from_iterable(                        (rf.attname for rf in rel.field.foreign_related_fields)                        for rel in get_candidate_relations_to_delete(related_model._meta)                    ))                    sub_objs = sub_objs.only(*tuple(referenced_fields))                if sub_objs:                    field.remote_field.on_delete(self, field, sub_objs, self.using)        for related_model, related_fields in model_fast_deletes.items():            batches = self.get_del_batches(new_objs, related_fields)            for batch in batches:                sub_objs = self.related_objects(related_model, related_fields, batch)                self.fast_deletes.append(sub_objs)        for field in model._meta.private_fields:            if hasattr(field, 'bulk_related_objects'):                # It's something like generic foreign key.                sub_objs = field.bulk_related_objects(new_objs, self.using)                self.collect(sub_objs, source=model, nullable=True)    def related_objects(self, related_model, related_fields, objs):         """        Get a QuerySet of objects related to `objs` via the relation `related`.        Get a QuerySet of the related model to objs via related fields.         """        return related.related_model._base_manager.using(self.using).filter(            **{"%s__in" % related.field.name: objs}        )        predicate = reduce(operator.or_, (            query_utils.Q(**{'%s__in' % related_field.name: objs})            for related_field in related_fields        ))        return related_model._base_manager.using(self.using).filter(predicate)      def instances_with_model(self):         for model, instances in self.data.items():


Question: ConditionalGetMiddleware returns 304 if ETag is the same but Last-Modified has changed. Description (last modified by Mariusz Felisiak) ConditionalGetMiddleware in combination with apache x-sendfile (django-sendfile) doesn't work properly. Each response gets a ETag generated based on response.content which is an empty string in the case of a x-sendfile response, so each time the file is accessed, the ETag generated by ConditionalGetMiddleware is the same. Regardless of the changed file/changed mtime. In getconditionalresponse() the ETag (which is always the same hash of empty string) is checked first and returns a 304 because it ignores Last-Modified time. Django shouldn't return 304 if ETag is the same but Last-Modified has changed. Related with .

def set_response_etag(response):    if not response.streaming:    if not response.streaming and response.content:         response['ETag'] = quote_etag(hashlib.md5(response.content).hexdigest())     return response


Question: DateTimeField doesn't accept ISO 8601 formatted date string Description DateTimeField doesn't accept ISO 8601 formatted date string. Differene is that ISO format allows date and time separator to be capital T letter. (Format being YYYY-MM-DDTHH:MM:SS. Django expects to have only space as a date and time separator.

URLInput, ) from django.utils import formatsfrom django.utils.dateparse import parse_durationfrom django.utils.dateparse import parse_datetime, parse_duration from django.utils.duration import duration_string from django.utils.ipv6 import clean_ipv6_address from django.utils.regex_helper import _lazy_re_compile


Question: DateTimeField doesn't accept ISO 8601 formatted date string Description DateTimeField doesn't accept ISO 8601 formatted date string. Differene is that ISO format allows date and time separator to be capital T letter. (Format being YYYY-MM-DDTHH:MM:SS. Django expects to have only space as a date and time separator.

if isinstance(value, datetime.date):             result = datetime.datetime(value.year, value.month, value.day)             return from_current_timezone(result)        result = super().to_python(value)        try:            result = parse_datetime(value.strip())        except ValueError:            raise ValidationError(self.error_messages['invalid'], code='invalid')        if not result:            result = super().to_python(value)         return from_current_timezone(result)      def strptime(self, value, format):


Question: Explicitly SameSite: None cookies. Description When setting cookies (with .setcookie and setsigned_cookie) the default value for the samesite argument is None but the problem here is that Django doesn't do anything with the None value. This behaviour used to be fine because most browsers assumed that a missing samesite attribute on a cookie meant None. But now, Chrome has issued a warning that samesite has to be explicitly set to None else the cookies won't be sent for cross-origin requests.

if httponly:             self.cookies[key]['httponly'] = True         if samesite:            if samesite.lower() not in ('lax', 'strict'):                raise ValueError('samesite must be "lax" or "strict".')            if samesite.lower() not in ('lax', 'none', 'strict'):                raise ValueError('samesite must be "lax", "none", or "strict".')             self.cookies[key]['samesite'] = samesite      def setdefault(self, key, value):


Question: ManagementUtility.fetchcommand prints "No Django settings specified." even if they are. Description fetchcommand(...) currently does the following: if ('DJANGOSETTINGSMODULE'): # If is missing due to misconfigured settings, the # following line will retrigger an ImproperlyConfigured exception # (getcommands() swallows the original one) so the user is # informed about it. settings.INSTALLEDAPPS else: ("No Django settings specified.\n") which doesn't account for settings being set via a UserSettingsHolder by doing settings.configure(...) But the parent execute method correctly checks if settings.configured: I've not checked deeply, but I don't think the intent or outcome depends specifically on the LazySettings having been configured via a Settings through a named module import, and it would seem that if settings.configured: could/should apply here too.

# (get_commands() swallows the original one) so the user is                 # informed about it.                 settings.INSTALLED_APPS            else:            elif not settings.configured:                 sys.stderr.write("No Django settings specified.\n")             possible_matches = get_close_matches(subcommand, commands)             sys.stderr.write('Unknown command: %r' % subcommand)


Question: Prevent using isnull lookup with non-boolean value. Description (last modified by Mariusz Felisiak) isnull should not allow for non-boolean values. Using truthy/falsey doesn't promote INNER JOIN to an OUTER JOIN but works fine for a simple queries. Using non-boolean values is undocumented and untested. IMO we should raise an error for non-boolean values to avoid confusion and for consistency.

import itertools import math<ins>import warnings</ins> from copy import copy  from django.core.exceptions import EmptyResultSet


Question: Prevent using isnull lookup with non-boolean value. Description (last modified by Mariusz Felisiak) isnull should not allow for non-boolean values. Using truthy/falsey doesn't promote INNER JOIN to an OUTER JOIN but works fine for a simple queries. Using non-boolean values is undocumented and untested. IMO we should raise an error for non-boolean values to avoid confusion and for consistency.

) from django.db.models.query_utils import RegisterLookupMixin from django.utils.datastructures import OrderedSet<ins>from django.utils.deprecation import RemovedInDjango40Warning</ins> from django.utils.functional import cached_property


Question: Prevent using isnull lookup with non-boolean value. Description (last modified by Mariusz Felisiak) isnull should not allow for non-boolean values. Using truthy/falsey doesn't promote INNER JOIN to an OUTER JOIN but works fine for a simple queries. Using non-boolean values is undocumented and untested. IMO we should raise an error for non-boolean values to avoid confusion and for consistency.

prepare_rhs = False      def as_sql(self, compiler, connection):<ins>        if not isinstance(self.rhs, bool):            # When the deprecation ends, replace with:            # raise ValueError(            #     'The QuerySet value for an isnull lookup must be True or '            #     'False.'            # )            warnings.warn(                'Using a non-boolean value for an isnull lookup is '                'deprecated, use True or False instead.',                RemovedInDjango40Warning,            )</ins>         sql, params = compiler.compile(self.lhs)         if self.rhs:             return "%s IS NULL" % sql, params


Question: ForeignKey's tofield parameter gets the old field's name when renaming a PrimaryKey. Description Having these two models class ModelA(models.Model): fieldwrong = models.CharField('field1', maxlength=50, primarykey=True) # I'm a Primary key. class ModelB(models.Model): fieldfk = models.ForeignKey(ModelA, blank=True, null=True, ondelete=models.CASCADE) ... migrations applyed ... the ModelA.fieldwrong field has been renamed ... and Django recognizes the "renaming" class ModelA(models.Model): fieldfixed = models.CharField('field1', maxlength=50, primarykey=True) # I'm a Primary key. Attempts to tofield parameter. The tofield points to the oldname (fieldtypo) and not to the new one ("fieldfixed") class Migration(migrations.Migration): dependencies = [ ('app1', '0001initial'), ] operations = [ migrations.RenameField( modelname='modela', oldname='fieldwrong', newname='fieldfixed', ), migrations.AlterField( modelname='modelb', name='modela', field=models.ForeignKey(blank=True, null=True, , to='app1.ModelB', tofield='field_wrong'), ), ]

if remote_field_name:                     to_field_rename_key = rename_key + (remote_field_name,)                     if to_field_rename_key in self.renamed_fields:<ins>                        # Repoint both model and field name because to_field                        # inclusion in ForeignKey.deconstruct() is based on                        # both.                        new_field.remote_field.model = old_field.remote_field.model</ins>                         new_field.remote_field.field_name = old_field.remote_field.field_name                 # Handle ForeignObjects which can have multiple from_fields/to_fields.                 from_fields = getattr(new_field, 'from_fields', None)


Question: "migrate --plan" outputs "IRREVERSIBLE" on RunPython operations without docstrings. Description Given a migration like: from import migrations def forward(apps, schemaeditor): pass def reverse(apps, schemaeditor): pass class Migration(migrations.Migration): operations = [ migrations.RunPython(forward, reverse) ] migrate --plan will output: Planned operations: example.0001initial Raw Python operation -IRREVERSIBLE The migration should not be described as "irreversible". This error is in the definition of describeoperation in , reproduced below with line numbers from 2.2.6 tag. 343 344 def describeoperation(operation, backwards): 345 """Return a string that describes a migration operation for --plan.""" 346 prefix = '' 347 if hasattr(operation, 'code'): 348 code = operation.reversecode if backwards else 349 action = if code else '' 350 elif hasattr(operation, 'sql'): 351 action = operation.reversesql if backwards else 352 else: 353 action = '' 354 if backwards: 355 prefix = 'Undo ' 356 if action is None: 357 action = 'IRREVERSIBLE' 358 iserror = True 359 else: 360 action = str(action).replace('\n', '') 361 iserror = False 362 if action: 363 action = ' -' + action 364 truncated = Truncator(action) 365 return prefix + operation.describe() + (40), iserror Line 349 uses the docstring as the output string. Line 356 tests that value and sets action = 'IRREVERSIBLE' on line 357 because the dosctring is None. It would appear that the intention is to use a docstring to describe the operation, if available, and leave it blank otherwise. However, because it tests against code instead of it actually sets action = None resulting in 'IRREVERSIBLE' being displayed. Proposed Solutions below For a localized fix, I believe line 349 should be replaced by if code: action = if else '' else: action = None However, a more holistic view suggests that displaying "IRREVERSIBLE" isn't really the correct thing to do. "IRREVERSIBLE" is set when iserror is also set to True and seems to be trying to indicate that the migration operation is invalid rather than irreversible. That is, if code/reversecode is None (line 348) or sql/reversesql is None (line 351) the migration can't run. Since sql and code are required parameters for their respective Operations, action should only possibly be None in the reverse case, which seems to be what this code is trying to capture and explain. Given that, a better approach would probably make use of the reversible property defined on RunSQL and RunPython operations. This is a little verbose and could probably be pared down, but I think it has the right logic: def describeoperation(operation, backwards): """Return a string that describes a migration operation for --plan.""" prefix = '' action = '' iserror = False if backwards: prefix = 'Undo ' if hasattr(operation, 'reversible') and not operation.reversible: action = 'INVALID' iserror = True elif hasattr(operation, 'reversecode'): action = if else '' elif hasattr(operation, 'reversesql'): action = if else '' else: if hasattr(operation, 'code'): action = if else '' elif hasattr(operation, 'sql'): action = if else '' action = ' -' + str(action).replace('\n', '') truncated = Truncator(action) return prefix + operation.describe() + (40), is_error

def describe_operation(operation, backwards):         """Return a string that describes a migration operation for --plan."""         prefix = ''        is_error = False         if hasattr(operation, 'code'):             code = operation.reverse_code if backwards else operation.code            action = code.__doc__ if code else ''            action = (code.__doc__ or '') if code else None         elif hasattr(operation, 'sql'):             action = operation.reverse_sql if backwards else operation.sql         else:             action = ''             if backwards:                 prefix = 'Undo '        if action is None:        if action is not None:            action = str(action).replace('\n', '')        elif backwards:             action = 'IRREVERSIBLE'             is_error = True        else:            action = str(action).replace('\n', '')            is_error = False         if action:             action = ' -> ' + action         truncated = Truncator(action)


Question: Make prefetchrelated faster by lazily creating related querysets Description In one project of mine I will need to prefetch the following things for each "machine": computerdataoperatingsystem, identifiers. computerdata is one-to-one to machine, operatingsystem is manytomany, and identifiers are many-to-one. The data is distributed in a way that any one machine have on average one operating system, and a couple of identifiers. Fetching data results in this profile: 1 0.000 0.000 6.835 6.835 (<module) 1 0.000 0.000 6.795 6.795 (executefromcommandline) 1 0.000 0.000 6.795 6.795 (execute) 1 0.000 0.000 6.207 6.207 (runfromargv) 1 0.000 0.000 6.199 6.199 (execute) 1 0.000 0.000 6.072 6.072 (handle) 10/2 0.016 0.002 6.069 3.034 (fetchall) 6/1 0.000 0.000 6.043 6.043 (iter) 1 0.000 0.000 5.837 5.837 (prefetchrelatedobjects) 1 0.009 0.009 5.837 5.837 (prefetchrelatedobjects) 3 0.080 0.027 5.819 1.940 (prefetchonelevel) 4640 0.018 0.000 3.917 0.001 (all) 4646 0.014 0.000 3.206 0.001 (filter) 4648 0.037 0.000 3.193 0.001 (filterorexclude) 4648 0.031 0.000 2.661 0.001 (addq) 4648 0.053 0.000 2.401 0.001 (addq) 4648 0.144 0.000 2.284 0.000 (buildfilter) 2320 0.040 0.000 2.076 0.001 (getqueryset) 2320 0.063 0.000 1.823 0.001 (getqueryset) 0.068 0.000 1.052 0.000 (iterator) 1 0.023 0.023 0.993 0.993 (getprefetchqueryset) 9299 0.067 0.000 0.841 0.000 (clone) 4649 0.086 0.000 0.752 0.000 (setupjoins) 9299 0.226 0.000 0.738 0.000 (clone) 4644 0.177 0.000 0.668 0.000 (getlookupconstraint) 1 0.000 0.000 0.577 0.577 (fetchcommand) 0.330 0.000 0.548 0.000 (init) 127/79 0.007 0.000 0.447 0.006 {import_} 4645 0.012 0.000 0.443 0.000 (using) 0.017 0.000 0.433 0.000 (resultsiter) <SNIP5 0.197 0.039 0.197 0.039 {method 'execute' of '' objects} If I am reading this correctly, the actual data fetching costs 0.2 seconds of the total runtime of 6.8 seconds. (In reality the ratio is 0.2 vs 2 seconds due to overhead of profiling not affecting the execute time but having a big effect on other parts). The big "failure" in above profile is the creation of related querysets: 4640 0.018 0.000 3.917 0.001 (all) this takes more than half (approx 57%) of the total runtime. Every cycle here is wasted - we don't ever need the related queryset when using the prefetched results. I see two options here: Allow assigning the results to somewhere else than () (that is, a plain list you can name). This would naturally get rid of the overhead, but then you will need to alter code to explicitly use the named list when that is available. Somehow lazily instantiate the .all() queryset. If prefetch is in effect calling () will not create a queryset, it just creates a proxy which when iterated gives the related objects, but works otherwise like the related queryset (not like manager). I prefer option 2 as this doesn't require any new APIs or changes to user code to take advantage of this feature. However creating a proxy object that works like the related queryset except for iteration, and which doesn't involve actually creating that queryset will likely be an interesting technical challenge (for example it would be nice to ensure isinstance((), QuerySet) == True. Solving this challenge will likely speed up some prefetch_related queries by 50% or so.

queryset._add_hints(instance=self.instance)             if self._db:                 queryset = queryset.using(self._db)            queryset._defer_next_filter = True             return queryset._next_is_sticky().filter(**self.core_filters)          def _remove_prefetched_objects(self):diff --git a/django/db/models/query.py b/django/db/models/query.py-- a/django/db/models/query.py++ b/django/db/models/query.py


Question: Make prefetchrelated faster by lazily creating related querysets Description In one project of mine I will need to prefetch the following things for each "machine": computerdataoperatingsystem, identifiers. computerdata is one-to-one to machine, operatingsystem is manytomany, and identifiers are many-to-one. The data is distributed in a way that any one machine have on average one operating system, and a couple of identifiers. Fetching data results in this profile: 1 0.000 0.000 6.835 6.835 (<module) 1 0.000 0.000 6.795 6.795 (executefromcommandline) 1 0.000 0.000 6.795 6.795 (execute) 1 0.000 0.000 6.207 6.207 (runfromargv) 1 0.000 0.000 6.199 6.199 (execute) 1 0.000 0.000 6.072 6.072 (handle) 10/2 0.016 0.002 6.069 3.034 (fetchall) 6/1 0.000 0.000 6.043 6.043 (iter) 1 0.000 0.000 5.837 5.837 (prefetchrelatedobjects) 1 0.009 0.009 5.837 5.837 (prefetchrelatedobjects) 3 0.080 0.027 5.819 1.940 (prefetchonelevel) 4640 0.018 0.000 3.917 0.001 (all) 4646 0.014 0.000 3.206 0.001 (filter) 4648 0.037 0.000 3.193 0.001 (filterorexclude) 4648 0.031 0.000 2.661 0.001 (addq) 4648 0.053 0.000 2.401 0.001 (addq) 4648 0.144 0.000 2.284 0.000 (buildfilter) 2320 0.040 0.000 2.076 0.001 (getqueryset) 2320 0.063 0.000 1.823 0.001 (getqueryset) 0.068 0.000 1.052 0.000 (iterator) 1 0.023 0.023 0.993 0.993 (getprefetchqueryset) 9299 0.067 0.000 0.841 0.000 (clone) 4649 0.086 0.000 0.752 0.000 (setupjoins) 9299 0.226 0.000 0.738 0.000 (clone) 4644 0.177 0.000 0.668 0.000 (getlookupconstraint) 1 0.000 0.000 0.577 0.577 (fetchcommand) 0.330 0.000 0.548 0.000 (init) 127/79 0.007 0.000 0.447 0.006 {import_} 4645 0.012 0.000 0.443 0.000 (using) 0.017 0.000 0.433 0.000 (resultsiter) <SNIP5 0.197 0.039 0.197 0.039 {method 'execute' of '' objects} If I am reading this correctly, the actual data fetching costs 0.2 seconds of the total runtime of 6.8 seconds. (In reality the ratio is 0.2 vs 2 seconds due to overhead of profiling not affecting the execute time but having a big effect on other parts). The big "failure" in above profile is the creation of related querysets: 4640 0.018 0.000 3.917 0.001 (all) this takes more than half (approx 57%) of the total runtime. Every cycle here is wasted - we don't ever need the related queryset when using the prefetched results. I see two options here: Allow assigning the results to somewhere else than () (that is, a plain list you can name). This would naturally get rid of the overhead, but then you will need to alter code to explicitly use the named list when that is available. Somehow lazily instantiate the .all() queryset. If prefetch is in effect calling () will not create a queryset, it just creates a proxy which when iterated gives the related objects, but works otherwise like the related queryset (not like manager). I prefer option 2 as this doesn't require any new APIs or changes to user code to take advantage of this feature. However creating a proxy object that works like the related queryset except for iteration, and which doesn't involve actually creating that queryset will likely be an interesting technical challenge (for example it would be nice to ensure isinstance((), QuerySet) == True. Solving this challenge will likely speed up some prefetch_related queries by 50% or so.

self.model = model         self._db = using         self._hints = hints or {}        self.query = query or sql.Query(self.model)        self._query = query or sql.Query(self.model)         self._result_cache = None         self._sticky_filter = False         self._for_write = False


Question: Make prefetchrelated faster by lazily creating related querysets Description In one project of mine I will need to prefetch the following things for each "machine": computerdataoperatingsystem, identifiers. computerdata is one-to-one to machine, operatingsystem is manytomany, and identifiers are many-to-one. The data is distributed in a way that any one machine have on average one operating system, and a couple of identifiers. Fetching data results in this profile: 1 0.000 0.000 6.835 6.835 (<module) 1 0.000 0.000 6.795 6.795 (executefromcommandline) 1 0.000 0.000 6.795 6.795 (execute) 1 0.000 0.000 6.207 6.207 (runfromargv) 1 0.000 0.000 6.199 6.199 (execute) 1 0.000 0.000 6.072 6.072 (handle) 10/2 0.016 0.002 6.069 3.034 (fetchall) 6/1 0.000 0.000 6.043 6.043 (iter) 1 0.000 0.000 5.837 5.837 (prefetchrelatedobjects) 1 0.009 0.009 5.837 5.837 (prefetchrelatedobjects) 3 0.080 0.027 5.819 1.940 (prefetchonelevel) 4640 0.018 0.000 3.917 0.001 (all) 4646 0.014 0.000 3.206 0.001 (filter) 4648 0.037 0.000 3.193 0.001 (filterorexclude) 4648 0.031 0.000 2.661 0.001 (addq) 4648 0.053 0.000 2.401 0.001 (addq) 4648 0.144 0.000 2.284 0.000 (buildfilter) 2320 0.040 0.000 2.076 0.001 (getqueryset) 2320 0.063 0.000 1.823 0.001 (getqueryset) 0.068 0.000 1.052 0.000 (iterator) 1 0.023 0.023 0.993 0.993 (getprefetchqueryset) 9299 0.067 0.000 0.841 0.000 (clone) 4649 0.086 0.000 0.752 0.000 (setupjoins) 9299 0.226 0.000 0.738 0.000 (clone) 4644 0.177 0.000 0.668 0.000 (getlookupconstraint) 1 0.000 0.000 0.577 0.577 (fetchcommand) 0.330 0.000 0.548 0.000 (init) 127/79 0.007 0.000 0.447 0.006 {import_} 4645 0.012 0.000 0.443 0.000 (using) 0.017 0.000 0.433 0.000 (resultsiter) <SNIP5 0.197 0.039 0.197 0.039 {method 'execute' of '' objects} If I am reading this correctly, the actual data fetching costs 0.2 seconds of the total runtime of 6.8 seconds. (In reality the ratio is 0.2 vs 2 seconds due to overhead of profiling not affecting the execute time but having a big effect on other parts). The big "failure" in above profile is the creation of related querysets: 4640 0.018 0.000 3.917 0.001 (all) this takes more than half (approx 57%) of the total runtime. Every cycle here is wasted - we don't ever need the related queryset when using the prefetched results. I see two options here: Allow assigning the results to somewhere else than () (that is, a plain list you can name). This would naturally get rid of the overhead, but then you will need to alter code to explicitly use the named list when that is available. Somehow lazily instantiate the .all() queryset. If prefetch is in effect calling () will not create a queryset, it just creates a proxy which when iterated gives the related objects, but works otherwise like the related queryset (not like manager). I prefer option 2 as this doesn't require any new APIs or changes to user code to take advantage of this feature. However creating a proxy object that works like the related queryset except for iteration, and which doesn't involve actually creating that queryset will likely be an interesting technical challenge (for example it would be nice to ensure isinstance((), QuerySet) == True. Solving this challenge will likely speed up some prefetch_related queries by 50% or so.

self._known_related_objects = {}  # {rel_field: {pk: rel_obj}}         self._iterable_class = ModelIterable         self._fields = None<ins>        self._defer_next_filter = False        self._deferred_filter = None    @property    def query(self):        if self._deferred_filter:            negate, args, kwargs = self._deferred_filter            self._filter_or_exclude_inplace(negate, *args, **kwargs)            self._deferred_filter = None        return self._query    @query.setter    def query(self, value):        self._query = value</ins>      def as_manager(cls):         # Address the circular dependency between `Queryset` and `Manager`.


Question: Make prefetchrelated faster by lazily creating related querysets Description In one project of mine I will need to prefetch the following things for each "machine": computerdataoperatingsystem, identifiers. computerdata is one-to-one to machine, operatingsystem is manytomany, and identifiers are many-to-one. The data is distributed in a way that any one machine have on average one operating system, and a couple of identifiers. Fetching data results in this profile: 1 0.000 0.000 6.835 6.835 (<module) 1 0.000 0.000 6.795 6.795 (executefromcommandline) 1 0.000 0.000 6.795 6.795 (execute) 1 0.000 0.000 6.207 6.207 (runfromargv) 1 0.000 0.000 6.199 6.199 (execute) 1 0.000 0.000 6.072 6.072 (handle) 10/2 0.016 0.002 6.069 3.034 (fetchall) 6/1 0.000 0.000 6.043 6.043 (iter) 1 0.000 0.000 5.837 5.837 (prefetchrelatedobjects) 1 0.009 0.009 5.837 5.837 (prefetchrelatedobjects) 3 0.080 0.027 5.819 1.940 (prefetchonelevel) 4640 0.018 0.000 3.917 0.001 (all) 4646 0.014 0.000 3.206 0.001 (filter) 4648 0.037 0.000 3.193 0.001 (filterorexclude) 4648 0.031 0.000 2.661 0.001 (addq) 4648 0.053 0.000 2.401 0.001 (addq) 4648 0.144 0.000 2.284 0.000 (buildfilter) 2320 0.040 0.000 2.076 0.001 (getqueryset) 2320 0.063 0.000 1.823 0.001 (getqueryset) 0.068 0.000 1.052 0.000 (iterator) 1 0.023 0.023 0.993 0.993 (getprefetchqueryset) 9299 0.067 0.000 0.841 0.000 (clone) 4649 0.086 0.000 0.752 0.000 (setupjoins) 9299 0.226 0.000 0.738 0.000 (clone) 4644 0.177 0.000 0.668 0.000 (getlookupconstraint) 1 0.000 0.000 0.577 0.577 (fetchcommand) 0.330 0.000 0.548 0.000 (init) 127/79 0.007 0.000 0.447 0.006 {import_} 4645 0.012 0.000 0.443 0.000 (using) 0.017 0.000 0.433 0.000 (resultsiter) <SNIP5 0.197 0.039 0.197 0.039 {method 'execute' of '' objects} If I am reading this correctly, the actual data fetching costs 0.2 seconds of the total runtime of 6.8 seconds. (In reality the ratio is 0.2 vs 2 seconds due to overhead of profiling not affecting the execute time but having a big effect on other parts). The big "failure" in above profile is the creation of related querysets: 4640 0.018 0.000 3.917 0.001 (all) this takes more than half (approx 57%) of the total runtime. Every cycle here is wasted - we don't ever need the related queryset when using the prefetched results. I see two options here: Allow assigning the results to somewhere else than () (that is, a plain list you can name). This would naturally get rid of the overhead, but then you will need to alter code to explicitly use the named list when that is available. Somehow lazily instantiate the .all() queryset. If prefetch is in effect calling () will not create a queryset, it just creates a proxy which when iterated gives the related objects, but works otherwise like the related queryset (not like manager). I prefer option 2 as this doesn't require any new APIs or changes to user code to take advantage of this feature. However creating a proxy object that works like the related queryset except for iteration, and which doesn't involve actually creating that queryset will likely be an interesting technical challenge (for example it would be nice to ensure isinstance((), QuerySet) == True. Solving this challenge will likely speed up some prefetch_related queries by 50% or so.

"Cannot filter a query once a slice has been taken."          clone = self._chain()        if negate:            clone.query.add_q(~Q(*args, **kwargs))        if self._defer_next_filter:            self._defer_next_filter = False            clone._deferred_filter = negate, args, kwargs         else:            clone.query.add_q(Q(*args, **kwargs))            clone._filter_or_exclude_inplace(negate, *args, **kwargs)         return clone     def _filter_or_exclude_inplace(self, negate, *args, **kwargs):        if negate:            self._query.add_q(~Q(*args, **kwargs))        else:            self._query.add_q(Q(*args, **kwargs))     def complex_filter(self, filter_obj):         """         Return a new QuerySet instance with filter_obj added to the filters.


Question: Make prefetchrelated faster by lazily creating related querysets Description In one project of mine I will need to prefetch the following things for each "machine": computerdataoperatingsystem, identifiers. computerdata is one-to-one to machine, operatingsystem is manytomany, and identifiers are many-to-one. The data is distributed in a way that any one machine have on average one operating system, and a couple of identifiers. Fetching data results in this profile: 1 0.000 0.000 6.835 6.835 (<module) 1 0.000 0.000 6.795 6.795 (executefromcommandline) 1 0.000 0.000 6.795 6.795 (execute) 1 0.000 0.000 6.207 6.207 (runfromargv) 1 0.000 0.000 6.199 6.199 (execute) 1 0.000 0.000 6.072 6.072 (handle) 10/2 0.016 0.002 6.069 3.034 (fetchall) 6/1 0.000 0.000 6.043 6.043 (iter) 1 0.000 0.000 5.837 5.837 (prefetchrelatedobjects) 1 0.009 0.009 5.837 5.837 (prefetchrelatedobjects) 3 0.080 0.027 5.819 1.940 (prefetchonelevel) 4640 0.018 0.000 3.917 0.001 (all) 4646 0.014 0.000 3.206 0.001 (filter) 4648 0.037 0.000 3.193 0.001 (filterorexclude) 4648 0.031 0.000 2.661 0.001 (addq) 4648 0.053 0.000 2.401 0.001 (addq) 4648 0.144 0.000 2.284 0.000 (buildfilter) 2320 0.040 0.000 2.076 0.001 (getqueryset) 2320 0.063 0.000 1.823 0.001 (getqueryset) 0.068 0.000 1.052 0.000 (iterator) 1 0.023 0.023 0.993 0.993 (getprefetchqueryset) 9299 0.067 0.000 0.841 0.000 (clone) 4649 0.086 0.000 0.752 0.000 (setupjoins) 9299 0.226 0.000 0.738 0.000 (clone) 4644 0.177 0.000 0.668 0.000 (getlookupconstraint) 1 0.000 0.000 0.577 0.577 (fetchcommand) 0.330 0.000 0.548 0.000 (init) 127/79 0.007 0.000 0.447 0.006 {import_} 4645 0.012 0.000 0.443 0.000 (using) 0.017 0.000 0.433 0.000 (resultsiter) <SNIP5 0.197 0.039 0.197 0.039 {method 'execute' of '' objects} If I am reading this correctly, the actual data fetching costs 0.2 seconds of the total runtime of 6.8 seconds. (In reality the ratio is 0.2 vs 2 seconds due to overhead of profiling not affecting the execute time but having a big effect on other parts). The big "failure" in above profile is the creation of related querysets: 4640 0.018 0.000 3.917 0.001 (all) this takes more than half (approx 57%) of the total runtime. Every cycle here is wasted - we don't ever need the related queryset when using the prefetched results. I see two options here: Allow assigning the results to somewhere else than () (that is, a plain list you can name). This would naturally get rid of the overhead, but then you will need to alter code to explicitly use the named list when that is available. Somehow lazily instantiate the .all() queryset. If prefetch is in effect calling () will not create a queryset, it just creates a proxy which when iterated gives the related objects, but works otherwise like the related queryset (not like manager). I prefer option 2 as this doesn't require any new APIs or changes to user code to take advantage of this feature. However creating a proxy object that works like the related queryset except for iteration, and which doesn't involve actually creating that queryset will likely be an interesting technical challenge (for example it would be nice to ensure isinstance((), QuerySet) == True. Solving this challenge will likely speed up some prefetch_related queries by 50% or so.

clone.query.add_q(filter_obj)             return clone         else:            return self._filter_or_exclude(None, **filter_obj)            return self._filter_or_exclude(False, **filter_obj)      def _combinator_query(self, combinator, *other_qs, all=False):         # Clone the query to inherit the select list and everything


Question: bulkcreate batchsize param overrides the compatible batch size calculation Description (last modified by Ahmet Kucuk) At this line:  batchsize param overrides compatible batch size calculation. This looks like a bug as bulkupdate properly picks the minimum of two:  I suggest using similar batchsize = min(batchsize, maxbatchsize) if batchsize else maxbatchsize logic in bulkcreate as well. I am happy to open a PR for it.

if ignore_conflicts and not connections[self.db].features.supports_ignore_conflicts:             raise NotSupportedError('This database backend does not support ignoring conflicts.')         ops = connections[self.db].ops        batch_size = (batch_size or max(ops.bulk_batch_size(fields, objs), 1))        max_batch_size = max(ops.bulk_batch_size(fields, objs), 1)        batch_size = min(batch_size, max_batch_size) if batch_size else max_batch_size         inserted_rows = []         bulk_return = connections[self.db].features.can_return_rows_from_bulk_insert         for item in [objs[i:i + batch_size] for i in range(0, len(objs), batch_size)]:


Question: The value of a TextChoices/IntegerChoices field has a differing type Description If we create an instance of a model having a CharField or IntegerField with the keyword choices pointing to IntegerChoices or TextChoices, the value returned by the getter of the field will be of the same type as the one created by enum.Enum (enum value). For example, this model: from import models from import gettextlazy as class MyChoice(models.TextChoices): FIRSTCHOICE = "first", ("The first choice, it is") SECONDCHOICE = "second", ("The second choice, it is") class MyObject(models.Model): mystrvalue = models.CharField(maxlength=10, choices=MyChoice.choices) Then this test: from import TestCase from import MyObject, MyChoice class EnumTest(TestCase): def setUp(self) -None: self.myobject = (mystrvalue=MyChoice.FIRSTCHOICE) def testcreatedobjectisstr(self): myobject = self.myobject self.assertIsInstance(myobject.mystrvalue, str) self.assertEqual(str(myobject.mystrvalue), "first") def testretrievedobjectisstr(self): myobject = () self.assertIsInstance(myobject.mystrvalue, str) self.assertEqual(str(myobject.mystrvalue), "first") And then the results: (django30-venv) django30 test Creating test database for alias 'default'... System check identified no issues (0 silenced). F. FAIL: testcreatedobjectisstr () Traceback (most recent call last): File "", line 14, in testcreatedobjectisstr self.assertEqual(str(myobject.mystrvalue), "first") AssertionError: 'MyChoice.FIRSTCHOICE' != 'first' MyChoice.FIRSTCHOICE first Ran 2 tests in 0.002s FAILED (failures=1) We notice when invoking str(...) we don't actually get the value property of the enum value which can lead to some unexpected issues, especially when communicating to an external API with a freshly created instance that will send MyEnum.MyValue, and the one that was retrieved would send myvalue.

class Choices(enum.Enum, metaclass=ChoicesMeta):     """Class for creating enumerated choices."""    pass    def __str__(self):        """        Use value when cast to str, so that Choices set as model instance        attributes are rendered as expected in templates and similar contexts.        """        return str(self.value)   class IntegerChoices(int, Choices):


Question: Admin's datehierarchy excludes 31 october when using timezone with DST in northern hemisphere. Description introduced a subtle bug where it accidentally excludes 31 october in the admin datehierarchy filter after selecting october. The underlying reason is that the generated sql has < 2019-10-31 as opposed to < 2019-11-01 as it should be. This in turn is caused by applying the timezone for 2019-10-01 (with DST in countries in the northern hemisphere) to the date used for 2019-11-01. This causes the date conversion to actually operate on 2019-10-31 23:00 instead of 2019-11-01 00:00. It's a bit hard to explain in words, PR incoming that hopefully explains better in code.

)                 except ValueError as e:                     raise IncorrectLookupParameters(e) from e<del>                if settings.USE_TZ:                    from_date = make_aware(from_date)</del>                 if day:                     to_date = from_date + timedelta(days=1)                 elif month:


Question: Admin's datehierarchy excludes 31 october when using timezone with DST in northern hemisphere. Description introduced a subtle bug where it accidentally excludes 31 october in the admin datehierarchy filter after selecting october. The underlying reason is that the generated sql has < 2019-10-31 as opposed to < 2019-11-01 as it should be. This in turn is caused by applying the timezone for 2019-10-01 (with DST in countries in the northern hemisphere) to the date used for 2019-11-01. This causes the date conversion to actually operate on 2019-10-31 23:00 instead of 2019-11-01 00:00. It's a bit hard to explain in words, PR incoming that hopefully explains better in code.

to_date = (from_date + timedelta(days=32)).replace(day=1)                 else:                     to_date = from_date.replace(year=from_date.year + 1)<ins>                if settings.USE_TZ:                    from_date = make_aware(from_date)                    to_date = make_aware(to_date)</ins>                 lookup_params.update({                     '%s__gte' % self.date_hierarchy: from_date,                     '%s__lt' % self.date_hierarchy: to_date,


Question: The floatformat filter sometimes returns "-0" instead of "0". Description For values between 0 and -0.5, the floatformat filter returns "-0" where I would expect it to return "0". For example: $ python -m django --version 5 $ shell Python 3.5.3 (default, Sep 27 2018, 17:25:39) [GCC 6.3.0 ] on linux Type "help", "copyright", "credits" or "license" for more information. (InteractiveConsole) from django.template.defaultfilters import floatformat floatformat(-0.1, 0) '-0' floatformat(-0.01, 1) '-0.0' floatformat(-0.001, 2) '-0.00' floatformat(-0.4, 0) '-0' If others agree that this is a bug, I'll submit a patch.

# Avoid conversion to scientific notation by accessing `sign`, `digits`,     # and `exponent` from Decimal.as_tuple() directly.    sign, digits, exponent = d.quantize(exp, ROUND_HALF_UP, Context(prec=prec)).as_tuple()    rounded_d = d.quantize(exp, ROUND_HALF_UP, Context(prec=prec))    sign, digits, exponent = rounded_d.as_tuple()     digits = [str(digit) for digit in reversed(digits)]     while len(digits) <= abs(exponent):         digits.append('0')     digits.insert(-exponent, '.')    if sign:    if sign and rounded_d:         digits.append('-')     number = ''.join(reversed(digits))     return mark_safe(formats.number_format(number, abs(p)))


Question: Cannot override getFOOdisplay() in Django 2.2+. Description I cannot override the getFIELDdisplay function on models since version 2.2. It works in version 2.1. Example: class FooBar(models.Model): foobar = models.CharField(("foo"), choices=[(1, 'foo'), (2, 'bar')]) def str(self): return self.getfoobardisplay() # This returns 'foo' or 'bar' in 2.2, but 'something' in 2.1 def getfoobardisplay(self): return "something" What I expect is that I should be able to override this function.

if not getattr(cls, self.attname, None):                 setattr(cls, self.attname, self.descriptor_class(self))         if self.choices is not None:            setattr(cls, 'get_%s_display' % self.name,                    partialmethod(cls._get_FIELD_display, field=self))            if not hasattr(cls, 'get_%s_display' % self.name):                setattr(                    cls,                    'get_%s_display' % self.name,                    partialmethod(cls._get_FIELD_display, field=self),                )      def get_filter_kwargs_for_object(self, obj):         """


Question: Django installs /usr/bin/django-admin and /usr/bin/django- Description Django (since 1.7) installs /usr/bin/django-admin and /usr/bin/django- Both of them execute (). /usr/bin/django- does it directly, while /usr/bin/django-admin does it through pkgresources module of Setuptools. /usr/bin/django- #!/usr/bin/python3.4 from import management if name == "main": management.executefromcommandline() /usr/bin/django-admin: #!/usr/bin/python3.4 requires = 'Django==1.7' import sys from pkgresources import loadentrypoint if name == 'main': ( loadentrypoint('Django==1.7', 'consolescripts', 'django-admin')() ) /usr/lib64/python3.4/site-packages/Django-1.7- [consolescripts] django-admin = Installation of /usr/bin/django- is caused by scripts=['django/bin/django-'] in Installation of /usr/bin/django-admin is caused by entrypoints={'consolescripts': ['django-admin = ',]} in I think that it would suffice to install only one of these scripts.

#!/usr/bin/env python# When the django-admin.py deprecation ends, remove this script.import warnings from django.core import management try:    from django.utils.deprecation import RemovedInDjango40Warningexcept ImportError:    raise ImportError(        'django-admin.py was deprecated in Django 3.1 and removed in Django '        '4.0. Please manually remove this script from your virtual environment '        'and use django-admin instead.'    ) if __name__ == "__main__":    warnings.warn(        'django-admin.py is deprecated in favor of django-admin.',        RemovedInDjango40Warning,    )     management.execute_from_command_line()