Question: Optimize .delete() to use only required fields. Description Hi! We're in the process of upgrading our Django 1.11 installation from Python 2.7 to Python 3.6, however are hitting an unexpected UnicodeDecodeError during a .delete() run by our daily data purging management command. STR: Have an existing Django 1.11 project running under Python 2.7.15 that uses mysqlclient-python v1.3.13 to connect to MySQL server v5.7.23, with Django's DATABASES options including 'charset': 'utf8mb4' () Update to Python 3.6.8 Run the daily cycledata Django management command against the dev instance's DB:   Expected: That the cycledata management command succeeds, like it did under Python 2. Actual: Traceback (most recent call last): File "", line 16, in <moduleexecutefromcommandline() File "", line 364, in executefromcommandline utility.execute() File "", line 356, in execute self.fetchcommand(subcommand).runfromargv() File "", line 988, in nrwrapperBaseCommandrunfromargv return wrapped(args, kwargs) File "", line 283, in runfromargv self.execute(args, cmdoptions) File "", line 330, in execute output = (args, options) File "", line 139, in literalwrapper return wrapped(args, kwargs) File "", line 62, in handle options['sleeptime']) File "", line 461, in cycledata (guidin=jobschunk).delete() File "", line 619, in delete collector.collect(delquery) File "", line 223, in collect field.remotefield.ondelete(self, field, subobjs, ) File "", line 17, in CASCADE , ) File "", line 222, in collect elif subobjs: File "", line 254, in bool self.fetchall() File "", line 1121, in fetchall self.resultcache = list(self.iterableclass(self)) File "", line 53, in iter results = compiler.executesql(chunkedfetch=self.chunkedfetch) File "", line 899, in executesql raise originalexception File "", line 889, in executesql cursor.execute(sql, params) File "", line 64, in execute return (sql, params) File "", line 101, in execute return (query, args) File "", line 25, in execute args, *kwargs) File "", line 250, in execute self.errorhandler(self, exc, value) File "", line 50, in defaulterrorhandler raise errorvalue File "", line 247, in execute res = (query) File "", line 413, in query self.postgetresult() File "", line 417, in postgetresult = self.fetchrow(0) File "", line 385, in fetchrow return (size, self.fetchtype) File "", line 231, in stringdecoder return s.decode(db.encoding) UnicodeDecodeError: 'utf-8' codec can't decode byte 0xed in position 78: invalid continuation byte The exception occurs during the .delete() of Jobs, here:  Enabling debug logging of Django's DB backend, shows the generated SQL to be: SELECT FROM job WHERE (job.repositoryid = 1 AND job.submittime < '2018-10-21 11:03:32.') LIMIT 1; args=(1, '2018-10-21 11:03:32.') SELECT , failureline.jobguid, failureline.repositoryid, failureline.joblogid, , , , failureline.subtest, , failureline.expected, failureline.message, failureline.signature, , , failureline.stackwalkstdout, failureline.stackwalkstderr, failureline.bestclassificationid, failureline.bestisverified, failureline.created, failureline.modified FROM failureline WHERE failureline.jobguid IN ('-b854-4300-969a-/0'); args=('-b854-4300-969a-/0',) SELECT , job.repositoryid, , job.projectspecificid, job.autoclassifystatus, job.coalescedtoguid, job.signatureid, job.buildplatformid, job.machineplatformid, job.machineid, job.optioncollectionhash, job.jobtypeid, job.jobgroupid, job.productid, job.failureclassificationid, , , , , job.submittime, job.starttime, job.endtime, job.lastmodified, job.runningeta, , FROM job WHERE IN ('-b854-4300-969a-/0'); args=('-b854-4300-969a-/0',) SELECT , , , , FROM joblog WHERE IN (); args=(,) [2019-02-18 11:03:33,403] DEBUG [] (0.107) SELECT , failureline.jobguid, failureline.repositoryid, failureline.joblogid, , , , failureline.subtest, , failureline.expected, failureline.message, failureline.signature, , , failureline.stackwalkstdout, failureline.stackwalkstderr, failureline.bestclassificationid, failureline.bestisverified, failureline.created, failureline.modified FROM failureline WHERE failureline.joblogid IN (, ); args=(, ) SELECT , , , textlogstep.started, textlogstep.finished, textlogstep.startedlinenumber, textlogstep.finishedlinenumber, FROM textlogstep WHERE IN (); args=(,) SELECT , , , textlogerror.linenumber FROM textlogerror WHERE IN (); args=(,) Querying the textlogerror table for those ids shows there to be junk values in its line field. These are from data inserted when using Python 2.7, which presumably wasn't validating the unicode escape sequences being used. There appear to be two issues here: mysqlclient-python's behaviour differs depending on Python version - under Python 3 it defaults useunicode to True, which means it attempts to decode the line field but fails (since it doesn't use 'replace' or 'ignore'). This seems like something that the Django ORM should try to protect against (eg by setting useunicode to the same value on all Python versions and handling the unicode conversion itself), given it generally handles any implementation differences in layers lower than the ORM. the UnicodeDecodeError is occurring for a field () that is not actually needed for the .delete() (it's not a primary key etc), so Django shouldn't be fetching that field regardless when making the textlogerror SELECT query (Plus ideally Django would support cascade deletes, so we wouldn't need to use the current .delete() approach; ticket ) Fixing issue (2) would presumably also improve .delete() performance. Related:

model, {}).setdefault(             (field, value), set()).update(objs) <ins>    def _has_signal_listeners(self, model):        return (            signals.pre_delete.has_listeners(model) or            signals.post_delete.has_listeners(model)        )</ins>     def can_fast_delete(self, objs, from_field=None):         """         Determine if the objects in the given queryset-like or single object


Question: Optimize .delete() to use only required fields. Description Hi! We're in the process of upgrading our Django 1.11 installation from Python 2.7 to Python 3.6, however are hitting an unexpected UnicodeDecodeError during a .delete() run by our daily data purging management command. STR: Have an existing Django 1.11 project running under Python 2.7.15 that uses mysqlclient-python v1.3.13 to connect to MySQL server v5.7.23, with Django's DATABASES options including 'charset': 'utf8mb4' () Update to Python 3.6.8 Run the daily cycledata Django management command against the dev instance's DB:   Expected: That the cycledata management command succeeds, like it did under Python 2. Actual: Traceback (most recent call last): File "", line 16, in <moduleexecutefromcommandline() File "", line 364, in executefromcommandline utility.execute() File "", line 356, in execute self.fetchcommand(subcommand).runfromargv() File "", line 988, in nrwrapperBaseCommandrunfromargv return wrapped(args, kwargs) File "", line 283, in runfromargv self.execute(args, cmdoptions) File "", line 330, in execute output = (args, options) File "", line 139, in literalwrapper return wrapped(args, kwargs) File "", line 62, in handle options['sleeptime']) File "", line 461, in cycledata (guidin=jobschunk).delete() File "", line 619, in delete collector.collect(delquery) File "", line 223, in collect field.remotefield.ondelete(self, field, subobjs, ) File "", line 17, in CASCADE , ) File "", line 222, in collect elif subobjs: File "", line 254, in bool self.fetchall() File "", line 1121, in fetchall self.resultcache = list(self.iterableclass(self)) File "", line 53, in iter results = compiler.executesql(chunkedfetch=self.chunkedfetch) File "", line 899, in executesql raise originalexception File "", line 889, in executesql cursor.execute(sql, params) File "", line 64, in execute return (sql, params) File "", line 101, in execute return (query, args) File "", line 25, in execute args, *kwargs) File "", line 250, in execute self.errorhandler(self, exc, value) File "", line 50, in defaulterrorhandler raise errorvalue File "", line 247, in execute res = (query) File "", line 413, in query self.postgetresult() File "", line 417, in postgetresult = self.fetchrow(0) File "", line 385, in fetchrow return (size, self.fetchtype) File "", line 231, in stringdecoder return s.decode(db.encoding) UnicodeDecodeError: 'utf-8' codec can't decode byte 0xed in position 78: invalid continuation byte The exception occurs during the .delete() of Jobs, here:  Enabling debug logging of Django's DB backend, shows the generated SQL to be: SELECT FROM job WHERE (job.repositoryid = 1 AND job.submittime < '2018-10-21 11:03:32.') LIMIT 1; args=(1, '2018-10-21 11:03:32.') SELECT , failureline.jobguid, failureline.repositoryid, failureline.joblogid, , , , failureline.subtest, , failureline.expected, failureline.message, failureline.signature, , , failureline.stackwalkstdout, failureline.stackwalkstderr, failureline.bestclassificationid, failureline.bestisverified, failureline.created, failureline.modified FROM failureline WHERE failureline.jobguid IN ('-b854-4300-969a-/0'); args=('-b854-4300-969a-/0',) SELECT , job.repositoryid, , job.projectspecificid, job.autoclassifystatus, job.coalescedtoguid, job.signatureid, job.buildplatformid, job.machineplatformid, job.machineid, job.optioncollectionhash, job.jobtypeid, job.jobgroupid, job.productid, job.failureclassificationid, , , , , job.submittime, job.starttime, job.endtime, job.lastmodified, job.runningeta, , FROM job WHERE IN ('-b854-4300-969a-/0'); args=('-b854-4300-969a-/0',) SELECT , , , , FROM joblog WHERE IN (); args=(,) [2019-02-18 11:03:33,403] DEBUG [] (0.107) SELECT , failureline.jobguid, failureline.repositoryid, failureline.joblogid, , , , failureline.subtest, , failureline.expected, failureline.message, failureline.signature, , , failureline.stackwalkstdout, failureline.stackwalkstderr, failureline.bestclassificationid, failureline.bestisverified, failureline.created, failureline.modified FROM failureline WHERE failureline.joblogid IN (, ); args=(, ) SELECT , , , textlogstep.started, textlogstep.finished, textlogstep.startedlinenumber, textlogstep.finishedlinenumber, FROM textlogstep WHERE IN (); args=(,) SELECT , , , textlogerror.linenumber FROM textlogerror WHERE IN (); args=(,) Querying the textlogerror table for those ids shows there to be junk values in its line field. These are from data inserted when using Python 2.7, which presumably wasn't validating the unicode escape sequences being used. There appear to be two issues here: mysqlclient-python's behaviour differs depending on Python version - under Python 3 it defaults useunicode to True, which means it attempts to decode the line field but fails (since it doesn't use 'replace' or 'ignore'). This seems like something that the Django ORM should try to protect against (eg by setting useunicode to the same value on all Python versions and handling the unicode conversion itself), given it generally handles any implementation differences in layers lower than the ORM. the UnicodeDecodeError is occurring for a field () that is not actually needed for the .delete() (it's not a primary key etc), so Django shouldn't be fetching that field regardless when making the textlogerror SELECT query (Plus ideally Django would support cascade deletes, so we wouldn't need to use the current .delete() approach; ticket ) Fixing issue (2) would presumably also improve .delete() performance. Related:

model = objs.model         else:             return False        if (signals.pre_delete.has_listeners(model) or                signals.post_delete.has_listeners(model) or                signals.m2m_changed.has_listeners(model)):        if self._has_signal_listeners(model):             return False         # The use of from_field comes from the need to avoid cascade back to         # parent when parent delete is cascading to child.


Question: Optimize .delete() to use only required fields. Description Hi! We're in the process of upgrading our Django 1.11 installation from Python 2.7 to Python 3.6, however are hitting an unexpected UnicodeDecodeError during a .delete() run by our daily data purging management command. STR: Have an existing Django 1.11 project running under Python 2.7.15 that uses mysqlclient-python v1.3.13 to connect to MySQL server v5.7.23, with Django's DATABASES options including 'charset': 'utf8mb4' () Update to Python 3.6.8 Run the daily cycledata Django management command against the dev instance's DB:   Expected: That the cycledata management command succeeds, like it did under Python 2. Actual: Traceback (most recent call last): File "", line 16, in <moduleexecutefromcommandline() File "", line 364, in executefromcommandline utility.execute() File "", line 356, in execute self.fetchcommand(subcommand).runfromargv() File "", line 988, in nrwrapperBaseCommandrunfromargv return wrapped(args, kwargs) File "", line 283, in runfromargv self.execute(args, cmdoptions) File "", line 330, in execute output = (args, options) File "", line 139, in literalwrapper return wrapped(args, kwargs) File "", line 62, in handle options['sleeptime']) File "", line 461, in cycledata (guidin=jobschunk).delete() File "", line 619, in delete collector.collect(delquery) File "", line 223, in collect field.remotefield.ondelete(self, field, subobjs, ) File "", line 17, in CASCADE , ) File "", line 222, in collect elif subobjs: File "", line 254, in bool self.fetchall() File "", line 1121, in fetchall self.resultcache = list(self.iterableclass(self)) File "", line 53, in iter results = compiler.executesql(chunkedfetch=self.chunkedfetch) File "", line 899, in executesql raise originalexception File "", line 889, in executesql cursor.execute(sql, params) File "", line 64, in execute return (sql, params) File "", line 101, in execute return (query, args) File "", line 25, in execute args, *kwargs) File "", line 250, in execute self.errorhandler(self, exc, value) File "", line 50, in defaulterrorhandler raise errorvalue File "", line 247, in execute res = (query) File "", line 413, in query self.postgetresult() File "", line 417, in postgetresult = self.fetchrow(0) File "", line 385, in fetchrow return (size, self.fetchtype) File "", line 231, in stringdecoder return s.decode(db.encoding) UnicodeDecodeError: 'utf-8' codec can't decode byte 0xed in position 78: invalid continuation byte The exception occurs during the .delete() of Jobs, here:  Enabling debug logging of Django's DB backend, shows the generated SQL to be: SELECT FROM job WHERE (job.repositoryid = 1 AND job.submittime < '2018-10-21 11:03:32.') LIMIT 1; args=(1, '2018-10-21 11:03:32.') SELECT , failureline.jobguid, failureline.repositoryid, failureline.joblogid, , , , failureline.subtest, , failureline.expected, failureline.message, failureline.signature, , , failureline.stackwalkstdout, failureline.stackwalkstderr, failureline.bestclassificationid, failureline.bestisverified, failureline.created, failureline.modified FROM failureline WHERE failureline.jobguid IN ('-b854-4300-969a-/0'); args=('-b854-4300-969a-/0',) SELECT , job.repositoryid, , job.projectspecificid, job.autoclassifystatus, job.coalescedtoguid, job.signatureid, job.buildplatformid, job.machineplatformid, job.machineid, job.optioncollectionhash, job.jobtypeid, job.jobgroupid, job.productid, job.failureclassificationid, , , , , job.submittime, job.starttime, job.endtime, job.lastmodified, job.runningeta, , FROM job WHERE IN ('-b854-4300-969a-/0'); args=('-b854-4300-969a-/0',) SELECT , , , , FROM joblog WHERE IN (); args=(,) [2019-02-18 11:03:33,403] DEBUG [] (0.107) SELECT , failureline.jobguid, failureline.repositoryid, failureline.joblogid, , , , failureline.subtest, , failureline.expected, failureline.message, failureline.signature, , , failureline.stackwalkstdout, failureline.stackwalkstderr, failureline.bestclassificationid, failureline.bestisverified, failureline.created, failureline.modified FROM failureline WHERE failureline.joblogid IN (, ); args=(, ) SELECT , , , textlogstep.started, textlogstep.finished, textlogstep.startedlinenumber, textlogstep.finishedlinenumber, FROM textlogstep WHERE IN (); args=(,) SELECT , , , textlogerror.linenumber FROM textlogerror WHERE IN (); args=(,) Querying the textlogerror table for those ids shows there to be junk values in its line field. These are from data inserted when using Python 2.7, which presumably wasn't validating the unicode escape sequences being used. There appear to be two issues here: mysqlclient-python's behaviour differs depending on Python version - under Python 3 it defaults useunicode to True, which means it attempts to decode the line field but fails (since it doesn't use 'replace' or 'ignore'). This seems like something that the Django ORM should try to protect against (eg by setting useunicode to the same value on all Python versions and handling the unicode conversion itself), given it generally handles any implementation differences in layers lower than the ORM. the UnicodeDecodeError is occurring for a field () that is not actually needed for the .delete() (it's not a primary key etc), so Django shouldn't be fetching that field regardless when making the textlogerror SELECT query (Plus ideally Django would support cascade deletes, so we wouldn't need to use the current .delete() approach; ticket ) Fixing issue (2) would presumably also improve .delete() performance. Related:

sub_objs = self.related_objects(related, batch)                     if self.can_fast_delete(sub_objs, from_field=field):                         self.fast_deletes.append(sub_objs)                    elif sub_objs:                        field.remote_field.on_delete(self, field, sub_objs, self.using)                    else:                        related_model = related.related_model                        # Non-referenced fields can be deferred if no signal                        # receivers are connected for the related model as                        # they'll never be exposed to the user. Skip field                        # deferring when some relationships are select_related                        # as interactions between both features are hard to                        # get right. This should only happen in the rare                        # cases where .related_objects is overridden anyway.                        if not (sub_objs.query.select_related or self._has_signal_listeners(related_model)):                            referenced_fields = set(chain.from_iterable(                                (rf.attname for rf in rel.field.foreign_related_fields)                                for rel in get_candidate_relations_to_delete(related_model._meta)                            ))                            sub_objs = sub_objs.only(*tuple(referenced_fields))                        if sub_objs:                            field.remote_field.on_delete(self, field, sub_objs, self.using)             for field in model._meta.private_fields:                 if hasattr(field, 'bulk_related_objects'):                     # It's something like generic foreign key.


Question: default='' (non-bytestring) on BinaryField crashes some migration operations Description (last modified by James) Description Initial migration has a default value '' for BinaryField. Later, change default value to b'' and migrate. Trying to undo this migration fails. It seems like '' is allowed during migration, but not in reverse migration. Related issue Default value for BinaryField Reproduce Python 3.6.0, Django 1.10.6, Postgres 9.5.4 startproject djangoproject startapp firstapp class TableOne(models.Model): field1 = models.BinaryField(default = '') makemigrations firstapp migrate firstapp 0001 Modify class TableOne(models.Model): field1 = models.BinaryField(default = b'') migrate firstapp 0002 migrate firstapp 0001 Error: TypeError: can't escape str to binary Traceback (most recent call last): File "", line 22, in <moduleexecutefromcommandline() File "C:\Py\py364\lib\site-packages\django\core\management\", line 367, in executefromcommandline utility.execute() File "C:\Py\py364\lib\site-packages\django\core\management\", line 359, in execute self.fetchcommand(subcommand).runfromargv() File "C:\Py\py364\lib\site-packages\django\core\management\", line 294, in runfromargv self.execute(args, cmdoptions) File "C:\Py\py364\lib\site-packages\django\core\management\", line 345, in execute output = (args, options) File "C:\Py\py364\lib\site-packages\django\core\management\commands\", line 204, in handle fakeinitial=fakeinitial, File "C:\Py\py364\lib\site-packages\django\db\migrations\", line 119, in migrate state = self.migrateallbackwards(plan, fullplan, fake=fake) File "C:\Py\py364\lib\site-packages\django\db\migrations\", line 194, in migrateallbackwards self.unapplymigration(states[migration], migration, fake=fake) File "C:\Py\py364\lib\site-packages\django\db\migrations\", line 264, in unapplymigration state = migration.unapply(state, schemaeditor) File "C:\Py\py364\lib\site-packages\django\db\migrations\", line 178, in unapply operation.databasebackwards(self.applabel, schemaeditor, fromstate, tostate) File "C:\Py\py364\lib\site-packages\django\db\migrations\operations\", line 210, in databasebackwards self.databaseforwards(applabel, schemaeditor, fromstate, tostate) File "C:\Py\py364\lib\site-packages\django\db\migrations\operations\", line 205, in databaseforwards schemaeditor.alterfield(frommodel, fromfield, tofield) File "C:\Py\py364\lib\site-packages\django\db\backends\base\", line 506, in alterfield olddbparams, newdbparams, strict) File "C:\Py\py364\lib\site-packages\django\db\backends\postgresql\", line 118, in alterfield newdbparams, strict, File "C:\Py\py364\lib\site-packages\django\db\backends\base\", line 660, in alterfield params, File "C:\Py\py364\lib\site-packages\django\db\backends\base\", line 112, in execute cursor.execute(sql, params) File "C:\Py\py364\lib\site-packages\django\db\backends\", line 80, in execute return super(CursorDebugWrapper, self).execute(sql, params) File "C:\Py\py364\lib\site-packages\django\db\backends\", line 65, in execute return (sql, params) TypeError: can't escape str to binary Notes site-packages\django\db\backends\base\ def effectivedefault(self, field): determines default as an empty <class 'str', when (default = '') Possible Fix? site-packages\django\db\backends\base\ ~line 197 def effectivedefault(self, field): if field.hasdefault(): default = field.getdefault() if field.getinternaltype() == "BinaryField" and not default: default = six.binarytype() elif not and and field.emptystringsallowed: if field.getinternaltype() == "BinaryField": default = six.binarytype() else: default = six.texttype() elif getattr(field, 'auto_now', False)

if self.max_length is not None:             self.validators.append(validators.MaxLengthValidator(self.max_length)) <ins>    def check(self, **kwargs):        return [*super().check(**kwargs), *self._check_str_default_value()]    def _check_str_default_value(self):        if self.has_default() and isinstance(self.default, str):            return [                checks.Error(                    "BinaryField's default cannot be a string. Use bytes "                    "content instead.",                    obj=self,                    id='fields.E170',                )            ]        return []</ins>     def deconstruct(self):         name, path, args, kwargs = super().deconstruct()         if self.editable:


Question: add ModelAdmin.getinlines() hook to allow set inlines based on the request or model instance. Description add ModelAdmin.getinlines() hook to allow set inlines based on the request or model instance. Currently, We can override the method getinlineinstances to do such a thing, but a for loop should be copied to my code. So I wished add a hook get_inlines(request, obj=None)

return self.fieldsets         return [(None, {'fields': self.get_fields(request, obj)})] <ins>    def get_inlines(self, request, obj):        """Hook for specifying custom inlines."""        return self.inlines</ins>     def get_ordering(self, request):         """         Hook for specifying field ordering.


Question: add ModelAdmin.getinlines() hook to allow set inlines based on the request or model instance. Description add ModelAdmin.getinlines() hook to allow set inlines based on the request or model instance. Currently, We can override the method getinlineinstances to do such a thing, but a for loop should be copied to my code. So I wished add a hook get_inlines(request, obj=None)

def get_inline_instances(self, request, obj=None):         inline_instances = []        for inline_class in self.inlines:        for inline_class in self.get_inlines(request, obj):             inline = inline_class(self.model, self.admin_site)             if request:                 if not (inline.has_view_or_change_permission(request, obj) or


Question: Support the use of adminorderfield on properties in the admin. Description The ModelAdmin class allows to specify properties from the model in listdisplay. Normally non-fields cannot be sorted. This can be changed by setting adminorderfield. This doesn't work on properties, while it is possible to do so with shortdescription. The code that checks for shortdescription checks if it handles a property and returns if available. The files that check for adminorderfield do not handle the case that it might be a property. The relevant files are probably: (GitHub) ( GitHub)

continue              admin_order_field = getattr(attr, "admin_order_field", None)            # Set ordering for attr that is a property, if defined.            if isinstance(attr, property) and hasattr(attr, 'fget'):                admin_order_field = getattr(attr.fget, 'admin_order_field')             if not admin_order_field:                 is_field_sortable = False diff --git a/django/contrib/admin/views/main.py b/django/contrib/admin/views/main.py-- a/django/contrib/admin/views/main.py++ b/django/contrib/admin/views/main.py


Question: Support the use of adminorderfield on properties in the admin. Description The ModelAdmin class allows to specify properties from the model in listdisplay. Normally non-fields cannot be sorted. This can be changed by setting adminorderfield. This doesn't work on properties, while it is possible to do so with shortdescription. The code that checks for shortdescription checks if it handles a property and returns if available. The files that check for adminorderfield do not handle the case that it might be a property. The relevant files are probably: (GitHub) ( GitHub)

attr = getattr(self.model_admin, field_name)             else:                 attr = getattr(self.model, field_name)<ins>            if isinstance(attr, property) and hasattr(attr, 'fget'):                attr = attr.fget</ins>             return getattr(attr, 'admin_order_field', None)      def get_ordering(self, request, queryset):


Question: UsernameValidator allows trailing newline in usernames Description ASCIIUsernameValidator and UnicodeUsernameValidator use the regex r'^[\w. The intent is to only allow alphanumeric characters as well as ., +, and -. However, a little known quirk of Python regexes is that $ will also match a trailing newline. Therefore, the user name validators will accept usernames which end with a newline. You can avoid this behavior by instead using \A and \Z to terminate regexes. For example, the validator regex could be changed to r'\A[\w. in order to reject usernames that end with a newline. I am not sure how to officially post a patch, but the required change is trivial - using the regex above in the two validators in

@deconstructible class ASCIIUsernameValidator(validators.RegexValidator):    regex = r'^[\w.@+-]+$'    regex = r'^[\w.@+-]+\Z'     message = _(         'Enter a valid username. This value may contain only English letters, '         'numbers, and @/./+/-/_ characters.'


Question: UsernameValidator allows trailing newline in usernames Description ASCIIUsernameValidator and UnicodeUsernameValidator use the regex r'^[\w. The intent is to only allow alphanumeric characters as well as ., +, and -. However, a little known quirk of Python regexes is that $ will also match a trailing newline. Therefore, the user name validators will accept usernames which end with a newline. You can avoid this behavior by instead using \A and \Z to terminate regexes. For example, the validator regex could be changed to r'\A[\w. in order to reject usernames that end with a newline. I am not sure how to officially post a patch, but the required change is trivial - using the regex above in the two validators in

@deconstructible class UnicodeUsernameValidator(validators.RegexValidator):    regex = r'^[\w.@+-]+$'    regex = r'^[\w.@+-]+\Z'     message = _(         'Enter a valid username. This value may contain only letters, '         'numbers, and @/./+/-/_ characters.'


Question: Make it easier to track down the offending models for AlreadyRegistered exceptions Description I've just updated 20+ packages locally in a project's requirements file, one of which has caused: [2016-10-18 15:00:18,667] ERROR [django.request:256] Internal Server Error: /browserid/csrf/ Traceback (most recent call last): File "/home/vagrant/venv/local/lib/python2.7/site-", line 223, in getresponse response = middlewaremethod(request, response) File "/home/vagrant/venv/local/lib/python2.7/site-", line 129, in processresponse panel.generatestats(request, response) File "/home/vagrant/venv/local/lib/python2.7/site-", line 41, in generatestats match = resolve() File "/home/vagrant/venv/local/lib/python2.7/site-", line 521, in resolve return getresolver(urlconf).resolve(path) File "/home/vagrant/venv/local/lib/python2.7/site-", line 365, in resolve for pattern in self.urlpatterns: File "/home/vagrant/venv/local/lib/python2.7/site-", line 401, in urlpatterns patterns = getattr(self.urlconfmodule, "urlpatterns", self.urlconfmodule) File "/home/vagrant/venv/local/lib/python2.7/site-", line 395, in urlconfmodule self.urlconfmodule = importmodule(self.urlconfname) File "", line 37, in importmodule import(name) File "", line 12, in <modulebrowseridadmin.copyregistry() File "/home/vagrant/venv/local/lib/python2.7/site-", line 39, in copyregistry self.register(model, ) File "/home/vagrant/venv/local/lib/python2.7/site-", line 90, in register raise AlreadyRegistered('The model %s is already registered' % ) AlreadyRegistered: The model Site is already registered This model isn't defined in the project's own models so in this particular case it must be django-browserid now clashing with one of the packages that was updated. Rather than having to bisect, it would be much more helpful if this exception gave more details about the already-registered model, so I know which package/app is clashing with the later app's attempts to re-register it.

<ins>import re</ins> from functools import update_wrapper from weakref import WeakSet


Question: Make it easier to track down the offending models for AlreadyRegistered exceptions Description I've just updated 20+ packages locally in a project's requirements file, one of which has caused: [2016-10-18 15:00:18,667] ERROR [django.request:256] Internal Server Error: /browserid/csrf/ Traceback (most recent call last): File "/home/vagrant/venv/local/lib/python2.7/site-", line 223, in getresponse response = middlewaremethod(request, response) File "/home/vagrant/venv/local/lib/python2.7/site-", line 129, in processresponse panel.generatestats(request, response) File "/home/vagrant/venv/local/lib/python2.7/site-", line 41, in generatestats match = resolve() File "/home/vagrant/venv/local/lib/python2.7/site-", line 521, in resolve return getresolver(urlconf).resolve(path) File "/home/vagrant/venv/local/lib/python2.7/site-", line 365, in resolve for pattern in self.urlpatterns: File "/home/vagrant/venv/local/lib/python2.7/site-", line 401, in urlpatterns patterns = getattr(self.urlconfmodule, "urlpatterns", self.urlconfmodule) File "/home/vagrant/venv/local/lib/python2.7/site-", line 395, in urlconfmodule self.urlconfmodule = importmodule(self.urlconfname) File "", line 37, in importmodule import(name) File "", line 12, in <modulebrowseridadmin.copyregistry() File "/home/vagrant/venv/local/lib/python2.7/site-", line 39, in copyregistry self.register(model, ) File "/home/vagrant/venv/local/lib/python2.7/site-", line 90, in register raise AlreadyRegistered('The model %s is already registered' % ) AlreadyRegistered: The model Site is already registered This model isn't defined in the project's own models so in this particular case it must be django-browserid now clashing with one of the packages that was updated. Rather than having to bisect, it would be much more helpful if this exception gave more details about the already-registered model, so I know which package/app is clashing with the later app's attempts to re-register it.

)              if model in self._registry:                raise AlreadyRegistered('The model %s is already registered' % model.__name__)                registered_admin = str(self._registry[model])                msg = 'The model %s is already registered ' % model.__name__                if registered_admin.endswith('.ModelAdmin'):                    # Most likely registered without a ModelAdmin subclass.                    msg += 'in app %r.' % re.sub(r'\.ModelAdmin$', '', registered_admin)                else:                    msg += 'with %r.' % registered_admin                raise AlreadyRegistered(msg)              # Ignore the registration if the model has been             # swapped out.


Question: Engine.rendertostring() should honor the autoescape attribute Description In Engine.rendertostring, a Context is created without specifying the engine autoescape attribute. So if you create en engine with autoescape=False and then call its rendertostring() method, the result will always be autoescaped. It was probably overlooked in [].

if isinstance(context, Context):             return t.render(context)         else:            return t.render(Context(context))            return t.render(Context(context, autoescape=self.autoescape))      def select_template(self, template_name_list):         """


Question: Incorrect error message with () that requires a join on an annotated F expression Description (last modified by Asif Saifuddin Auvi) I ran into a bit of a nasty error yesterday on Django 1.11, Postgres 9.5 where I was trying to do an update using an F expression where the value in that expression referred to an annotation of an annotation, each using a Case statement inside the annotation. The error message it was giving me was something like the following: ProgrammingError Traceback (most recent call last) <ipython-input-1-29e48364245cin <module() ----1 MyModel.objects.causingproblems() in causingproblems(self) 833 ).update( --834 myfield=F('myannotatedfield'), 835 ) 836 in update(self, **kwargs) 645 query.annotations = None 646 with (, savepoint=False): --647 rows = query.getcompiler().executesql(CURSOR) 648 self.resultcache = None 649 return rows in executesql(self, resulttype) 1189 related queries are not available. 1190 """ -1191 cursor = super(SQLUpdateCompiler, self).executesql(resulttype) 1192 try: 1193 rows = cursor.rowcount if cursor else 0 in executesql(self, resulttype, chunkedfetch) 884 # silencing when dropping Python 2 compatibility. 885 pass --886 raise originalexception 887 888 if resulttype == CURSOR: ProgrammingError: missing FROM-clause entry for table "myappmymodel" LINE 1: END, "afield" = CASE WHEN CASE WHEN ("myappmy... Spent a while trying to debug where it was going wrong, ended up narrowing down the problem to trying to update a field using an F expression on an annotated field which included a join in it. I'm not sure if better error message would have helped in my case but in the much simpler test attached it'd be useful if the first and second assertions raised the same or similar error message.

self.unref_alias(joins.pop())         return targets, joins[-1], joins     @classmethod    def _gen_col_aliases(cls, exprs):        for expr in exprs:            if isinstance(expr, Col):                yield expr.alias            else:                yield from cls._gen_col_aliases(expr.get_source_expressions())     def resolve_ref(self, name, allow_joins=True, reuse=None, summarize=False, simple_col=False):         if not allow_joins and LOOKUP_SEP in name:             raise FieldError("Joined field references are not permitted in this query")        if name in self.annotations:        annotation = self.annotations.get(name)        if annotation is not None:            if not allow_joins:                for alias in self._gen_col_aliases([annotation]):                    if isinstance(self.alias_map[alias], Join):                        raise FieldError(                            'Joined field references are not permitted in '                            'this query'                        )             if summarize:                 # Summarize currently means we are doing an aggregate() query                 # which is executed as a wrapped subquery if any of the


Question: Incorrect error message with () that requires a join on an annotated F expression Description (last modified by Asif Saifuddin Auvi) I ran into a bit of a nasty error yesterday on Django 1.11, Postgres 9.5 where I was trying to do an update using an F expression where the value in that expression referred to an annotation of an annotation, each using a Case statement inside the annotation. The error message it was giving me was something like the following: ProgrammingError Traceback (most recent call last) <ipython-input-1-29e48364245cin <module() ----1 MyModel.objects.causingproblems() in causingproblems(self) 833 ).update( --834 myfield=F('myannotatedfield'), 835 ) 836 in update(self, **kwargs) 645 query.annotations = None 646 with (, savepoint=False): --647 rows = query.getcompiler().executesql(CURSOR) 648 self.resultcache = None 649 return rows in executesql(self, resulttype) 1189 related queries are not available. 1190 """ -1191 cursor = super(SQLUpdateCompiler, self).executesql(resulttype) 1192 try: 1193 rows = cursor.rowcount if cursor else 0 in executesql(self, resulttype, chunkedfetch) 884 # silencing when dropping Python 2 compatibility. 885 pass --886 raise originalexception 887 888 if resulttype == CURSOR: ProgrammingError: missing FROM-clause entry for table "myappmymodel" LINE 1: END, "afield" = CASE WHEN CASE WHEN ("myappmy... Spent a while trying to debug where it was going wrong, ended up narrowing down the problem to trying to update a field using an F expression on an annotated field which included a join in it. I'm not sure if better error message would have helped in my case but in the much simpler test attached it'd be useful if the first and second assertions raised the same or similar error message.

# that case we need to return a Ref to the subquery's annotation.                 return Ref(name, self.annotation_select[name])             else:                return self.annotations[name]                return annotation         else:             field_list = name.split(LOOKUP_SEP)             join_info = self.setup_joins(field_list, self.get_meta(), self.get_initial_alias(), can_reuse=reuse)


Question: HttpResponse doesn't handle memoryview objects Description I am trying to write a BinaryField retrieved from the database into a HttpResponse. When the database is Sqlite this works correctly, but Postgresql returns the contents of the field as a memoryview object and it seems like current Django doesn't like this combination: from import HttpResponse response = HttpResponse("My Content") response.content response = HttpResponse(b"My Content") response.content response = HttpResponse(memoryview(b"My Content")) response.content

# Handle string types -- we can't rely on force_bytes here because:         # - Python attempts str conversion first         # - when self._charset != 'utf-8' it re-encodes the content        if isinstance(value, bytes):        if isinstance(value, (bytes, memoryview)):             return bytes(value)         if isinstance(value, str):             return bytes(value.encode(self.charset))


Question: TIMEZONE value in DATABASES settings is not used when making dates timezone-aware on MySQL, SQLite, and Oracle. Description (last modified by Victor Talpaert) (We assume the mysql backends) I can set TIMEZONE several times in , one for the global django app, and one for each database (see  (ref1)) Typical usage would be for a legacy database where datetimes are not stored in UTC. No date lookup Querying my database takes this setting into account, e.g. : In USETZ = True TIMEZONE = 'Europe/Paris' # tz1 DATABASES = { 'legacy': { 'ENGINE': '', 'OPTIONS': { 'readdefaultfile': '', }, 'TIMEZONE': 'Europe/Paris', # tz2 }, 'default' : { 'ENGINE': '', 'OPTIONS': { 'readdefaultfile': '', }, } } In the shell dt = timezone.makeaware(datetime.datetime(2017, 7, 6, 20, 50)) dt datetime.datetime(2017, 7, 6, 20, 50, tzinfo=<DstTzInfo 'Europe/Paris' CEST+2:00:00 DST) (mydatetimefield=dt).exists() True This works because my database reads '2017-07-06 20:50:00' With date lookup Related doc  (ref2) But this does not work, while it logically should (()).exists() False The related SQL query from DEBUG is : SELECT (1) AS FROM WHERE DATE(CONVERTTZ(., 'UTC', 'Europe/Paris')) = '2017-07-06' LIMIT 1; () Note that I haven't filled the timezone table in MySQL, so the result should be True in this case, but could be False close to midnight. Related doc is  Two things are wrong. First, conversion should be from Paris to Paris, instead of UTC to Paris. The conversion should go from the database timezone tz2 to the django app one tz1. Indeed from ref1 and ref2: When USETZ is True and the database doesn’t support time zones (e.g. SQLite, MySQL, Oracle), Django reads and writes datetimes in local time according to this option if it is set and in UTC if it isn’t. When USETZ is True, fields are converted to the current time zone before filtering Secondly, when tz1 == tz2, there should be no need to use CONVERTTZ and the query will work without timezone tables in MySQL. The explicit queries are : mysqlSELECT (1) AS FROM WHERE . = '2017-07-06 20:50:00' LIMIT 1; +---+ +---+ +---+ 1 row in set (0.00 sec) mysqlSELECT (1) AS FROM WHERE DATE(.) = '2017-07-06' LIMIT 1; +---+ +---+ +---+ 1 row in set (0.00 sec) I understand that the date lookup can have some history, but I find the behaviour illogical and undesired. Would you agree there is a problem here? EDIT : line where 'UTC' is forced disregarding the database setting  PS: stackoverflow question

return "DATE(%s)" % (field_name)      def _convert_field_to_tz(self, field_name, tzname):        if settings.USE_TZ:            field_name = "CONVERT_TZ(%s, 'UTC', '%s')" % (field_name, tzname)        if settings.USE_TZ and self.connection.timezone_name != tzname:            field_name = "CONVERT_TZ(%s, '%s', '%s')" % (field_name, self.connection.timezone_name, tzname)         return field_name      def datetime_cast_date_sql(self, field_name, tzname):diff --git a/django/db/backends/oracle/operations.py b/django/db/backends/oracle/operations.py-- a/django/db/backends/oracle/operations.py++ b/django/db/backends/oracle/operations.py


Question: TIMEZONE value in DATABASES settings is not used when making dates timezone-aware on MySQL, SQLite, and Oracle. Description (last modified by Victor Talpaert) (We assume the mysql backends) I can set TIMEZONE several times in , one for the global django app, and one for each database (see  (ref1)) Typical usage would be for a legacy database where datetimes are not stored in UTC. No date lookup Querying my database takes this setting into account, e.g. : In USETZ = True TIMEZONE = 'Europe/Paris' # tz1 DATABASES = { 'legacy': { 'ENGINE': '', 'OPTIONS': { 'readdefaultfile': '', }, 'TIMEZONE': 'Europe/Paris', # tz2 }, 'default' : { 'ENGINE': '', 'OPTIONS': { 'readdefaultfile': '', }, } } In the shell dt = timezone.makeaware(datetime.datetime(2017, 7, 6, 20, 50)) dt datetime.datetime(2017, 7, 6, 20, 50, tzinfo=<DstTzInfo 'Europe/Paris' CEST+2:00:00 DST) (mydatetimefield=dt).exists() True This works because my database reads '2017-07-06 20:50:00' With date lookup Related doc  (ref2) But this does not work, while it logically should (()).exists() False The related SQL query from DEBUG is : SELECT (1) AS FROM WHERE DATE(CONVERTTZ(., 'UTC', 'Europe/Paris')) = '2017-07-06' LIMIT 1; () Note that I haven't filled the timezone table in MySQL, so the result should be True in this case, but could be False close to midnight. Related doc is  Two things are wrong. First, conversion should be from Paris to Paris, instead of UTC to Paris. The conversion should go from the database timezone tz2 to the django app one tz1. Indeed from ref1 and ref2: When USETZ is True and the database doesn’t support time zones (e.g. SQLite, MySQL, Oracle), Django reads and writes datetimes in local time according to this option if it is set and in UTC if it isn’t. When USETZ is True, fields are converted to the current time zone before filtering Secondly, when tz1 == tz2, there should be no need to use CONVERTTZ and the query will work without timezone tables in MySQL. The explicit queries are : mysqlSELECT (1) AS FROM WHERE . = '2017-07-06 20:50:00' LIMIT 1; +---+ +---+ +---+ 1 row in set (0.00 sec) mysqlSELECT (1) AS FROM WHERE DATE(.) = '2017-07-06' LIMIT 1; +---+ +---+ +---+ 1 row in set (0.00 sec) I understand that the date lookup can have some history, but I find the behaviour illogical and undesired. Would you agree there is a problem here? EDIT : line where 'UTC' is forced disregarding the database setting  PS: stackoverflow question

return field_name         if not self._tzname_re.match(tzname):             raise ValueError("Invalid time zone name: %s" % tzname)        # Convert from UTC to local time, returning TIMESTAMP WITH TIME ZONE        # and cast it back to TIMESTAMP to strip the TIME ZONE details.        return "CAST((FROM_TZ(%s, '0:00') AT TIME ZONE '%s') AS TIMESTAMP)" % (field_name, tzname)        # Convert from connection timezone to the local time, returning        # TIMESTAMP WITH TIME ZONE and cast it back to TIMESTAMP to strip the        # TIME ZONE details.        if self.connection.timezone_name != tzname:            return "CAST((FROM_TZ(%s, '%s') AT TIME ZONE '%s') AS TIMESTAMP)" % (                field_name,                self.connection.timezone_name,                tzname,            )        return field_name      def datetime_cast_date_sql(self, field_name, tzname):         field_name = self._convert_field_to_tz(field_name, tzname)diff --git a/django/db/backends/sqlite3/base.py b/django/db/backends/sqlite3/base.py-- a/django/db/backends/sqlite3/base.py++ b/django/db/backends/sqlite3/base.py


Question: TIMEZONE value in DATABASES settings is not used when making dates timezone-aware on MySQL, SQLite, and Oracle. Description (last modified by Victor Talpaert) (We assume the mysql backends) I can set TIMEZONE several times in , one for the global django app, and one for each database (see  (ref1)) Typical usage would be for a legacy database where datetimes are not stored in UTC. No date lookup Querying my database takes this setting into account, e.g. : In USETZ = True TIMEZONE = 'Europe/Paris' # tz1 DATABASES = { 'legacy': { 'ENGINE': '', 'OPTIONS': { 'readdefaultfile': '', }, 'TIMEZONE': 'Europe/Paris', # tz2 }, 'default' : { 'ENGINE': '', 'OPTIONS': { 'readdefaultfile': '', }, } } In the shell dt = timezone.makeaware(datetime.datetime(2017, 7, 6, 20, 50)) dt datetime.datetime(2017, 7, 6, 20, 50, tzinfo=<DstTzInfo 'Europe/Paris' CEST+2:00:00 DST) (mydatetimefield=dt).exists() True This works because my database reads '2017-07-06 20:50:00' With date lookup Related doc  (ref2) But this does not work, while it logically should (()).exists() False The related SQL query from DEBUG is : SELECT (1) AS FROM WHERE DATE(CONVERTTZ(., 'UTC', 'Europe/Paris')) = '2017-07-06' LIMIT 1; () Note that I haven't filled the timezone table in MySQL, so the result should be True in this case, but could be False close to midnight. Related doc is  Two things are wrong. First, conversion should be from Paris to Paris, instead of UTC to Paris. The conversion should go from the database timezone tz2 to the django app one tz1. Indeed from ref1 and ref2: When USETZ is True and the database doesn’t support time zones (e.g. SQLite, MySQL, Oracle), Django reads and writes datetimes in local time according to this option if it is set and in UTC if it isn’t. When USETZ is True, fields are converted to the current time zone before filtering Secondly, when tz1 == tz2, there should be no need to use CONVERTTZ and the query will work without timezone tables in MySQL. The explicit queries are : mysqlSELECT (1) AS FROM WHERE . = '2017-07-06 20:50:00' LIMIT 1; +---+ +---+ +---+ 1 row in set (0.00 sec) mysqlSELECT (1) AS FROM WHERE DATE(.) = '2017-07-06' LIMIT 1; +---+ +---+ +---+ 1 row in set (0.00 sec) I understand that the date lookup can have some history, but I find the behaviour illogical and undesired. Would you agree there is a problem here? EDIT : line where 'UTC' is forced disregarding the database setting  PS: stackoverflow question

conn = Database.connect(**conn_params)         conn.create_function("django_date_extract", 2, _sqlite_datetime_extract)         conn.create_function("django_date_trunc", 2, _sqlite_date_trunc)        conn.create_function("django_datetime_cast_date", 2, _sqlite_datetime_cast_date)        conn.create_function("django_datetime_cast_time", 2, _sqlite_datetime_cast_time)        conn.create_function("django_datetime_extract", 3, _sqlite_datetime_extract)        conn.create_function("django_datetime_trunc", 3, _sqlite_datetime_trunc)        conn.create_function('django_datetime_cast_date', 3, _sqlite_datetime_cast_date)        conn.create_function('django_datetime_cast_time', 3, _sqlite_datetime_cast_time)        conn.create_function('django_datetime_extract', 4, _sqlite_datetime_extract)        conn.create_function('django_datetime_trunc', 4, _sqlite_datetime_trunc)         conn.create_function("django_time_extract", 2, _sqlite_time_extract)         conn.create_function("django_time_trunc", 2, _sqlite_time_trunc)         conn.create_function("django_time_diff", 2, _sqlite_time_diff)


Question: TIMEZONE value in DATABASES settings is not used when making dates timezone-aware on MySQL, SQLite, and Oracle. Description (last modified by Victor Talpaert) (We assume the mysql backends) I can set TIMEZONE several times in , one for the global django app, and one for each database (see  (ref1)) Typical usage would be for a legacy database where datetimes are not stored in UTC. No date lookup Querying my database takes this setting into account, e.g. : In USETZ = True TIMEZONE = 'Europe/Paris' # tz1 DATABASES = { 'legacy': { 'ENGINE': '', 'OPTIONS': { 'readdefaultfile': '', }, 'TIMEZONE': 'Europe/Paris', # tz2 }, 'default' : { 'ENGINE': '', 'OPTIONS': { 'readdefaultfile': '', }, } } In the shell dt = timezone.makeaware(datetime.datetime(2017, 7, 6, 20, 50)) dt datetime.datetime(2017, 7, 6, 20, 50, tzinfo=<DstTzInfo 'Europe/Paris' CEST+2:00:00 DST) (mydatetimefield=dt).exists() True This works because my database reads '2017-07-06 20:50:00' With date lookup Related doc  (ref2) But this does not work, while it logically should (()).exists() False The related SQL query from DEBUG is : SELECT (1) AS FROM WHERE DATE(CONVERTTZ(., 'UTC', 'Europe/Paris')) = '2017-07-06' LIMIT 1; () Note that I haven't filled the timezone table in MySQL, so the result should be True in this case, but could be False close to midnight. Related doc is  Two things are wrong. First, conversion should be from Paris to Paris, instead of UTC to Paris. The conversion should go from the database timezone tz2 to the django app one tz1. Indeed from ref1 and ref2: When USETZ is True and the database doesn’t support time zones (e.g. SQLite, MySQL, Oracle), Django reads and writes datetimes in local time according to this option if it is set and in UTC if it isn’t. When USETZ is True, fields are converted to the current time zone before filtering Secondly, when tz1 == tz2, there should be no need to use CONVERTTZ and the query will work without timezone tables in MySQL. The explicit queries are : mysqlSELECT (1) AS FROM WHERE . = '2017-07-06 20:50:00' LIMIT 1; +---+ +---+ +---+ 1 row in set (0.00 sec) mysqlSELECT (1) AS FROM WHERE DATE(.) = '2017-07-06' LIMIT 1; +---+ +---+ +---+ 1 row in set (0.00 sec) I understand that the date lookup can have some history, but I find the behaviour illogical and undesired. Would you agree there is a problem here? EDIT : line where 'UTC' is forced disregarding the database setting  PS: stackoverflow question

return FORMAT_QMARK_REGEX.sub('?', query).replace('%%', '%')  def _sqlite_datetime_parse(dt, tzname=None):def _sqlite_datetime_parse(dt, tzname=None, conn_tzname=None):     if dt is None:         return None     try:         dt = backend_utils.typecast_timestamp(dt)     except (TypeError, ValueError):         return None    if tzname is not None:    if conn_tzname:        dt = dt.replace(tzinfo=pytz.timezone(conn_tzname))    if tzname is not None and tzname != conn_tzname:         dt = timezone.localtime(dt, pytz.timezone(tzname))     return dt


Question: TIMEZONE value in DATABASES settings is not used when making dates timezone-aware on MySQL, SQLite, and Oracle. Description (last modified by Victor Talpaert) (We assume the mysql backends) I can set TIMEZONE several times in , one for the global django app, and one for each database (see  (ref1)) Typical usage would be for a legacy database where datetimes are not stored in UTC. No date lookup Querying my database takes this setting into account, e.g. : In USETZ = True TIMEZONE = 'Europe/Paris' # tz1 DATABASES = { 'legacy': { 'ENGINE': '', 'OPTIONS': { 'readdefaultfile': '', }, 'TIMEZONE': 'Europe/Paris', # tz2 }, 'default' : { 'ENGINE': '', 'OPTIONS': { 'readdefaultfile': '', }, } } In the shell dt = timezone.makeaware(datetime.datetime(2017, 7, 6, 20, 50)) dt datetime.datetime(2017, 7, 6, 20, 50, tzinfo=<DstTzInfo 'Europe/Paris' CEST+2:00:00 DST) (mydatetimefield=dt).exists() True This works because my database reads '2017-07-06 20:50:00' With date lookup Related doc  (ref2) But this does not work, while it logically should (()).exists() False The related SQL query from DEBUG is : SELECT (1) AS FROM WHERE DATE(CONVERTTZ(., 'UTC', 'Europe/Paris')) = '2017-07-06' LIMIT 1; () Note that I haven't filled the timezone table in MySQL, so the result should be True in this case, but could be False close to midnight. Related doc is  Two things are wrong. First, conversion should be from Paris to Paris, instead of UTC to Paris. The conversion should go from the database timezone tz2 to the django app one tz1. Indeed from ref1 and ref2: When USETZ is True and the database doesn’t support time zones (e.g. SQLite, MySQL, Oracle), Django reads and writes datetimes in local time according to this option if it is set and in UTC if it isn’t. When USETZ is True, fields are converted to the current time zone before filtering Secondly, when tz1 == tz2, there should be no need to use CONVERTTZ and the query will work without timezone tables in MySQL. The explicit queries are : mysqlSELECT (1) AS FROM WHERE . = '2017-07-06 20:50:00' LIMIT 1; +---+ +---+ +---+ 1 row in set (0.00 sec) mysqlSELECT (1) AS FROM WHERE DATE(.) = '2017-07-06' LIMIT 1; +---+ +---+ +---+ 1 row in set (0.00 sec) I understand that the date lookup can have some history, but I find the behaviour illogical and undesired. Would you agree there is a problem here? EDIT : line where 'UTC' is forced disregarding the database setting  PS: stackoverflow question

return "%02i:%02i:%02i" % (dt.hour, dt.minute, dt.second)  def _sqlite_datetime_cast_date(dt, tzname):    dt = _sqlite_datetime_parse(dt, tzname)def _sqlite_datetime_cast_date(dt, tzname, conn_tzname):    dt = _sqlite_datetime_parse(dt, tzname, conn_tzname)     if dt is None:         return None     return dt.date().isoformat()  def _sqlite_datetime_cast_time(dt, tzname):    dt = _sqlite_datetime_parse(dt, tzname)def _sqlite_datetime_cast_time(dt, tzname, conn_tzname):    dt = _sqlite_datetime_parse(dt, tzname, conn_tzname)     if dt is None:         return None     return dt.time().isoformat()  def _sqlite_datetime_extract(lookup_type, dt, tzname=None):    dt = _sqlite_datetime_parse(dt, tzname)def _sqlite_datetime_extract(lookup_type, dt, tzname=None, conn_tzname=None):    dt = _sqlite_datetime_parse(dt, tzname, conn_tzname)     if dt is None:         return None     if lookup_type == 'week_day':


Question: TIMEZONE value in DATABASES settings is not used when making dates timezone-aware on MySQL, SQLite, and Oracle. Description (last modified by Victor Talpaert) (We assume the mysql backends) I can set TIMEZONE several times in , one for the global django app, and one for each database (see  (ref1)) Typical usage would be for a legacy database where datetimes are not stored in UTC. No date lookup Querying my database takes this setting into account, e.g. : In USETZ = True TIMEZONE = 'Europe/Paris' # tz1 DATABASES = { 'legacy': { 'ENGINE': '', 'OPTIONS': { 'readdefaultfile': '', }, 'TIMEZONE': 'Europe/Paris', # tz2 }, 'default' : { 'ENGINE': '', 'OPTIONS': { 'readdefaultfile': '', }, } } In the shell dt = timezone.makeaware(datetime.datetime(2017, 7, 6, 20, 50)) dt datetime.datetime(2017, 7, 6, 20, 50, tzinfo=<DstTzInfo 'Europe/Paris' CEST+2:00:00 DST) (mydatetimefield=dt).exists() True This works because my database reads '2017-07-06 20:50:00' With date lookup Related doc  (ref2) But this does not work, while it logically should (()).exists() False The related SQL query from DEBUG is : SELECT (1) AS FROM WHERE DATE(CONVERTTZ(., 'UTC', 'Europe/Paris')) = '2017-07-06' LIMIT 1; () Note that I haven't filled the timezone table in MySQL, so the result should be True in this case, but could be False close to midnight. Related doc is  Two things are wrong. First, conversion should be from Paris to Paris, instead of UTC to Paris. The conversion should go from the database timezone tz2 to the django app one tz1. Indeed from ref1 and ref2: When USETZ is True and the database doesn’t support time zones (e.g. SQLite, MySQL, Oracle), Django reads and writes datetimes in local time according to this option if it is set and in UTC if it isn’t. When USETZ is True, fields are converted to the current time zone before filtering Secondly, when tz1 == tz2, there should be no need to use CONVERTTZ and the query will work without timezone tables in MySQL. The explicit queries are : mysqlSELECT (1) AS FROM WHERE . = '2017-07-06 20:50:00' LIMIT 1; +---+ +---+ +---+ 1 row in set (0.00 sec) mysqlSELECT (1) AS FROM WHERE DATE(.) = '2017-07-06' LIMIT 1; +---+ +---+ +---+ 1 row in set (0.00 sec) I understand that the date lookup can have some history, but I find the behaviour illogical and undesired. Would you agree there is a problem here? EDIT : line where 'UTC' is forced disregarding the database setting  PS: stackoverflow question

return getattr(dt, lookup_type)  def _sqlite_datetime_trunc(lookup_type, dt, tzname):    dt = _sqlite_datetime_parse(dt, tzname)def _sqlite_datetime_trunc(lookup_type, dt, tzname, conn_tzname):    dt = _sqlite_datetime_parse(dt, tzname, conn_tzname)     if dt is None:         return None     if lookup_type == 'year':diff --git a/django/db/backends/sqlite3/operations.py b/django/db/backends/sqlite3/operations.py-- a/django/db/backends/sqlite3/operations.py++ b/django/db/backends/sqlite3/operations.py


Question: TIMEZONE value in DATABASES settings is not used when making dates timezone-aware on MySQL, SQLite, and Oracle. Description (last modified by Victor Talpaert) (We assume the mysql backends) I can set TIMEZONE several times in , one for the global django app, and one for each database (see  (ref1)) Typical usage would be for a legacy database where datetimes are not stored in UTC. No date lookup Querying my database takes this setting into account, e.g. : In USETZ = True TIMEZONE = 'Europe/Paris' # tz1 DATABASES = { 'legacy': { 'ENGINE': '', 'OPTIONS': { 'readdefaultfile': '', }, 'TIMEZONE': 'Europe/Paris', # tz2 }, 'default' : { 'ENGINE': '', 'OPTIONS': { 'readdefaultfile': '', }, } } In the shell dt = timezone.makeaware(datetime.datetime(2017, 7, 6, 20, 50)) dt datetime.datetime(2017, 7, 6, 20, 50, tzinfo=<DstTzInfo 'Europe/Paris' CEST+2:00:00 DST) (mydatetimefield=dt).exists() True This works because my database reads '2017-07-06 20:50:00' With date lookup Related doc  (ref2) But this does not work, while it logically should (()).exists() False The related SQL query from DEBUG is : SELECT (1) AS FROM WHERE DATE(CONVERTTZ(., 'UTC', 'Europe/Paris')) = '2017-07-06' LIMIT 1; () Note that I haven't filled the timezone table in MySQL, so the result should be True in this case, but could be False close to midnight. Related doc is  Two things are wrong. First, conversion should be from Paris to Paris, instead of UTC to Paris. The conversion should go from the database timezone tz2 to the django app one tz1. Indeed from ref1 and ref2: When USETZ is True and the database doesn’t support time zones (e.g. SQLite, MySQL, Oracle), Django reads and writes datetimes in local time according to this option if it is set and in UTC if it isn’t. When USETZ is True, fields are converted to the current time zone before filtering Secondly, when tz1 == tz2, there should be no need to use CONVERTTZ and the query will work without timezone tables in MySQL. The explicit queries are : mysqlSELECT (1) AS FROM WHERE . = '2017-07-06 20:50:00' LIMIT 1; +---+ +---+ +---+ 1 row in set (0.00 sec) mysqlSELECT (1) AS FROM WHERE DATE(.) = '2017-07-06' LIMIT 1; +---+ +---+ +---+ 1 row in set (0.00 sec) I understand that the date lookup can have some history, but I find the behaviour illogical and undesired. Would you agree there is a problem here? EDIT : line where 'UTC' is forced disregarding the database setting  PS: stackoverflow question

def time_trunc_sql(self, lookup_type, field_name):         return "django_time_trunc('%s', %s)" % (lookup_type.lower(), field_name)     def _convert_tzname_to_sql(self, tzname):        return "'%s'" % tzname if settings.USE_TZ else 'NULL'    def _convert_tznames_to_sql(self, tzname):        if settings.USE_TZ:            return "'%s'" % tzname, "'%s'" % self.connection.timezone_name        return 'NULL', 'NULL'      def datetime_cast_date_sql(self, field_name, tzname):        return "django_datetime_cast_date(%s, %s)" % (            field_name, self._convert_tzname_to_sql(tzname),        return 'django_datetime_cast_date(%s, %s, %s)' % (            field_name, *self._convert_tznames_to_sql(tzname),         )      def datetime_cast_time_sql(self, field_name, tzname):        return "django_datetime_cast_time(%s, %s)" % (            field_name, self._convert_tzname_to_sql(tzname),        return 'django_datetime_cast_time(%s, %s, %s)' % (            field_name, *self._convert_tznames_to_sql(tzname),         )      def datetime_extract_sql(self, lookup_type, field_name, tzname):        return "django_datetime_extract('%s', %s, %s)" % (            lookup_type.lower(), field_name, self._convert_tzname_to_sql(tzname),        return "django_datetime_extract('%s', %s, %s, %s)" % (            lookup_type.lower(), field_name, *self._convert_tznames_to_sql(tzname),         )      def datetime_trunc_sql(self, lookup_type, field_name, tzname):        return "django_datetime_trunc('%s', %s, %s)" % (            lookup_type.lower(), field_name, self._convert_tzname_to_sql(tzname),        return "django_datetime_trunc('%s', %s, %s, %s)" % (            lookup_type.lower(), field_name, *self._convert_tznames_to_sql(tzname),         )      def time_extract_sql(self, lookup_type, field_name):


Question: Allow migrations directories without files Description (last modified by Tim Graham) Background: In python 3 a package with no is implicitly a namespace package, so it has no file attribute. The migrate command currently checks for existence of a file attribute on the migrations package. This check was introduced in , because the file attribute was used in migration file discovery. However, in migration file discovery was changed to use pkgutil.itermodules (), instead of direct filesystem access. pkgutil. itermodules() uses the package's path list, which exists on implicit namespace packages. As a result, the file check is no longer needed, and in fact prevents migrate from working on namespace packages (implicit or otherwise). Related work:

continue                 raise             else:<del>                # Empty directories are namespaces.                # getattr() needed on PY36 and older (replace w/attribute access).                if getattr(module, '__file__', None) is None:                    self.unmigrated_apps.add(app_config.label)                    continue</del>                 # Module is not a package (e.g. migrations.py).                 if not hasattr(module, '__path__'):                     self.unmigrated_apps.add(app_config.label)


Question: Allow migrations directories without files Description (last modified by Tim Graham) Background: In python 3 a package with no is implicitly a namespace package, so it has no file attribute. The migrate command currently checks for existence of a file attribute on the migrations package. This check was introduced in , because the file attribute was used in migration file discovery. However, in migration file discovery was changed to use pkgutil.itermodules (), instead of direct filesystem access. pkgutil. itermodules() uses the package's path list, which exists on implicit namespace packages. As a result, the file check is no longer needed, and in fact prevents migrate from working on namespace packages (implicit or otherwise). Related work:

# Force a reload if it's already loaded (tests need this)                 if was_loaded:                     reload(module)            self.migrated_apps.add(app_config.label)             migration_names = {                 name for _, name, is_pkg in pkgutil.iter_modules(module.__path__)                 if not is_pkg and name[0] not in '_~'             }            if migration_names or self.ignore_no_migrations:                self.migrated_apps.add(app_config.label)            else:                self.unmigrated_apps.add(app_config.label)             # Load migrations             for migration_name in migration_names:                 migration_path = '%s.%s' % (module_name, migration_name)


Question: Admin inlines for auto-created ManyToManyFields are editable if the user only has the view permission Description From Replying to Will Gordon: This seems to have regressed in (at least) 2.1. I have 2 view only permissions. I have a ManyToManyField represented in my main model as a TabularInline. But, my user with view only permissions can now add or remove these items at will! I am having the same issue, so I assume this is a bug. I did not find Will had created a separate ticket. class Photo(models.Model): pass class Report(models.Model): photos = models.ManyToManyField(Photo) class ReportPhotoInlineModelAdmin(admin.TabularInline): model = showchangelink = True

queryset = queryset.none()         return queryset     def _has_any_perms_for_target_model(self, request, perms):        """        This method is called only when the ModelAdmin's model is for an        ManyToManyField's implicit through model (if self.opts.auto_created).        Return True if the user has any of the given permissions ('add',        'change', etc.) for the model that points to the through model.        """        opts = self.opts        # Find the target model of an auto-created many-to-many relationship.        for field in opts.fields:            if field.remote_field and field.remote_field.model != self.parent_model:                opts = field.remote_field.model._meta                break        return any(            request.user.has_perm('%s.%s' % (opts.app_label, get_permission_codename(perm, opts)))            for perm in perms        )     def has_add_permission(self, request, obj):         if self.opts.auto_created:            # We're checking the rights to an auto-created intermediate model,            # which doesn't have its own individual permissions. The user needs            # to have the view permission for the related model in order to            # be able to do anything with the intermediate model.            return self.has_view_permission(request, obj)            # Auto-created intermediate models don't have their own            # permissions. The user needs to have the change permission for the            # related model in order to be able to do anything with the            # intermediate model.            return self._has_any_perms_for_target_model(request, ['change'])         return super().has_add_permission(request)      def has_change_permission(self, request, obj=None):         if self.opts.auto_created:            # We're checking the rights to an auto-created intermediate model,            # which doesn't have its own individual permissions. The user needs            # to have the view permission for the related model in order to            # be able to do anything with the intermediate model.            return self.has_view_permission(request, obj)            # Same comment as has_add_permission().            return self._has_any_perms_for_target_model(request, ['change'])         return super().has_change_permission(request)      def has_delete_permission(self, request, obj=None):         if self.opts.auto_created:            # We're checking the rights to an auto-created intermediate model,            # which doesn't have its own individual permissions. The user needs            # to have the view permission for the related model in order to            # be able to do anything with the intermediate model.            return self.has_view_permission(request, obj)            # Same comment as has_add_permission().            return self._has_any_perms_for_target_model(request, ['change'])         return super().has_delete_permission(request, obj)      def has_view_permission(self, request, obj=None):         if self.opts.auto_created:            opts = self.opts            # The model was auto-created as intermediary for a many-to-many            # Many-relationship; find the target model.            for field in opts.fields:                if field.remote_field and field.remote_field.model != self.parent_model:                    opts = field.remote_field.model._meta                    break            return (                request.user.has_perm('%s.%s' % (opts.app_label, get_permission_codename('view', opts))) or                request.user.has_perm('%s.%s' % (opts.app_label, get_permission_codename('change', opts)))            )            # Same comment as has_add_permission(). The 'change' permission            # also implies the 'view' permission.            return self._has_any_perms_for_target_model(request, ['view', 'change'])         return super().has_view_permission(request)


Question: Support setting Secure, HttpOnly, SameSite on the language cookie Description I propose to add the following settings, with the following default values: LANGUAGECOOKIESECURE = False LANGUAGECOOKIEHTTPONLY = False LANGUAGECOOKIESAMESITE = None The default values maintain the current behavior. These settings do not provide much security value, since the language is not secret or sensitive. This was also discussed briefly here:  The reasons I'd like to add them are: Sometimes auditors require them. I personally prefer to set them unless I have a reason not to. Browsers are starting to strongly nudge toward HttpOnly and Secure when possible, e.g.

LANGUAGE_COOKIE_AGE = None LANGUAGE_COOKIE_DOMAIN = None LANGUAGE_COOKIE_PATH = '/'LANGUAGE_COOKIE_SECURE = FalseLANGUAGE_COOKIE_HTTPONLY = FalseLANGUAGE_COOKIE_SAMESITE = None   # If you set this to True, Django will format dates, numbers and calendarsdiff --git a/django/views/i18n.py b/django/views/i18n.py-- a/django/views/i18n.py++ b/django/views/i18n.py


Question: Support setting Secure, HttpOnly, SameSite on the language cookie Description I propose to add the following settings, with the following default values: LANGUAGECOOKIESECURE = False LANGUAGECOOKIEHTTPONLY = False LANGUAGECOOKIESAMESITE = None The default values maintain the current behavior. These settings do not provide much security value, since the language is not secret or sensitive. This was also discussed briefly here:  The reasons I'd like to add them are: Sometimes auditors require them. I personally prefer to set them unless I have a reason not to. Browsers are starting to strongly nudge toward HttpOnly and Secure when possible, e.g.

max_age=settings.LANGUAGE_COOKIE_AGE,                 path=settings.LANGUAGE_COOKIE_PATH,                 domain=settings.LANGUAGE_COOKIE_DOMAIN,<ins>                secure=settings.LANGUAGE_COOKIE_SECURE,                httponly=settings.LANGUAGE_COOKIE_HTTPONLY,                samesite=settings.LANGUAGE_COOKIE_SAMESITE,</ins>             )     return response


Question: modeltodict() should return an empty dict for an empty list of fields. Description Been called as modeltodict(instance, fields=[]) function should return empty dict, because no fields were requested. But it returns all fields The problem point is if fields and f.name not in fields: which should be if fields is not None and f.name not in fields: PR:

for f in chain(opts.concrete_fields, opts.private_fields, opts.many_to_many):         if not getattr(f, 'editable', False):             continue        if fields and f.name not in fields:        if fields is not None and f.name not in fields:             continue         if exclude and f.name in exclude:             continue


Question: New HTTPRequest.headers not usable in templates because of hyphens Description With the release of 2.2, I took the opportunity from the new HTTPRequest.headers object to clean up old code using e.g. request.META['HTTPXREALIP'] to request.headers['X-Real-IP']. However, this new approach does not work with templates, as variable lookups cannot use hyphens. Could the object contain a parallel set of keys in underscored variables? e.g. request.headers['foo-bar'] is also available in request.headers['foobar'] and can be looked up with {{ request.headers.foo_bar }} in a template?

headers[name] = value         super().__init__(headers) <ins>    def __getitem__(self, key):        """Allow header lookup using underscores in place of hyphens."""        return super().__getitem__(key.replace('_', '-'))</ins>     @classmethod     def parse_header_name(cls, header):         if header.startswith(cls.HTTP_PREFIX):


Question: Admin app has too hard a dependency on sessions app Description Since (), released in 2.2, the admin app checks whether the django.contrib.sessions app is in INSTALLEDAPPS. Some projects may have opted to use a replacement session management app such as  – the admin app claims to be incompatible with such a configuration, even if it actually means "I'm going to need some session management that works like django.contrib.sessions". Maybe it would be better to get rid of the app check and do what's being done for various middleware in the checks function anyway, e.g. something like if not containssubclass('django.contrib.sessions.middleware.SessionMiddleware', settings.MIDDLEWARE): (checks.Error( "'django.contrib.sessions.middleware.SessionMiddleware' must " "be in MIDDLEWARE in order to use the admin application.", id='admin.E4XX', )) – this would be out-of-the-box compatible with e.g. Qsessions. The obvious workaround is to just re-add django.contrib.sessions back into INSTALLEDAPPS which kinda works, but has the mild but unfortunate side effect of forcibly enabling the model and migrations, (re-)adding a useless django_session table into the database upon migration.

('django.contrib.contenttypes', 401),         ('django.contrib.auth', 405),         ('django.contrib.messages', 406),<del>        ('django.contrib.sessions', 407),</del>     )     for app_name, error_code in app_dependencies:         if not apps.is_installed(app_name):


Question: Admin app has too hard a dependency on sessions app Description Since (), released in 2.2, the admin app checks whether the django.contrib.sessions app is in INSTALLEDAPPS. Some projects may have opted to use a replacement session management app such as  – the admin app claims to be incompatible with such a configuration, even if it actually means "I'm going to need some session management that works like django.contrib.sessions". Maybe it would be better to get rid of the app check and do what's being done for various middleware in the checks function anyway, e.g. something like if not containssubclass('django.contrib.sessions.middleware.SessionMiddleware', settings.MIDDLEWARE): (checks.Error( "'django.contrib.sessions.middleware.SessionMiddleware' must " "be in MIDDLEWARE in order to use the admin application.", id='admin.E4XX', )) – this would be out-of-the-box compatible with e.g. Qsessions. The obvious workaround is to just re-add django.contrib.sessions back into INSTALLEDAPPS which kinda works, but has the mild but unfortunate side effect of forcibly enabling the model and migrations, (re-)adding a useless django_session table into the database upon migration.

"be in MIDDLEWARE in order to use the admin application.",             id='admin.E409',         ))<ins>    if not _contains_subclass('django.contrib.sessions.middleware.SessionMiddleware', settings.MIDDLEWARE):        errors.append(checks.Error(            "'django.contrib.sessions.middleware.SessionMiddleware' must "            "be in MIDDLEWARE in order to use the admin application.",            id='admin.E410',        ))</ins>     return errors


Question: Add new system check message when custom error handler '' cannot be imported Description (last modified by Alasdair Nicol) checks for the signatures of custom error handlers. When the '' cannot be imported, it raises ModuleNotFoundError or ViewDoesNotExist, as seen in this Stack Overflow question:  I suggest we catch the exception, and add another check code, e.g. urls.E008: The custom ` cannot be imported.

from django.conf import settings from django.core.checks import Error, Warning from django.core.checks.urls import check_resolverfrom django.core.exceptions import ImproperlyConfiguredfrom django.core.exceptions import ImproperlyConfigured, ViewDoesNotExist from django.utils.datastructures import MultiValueDict from django.utils.functional import cached_property from django.utils.http import RFC3986_SUBDELIMS, escape_leading_slashes


Question: Add new system check message when custom error handler '' cannot be imported Description (last modified by Alasdair Nicol) checks for the signatures of custom error handlers. When the '' cannot be imported, it raises ModuleNotFoundError or ViewDoesNotExist, as seen in this Stack Overflow question:  I suggest we catch the exception, and add another check code, e.g. urls.E008: The custom ` cannot be imported.

# All handlers take (request, exception) arguments except handler500         # which takes (request).         for status_code, num_parameters in [(400, 2), (403, 2), (404, 2), (500, 1)]:            handler, param_dict = self.resolve_error_handler(status_code)            try:                handler, param_dict = self.resolve_error_handler(status_code)            except (ImportError, ViewDoesNotExist) as e:                path = getattr(self.urlconf_module, 'handler%s' % status_code)                msg = (                    "The custom handler{status_code} view '{path}' could not be imported."                ).format(status_code=status_code, path=path)                messages.append(Error(msg, hint=str(e), id='urls.E008'))                continue             signature = inspect.signature(handler)             args = [None] * num_parameters             try:


Question: UnicodeDecodeError when loading debug templates. Description While studying Django, I had an error where the page was not properly printed. In the log window, UnicodeDecodeError was continuously printed, and in the template, the sentence 'A server error occured. Please contact the administrator' was printed So when I checked the file of Django 2.2version, I found that the dotdotdot wrapped by the <spantag on the 239th line was changed to ellipsis. Apparently, the version of Django 2.1.8 was a dotdotdot. So I took steps to change the 239th line's ellipsis to dotdotdot. Or, when reading the file from inside the file, the encoding format was set to utf-8. This enabled me to resolve the error. Did you intentionally modify the technicalhtml file?

def get_traceback_html(self):         """Return HTML version of debug 500 HTTP error page."""        with Path(CURRENT_DIR, 'templates', 'technical_500.html').open() as fh:        with Path(CURRENT_DIR, 'templates', 'technical_500.html').open(encoding='utf-8') as fh:             t = DEBUG_ENGINE.from_string(fh.read())         c = Context(self.get_traceback_data(), use_l10n=False)         return t.render(c)      def get_traceback_text(self):         """Return plain text version of debug 500 HTTP error page."""        with Path(CURRENT_DIR, 'templates', 'technical_500.txt').open() as fh:        with Path(CURRENT_DIR, 'templates', 'technical_500.txt').open(encoding='utf-8') as fh:             t = DEBUG_ENGINE.from_string(fh.read())         c = Context(self.get_traceback_data(), autoescape=False, use_l10n=False)         return t.render(c)


Question: UnicodeDecodeError when loading debug templates. Description While studying Django, I had an error where the page was not properly printed. In the log window, UnicodeDecodeError was continuously printed, and in the template, the sentence 'A server error occured. Please contact the administrator' was printed So when I checked the file of Django 2.2version, I found that the dotdotdot wrapped by the <spantag on the 239th line was changed to ellipsis. Apparently, the version of Django 2.1.8 was a dotdotdot. So I took steps to change the 239th line's ellipsis to dotdotdot. Or, when reading the file from inside the file, the encoding format was set to utf-8. This enabled me to resolve the error. Did you intentionally modify the technicalhtml file?

module = obj.__module__             caller = '%s.%s' % (module, caller)     with Path(CURRENT_DIR, 'templates', 'technical_404.html').open() as fh:    with Path(CURRENT_DIR, 'templates', 'technical_404.html').open(encoding='utf-8') as fh:         t = DEBUG_ENGINE.from_string(fh.read())     c = Context({         'urlconf': urlconf,


Question: UnicodeDecodeError when loading debug templates. Description While studying Django, I had an error where the page was not properly printed. In the log window, UnicodeDecodeError was continuously printed, and in the template, the sentence 'A server error occured. Please contact the administrator' was printed So when I checked the file of Django 2.2version, I found that the dotdotdot wrapped by the <spantag on the 239th line was changed to ellipsis. Apparently, the version of Django 2.1.8 was a dotdotdot. So I took steps to change the 239th line's ellipsis to dotdotdot. Or, when reading the file from inside the file, the encoding format was set to utf-8. This enabled me to resolve the error. Did you intentionally modify the technicalhtml file?

def default_urlconf(request):     """Create an empty URLconf 404 error response."""    with Path(CURRENT_DIR, 'templates', 'default_urlconf.html').open() as fh:    with Path(CURRENT_DIR, 'templates', 'default_urlconf.html').open(encoding='utf-8') as fh:         t = DEBUG_ENGINE.from_string(fh.read())     c = Context({         'version': get_docs_version(),


Question: Django 2.2 autoreloader is failing intermittently (not using watchman) Description Since upgrading to Django 2.2 the autoreloader is working differently and is not always picking up changes to .py files to reload runserver. It seems to work consistently once on a file, but multiple changes to the same file will not necessarily trigger a reload. It's definitely a change in the behavior to previous versions of Django (up to 2.1.7) I'm not using watchman. I'm running runserver in a python 3.7 slim docker container like runserver 0.0.0.0:8001 As a workaround I have tried to add an edit a different .py -- that doesn't seem to help consistently. I've also noticed that the problem seems to sometimes go away by itself -- perhaps there is some change to the timing of the autoreloader which is not clear? For example and ignore interval of a certain duration.

def ensure_echo_on():    if termios:        fd = sys.stdin        if fd.isatty():            attr_list = termios.tcgetattr(fd)            if not attr_list[3] & termios.ECHO:                attr_list[3] |= termios.ECHO                if hasattr(signal, 'SIGTTOU'):                    old_handler = signal.signal(signal.SIGTTOU, signal.SIG_IGN)                else:                    old_handler = None                termios.tcsetattr(fd, termios.TCSANOW, attr_list)                if old_handler is not None:                    signal.signal(signal.SIGTTOU, old_handler)    """    Ensure that echo mode is enabled. Some tools such as PDB disable    it which causes usability issues after reload.    """    if not termios or not sys.stdin.isatty():        return    attr_list = termios.tcgetattr(sys.stdin)    if not attr_list[3] & termios.ECHO:        attr_list[3] |= termios.ECHO        if hasattr(signal, 'SIGTTOU'):            old_handler = signal.signal(signal.SIGTTOU, signal.SIG_IGN)        else:            old_handler = None        termios.tcsetattr(sys.stdin, termios.TCSANOW, attr_list)        if old_handler is not None:            signal.signal(signal.SIGTTOU, old_handler)   def iter_all_python_module_files():


Question: Django 2.2 autoreloader is failing intermittently (not using watchman) Description Since upgrading to Django 2.2 the autoreloader is working differently and is not always picking up changes to .py files to reload runserver. It seems to work consistently once on a file, but multiple changes to the same file will not necessarily trigger a reload. It's definitely a change in the behavior to previous versions of Django (up to 2.1.7) I'm not using watchman. I'm running runserver in a python 3.7 slim docker container like runserver 0.0.0.0:8001 As a workaround I have tried to add an edit a different .py -- that doesn't seem to help consistently. I've also noticed that the problem seems to sometimes go away by itself -- perhaps there is some change to the timing of the autoreloader which is not clear? For example and ignore interval of a certain duration.

from django.urls import get_resolver         # Prevent a race condition where URL modules aren't loaded when the         # reloader starts by accessing the urlconf_module property.        get_resolver().urlconf_module        try:            get_resolver().urlconf_module        except Exception:            # Loading the urlconf can result in errors during development.            # If this occurs then swallow the error and continue.            pass         logger.debug('Apps ready_event triggered. Sending autoreload_started signal.')         autoreload_started.send(sender=self)         self.run_loop()


Question: Django 2.2 autoreloader is failing intermittently (not using watchman) Description Since upgrading to Django 2.2 the autoreloader is working differently and is not always picking up changes to .py files to reload runserver. It seems to work consistently once on a file, but multiple changes to the same file will not necessarily trigger a reload. It's definitely a change in the behavior to previous versions of Django (up to 2.1.7) I'm not using watchman. I'm running runserver in a python 3.7 slim docker container like runserver 0.0.0.0:8001 As a workaround I have tried to add an edit a different .py -- that doesn't seem to help consistently. I've also noticed that the problem seems to sometimes go away by itself -- perhaps there is some change to the timing of the autoreloader which is not clear? For example and ignore interval of a certain duration.

SLEEP_TIME = 1  # Check for changes once per second.      def tick(self):        state, previous_timestamp = {}, time.time()        mtimes = {}         while True:            state.update(self.loop_files(state, previous_timestamp))            previous_timestamp = time.time()            for filepath, mtime in self.snapshot_files():                old_time = mtimes.get(filepath)                if old_time is None:                    logger.debug('File %s first seen with mtime %s', filepath, mtime)                    mtimes[filepath] = mtime                    continue                elif mtime > old_time:                    logger.debug('File %s previous mtime: %s, current mtime: %s', filepath, old_time, mtime)                    self.notify_file_changed(filepath)             time.sleep(self.SLEEP_TIME)             yield     def loop_files(self, previous_times, previous_timestamp):        updated_times = {}        for path, mtime in self.snapshot_files():            previous_time = previous_times.get(path)            # If there are overlapping globs, a file may be iterated twice.            if path in updated_times:                continue            # A new file has been detected. This could happen due to it being            # imported at runtime and only being polled now, or because the            # file was just created. Compare the file's mtime to the            # previous_timestamp and send a notification if it was created            # since the last poll.            is_newly_created = previous_time is None and mtime > previous_timestamp            is_changed = previous_time is not None and previous_time != mtime            if is_newly_created or is_changed:                logger.debug('File %s. is_changed: %s, is_new: %s', path, is_changed, is_newly_created)                logger.debug('File %s previous mtime: %s, current mtime: %s', path, previous_time, mtime)                self.notify_file_changed(path)                updated_times[path] = mtime        return updated_times     def snapshot_files(self):        # watched_files may produce duplicate paths if globs overlap.        seen_files = set()         for file in self.watched_files():            if file in seen_files:                continue             try:                 mtime = file.stat().st_mtime             except OSError:                 # This is thrown when the file does not exist.                 continue            seen_files.add(file)             yield file, mtime      @classmethod


Question: Django 2.2 autoreloader is failing intermittently (not using watchman) Description Since upgrading to Django 2.2 the autoreloader is working differently and is not always picking up changes to .py files to reload runserver. It seems to work consistently once on a file, but multiple changes to the same file will not necessarily trigger a reload. It's definitely a change in the behavior to previous versions of Django (up to 2.1.7) I'm not using watchman. I'm running runserver in a python 3.7 slim docker container like runserver 0.0.0.0:8001 As a workaround I have tried to add an edit a different .py -- that doesn't seem to help consistently. I've also noticed that the problem seems to sometimes go away by itself -- perhaps there is some change to the timing of the autoreloader which is not clear? For example and ignore interval of a certain duration.

ensure_echo_on()      main_func = check_errors(main_func)    django_main_thread = threading.Thread(target=main_func, args=args, kwargs=kwargs)    django_main_thread = threading.Thread(target=main_func, args=args, kwargs=kwargs, name='django-main-thread')     django_main_thread.setDaemon(True)     django_main_thread.start()


Question: delete() on instances of models without any dependencies doesn't clear PKs. Description Deleting any model with no dependencies not updates the PK on the model. It should be set to None after .delete() call. See Should update the model line 280.

if self.can_fast_delete(instance):                 with transaction.mark_for_rollback_on_error():                     count = sql.DeleteQuery(model).delete_batch([instance.pk], self.using)<ins>                setattr(instance, model._meta.pk.attname, None)</ins>                 return count, {model._meta.label: count}          with transaction.atomic(using=self.using, savepoint=False):


Question: (keep_parents=True) does not preserve all parent reverse relationships Description (last modified by Stephen Brown) There was a partial fix for this in , but it doesn't cater for relationships toward parents of parents, and so on.

collect_related=False,                                  reverse_dependency=True)         if collect_related:            parents = model._meta.parents            if keep_parents:                parents = set(model._meta.get_parent_list())             for related in get_candidate_relations_to_delete(model._meta):                 # Preserve parent reverse relationships if keep_parents=True.                 if keep_parents and related.model in parents:


Question: Allow specifying a Feed language different from settings.LANGUAGECODE Description RSS uses settings.LANGUAGECODE as "language tag" for the RSS feed I couldnt find a way to change it.. so I made this patch

from calendar import timegm <del>from django.conf import settings</del> from django.contrib.sites.shortcuts import get_current_site from django.core.exceptions import ImproperlyConfigured, ObjectDoesNotExist from django.http import Http404, HttpResponse


Question: Allow specifying a Feed language different from settings.LANGUAGECODE Description RSS uses settings.LANGUAGECODE as "language tag" for the RSS feed I couldnt find a way to change it.. so I made this patch

from django.utils.html import escape from django.utils.http import http_date from django.utils.timezone import get_default_timezone, is_naive, make_aware<ins>from django.utils.translation import get_language</ins>   def add_domain(domain, url, secure=False):


Question: Allow specifying a Feed language different from settings.LANGUAGECODE Description RSS uses settings.LANGUAGECODE as "language tag" for the RSS feed I couldnt find a way to change it.. so I made this patch

feed_type = feedgenerator.DefaultFeed     title_template = None     description_template = None<ins>    language = None</ins>      def __call__(self, request, *args, **kwargs):         try:


Question: Allow specifying a Feed language different from settings.LANGUAGECODE Description RSS uses settings.LANGUAGECODE as "language tag" for the RSS feed I couldnt find a way to change it.. so I made this patch

subtitle=self._get_dynamic_attr('subtitle', obj),             link=link,             description=self._get_dynamic_attr('description', obj),            language=settings.LANGUAGE_CODE,            language=self.language or get_language(),             feed_url=add_domain(                 current_site.domain,                 self._get_dynamic_attr('feed_url', obj) or request.path,


Question: Inefficient SQL generated when counting a ManyToMany Description When calling count() on an unfiltered many to many relation, a useless join is included in the SQL that makes it much slower than it should be. On my dataset, the difference is 1000ms to 100ms, because an index-only scan can be used. This is the SQL that is currently generated: SELECT COUNT() AS "count" FROM "appfoo" INNER JOIN "appfoobar" ON ("appfoo"."id" = "appfoobar"."fooid") WHERE "appfoobar"."fooid" = ?; This is the SQL that should be generated: SELECT COUNT() AS "count" FROM "appfoobar" WHERE "appfoobar"."foo_id" = ?; This optimization can only be applied when there are no filters applied, because then the join is used to satisfy the filters. In the no-filters case, only the through table needs to be consulted.

False,             ) <del>        @property        def constrained_target(self):            # If the through relation's target field's foreign integrity is            # enforced, the query can be performed solely against the through            # table as the INNER JOIN'ing against target table is unnecessary.            if not self.target_field.db_constraint:                return None            db = router.db_for_read(self.through, instance=self.instance)            if not connections[db].features.supports_foreign_keys:                return None            hints = {'instance': self.instance}            manager = self.through._base_manager.db_manager(db, hints=hints)            filters = {self.source_field_name: self.instance.pk}            # Nullable target rows must be excluded as well as they would have            # been filtered out from an INNER JOIN.            if self.target_field.null:                filters['%s__isnull' % self.target_field_name] = False            return manager.filter(**filters)        def exists(self):            constrained_target = self.constrained_target            return constrained_target.exists() if constrained_target else super().exists()        def count(self):            constrained_target = self.constrained_target            return constrained_target.count() if constrained_target else super().count()</del>         def add(self, *objs, through_defaults=None):             self._remove_prefetched_objects()             db = router.db_for_write(self.through, instance=self.instance)


Question: renders small decimals in exponential notation. Description When using utils.numberformat with decimalpos, extremely small numbers get displayed using exponential notation. from import format as nformat nformat(Decimal('1e-199'), '.', decimalpos=2) '0.00' nformat(Decimal('1e-200'), '.', decimalpos=2) '1.00e-200' This is caused by a hardcoded cut-off point in the internal logic, but I would argue that when a decimal_pos argument is supplied and the number to be formatted is smaller in absolute size than what can be encoded using the provided number of decimal positions, the returned string should be 0.0000...000 instead.

# sign     sign = ''     if isinstance(number, Decimal):<ins>        if decimal_pos is not None:            # If the provided number is too small to affect any of the visible            # decimal places, consider it equal to '0'.            cutoff = Decimal('0.' + '1'.rjust(decimal_pos, '0'))            if abs(number) < cutoff:                number = Decimal('0')</ins>         # Format values with more than 200 digits (an arbitrary cutoff) using         # scientific notation to avoid high memory usage in {:f}'.format().         _, digits, exponent = number.as_tuple()