Question: slugify() doesn't return a valid slug for "İ". Description While working on an international project, we discovered that the turkish/azerbaijani letter İ can not be properly processed when SlugField and slugify are run with allowunicode=True. The project itself runs with Django 2.2.6 and Wagtail 2.6.2. I first talked about this in the Wagtail Support Channel and while researching further, discovered that this is a Django/Python related issue. This was tested on Python 3.6 and on Python 3.7. (quick shoutout to Matt Wescott of Wagtail Fame for being a sparing partner in this) There is a rather detailed analysis (README) in a git repo I created  - it was also the basis for my initial call for help in wagtail's support channel. Meanwhile I have extended it with a Django-only project, as to be a 100% sure this has nothing to do with Wagtail. I was not able to find anything similar in trac. While I encourage whoever is reading this to actually read the README in the git repo, I want to honor your time and will try to provide a more concise version of the bug here. Explanation from import models from import slugify class Page(models.Model): title = models.CharField(maxlength=255) slug = models.SlugField(allowunicode=True) def str(self): return Using this in a shell/test like a (Model)Form might: from import slugify page = Page(title="Hello İstanbul") = slugify(, allowunicode=True) page.fullclean() fullclean() then raises {'slug': Enter a valid 'slug' consisting of Unicode letters, numbers, underscores, or hyphens.?} Why is that? slugify does the following internally: (r'[^\w\s-]', '', value).strip().lower() Thing is, Python's .lower() of the İ in İstanbul looks like this: [(character) for character in 'İ'.lower()] ['LATIN SMALL LETTER I', 'COMBINING DOT ABOVE'] So, while slugify() finishes, the result is then passed into SlugField which uses the slugunicodere (^[-\w]+\Z). However, the Combining Dot Above is not a valid \w: [(character, (character), (character)) for character in 'İ'.lower()] [ ('i', 'LATIN SMALL LETTER I', <re.Match object; span=(0, 1), match='i'), # EUREKA!! ('̇', 'COMBINING DOT ABOVE', None) ] So that's why the ValidationError is raised. Proposed Solution The culprit seems to be the order in which lower() is called in slugify. The assumption that the lowercase version of a string is still a valid slugunicodere, does not seem to hold true. Hence, instead of doing this in slugify() (r'[^\w\s-]', '', value).strip().lower() It might be better to do it like this (r'[^\w\s-]', '', ()).strip() Is Python the actual culprit? Yeah it might be. Matt ( urged me to also take a look if Python might be doing this wrong. The İ is the Latin Capital Letter I with Dot Above. It's codepoint is U+0130 According to the chart for the Latin Extended-A set (), it's lowercase version is U+0069. U+0069 lives in the C0 Controls and Basic Latin set (). Lo and behold: it is the Latin small letter I. So a latin lowercase i. Does this really mean that Python is doing something weird here by adding the Combining dot above? Honestly, I can't imagine that and I am probably missing an important thing here because my view is too naive. I hope this shorter explanation makes sense. If it does not, please try to read through the detailed analysis in the repo (). If that also does not make a ton of sense, let me know. In any case, thank you for taking the time to read this bug report. Looking forward to feedback and thoughts. I am happy to oblige in any capacity necessary.

value = unicodedata.normalize('NFKC', value)     else:         value = unicodedata.normalize('NFKD', value).encode('ascii', 'ignore').decode('ascii')    value = re.sub(r'[^\w\s-]', '', value).strip().lower()    value = re.sub(r'[^\w\s-]', '', value.lower()).strip()     return re.sub(r'[-\s]+', '-', value)


Question: Custom template tags raise TemplateSyntaxError when keyword-only arguments with defaults are provided. Description (last modified by P-Seebauer) When creating simple tags without variable keyword args, but an keyword argument with a default value. It's not possible to supply any other variable. def hello(, greeting='hello'): return f'{greeting} world' {% hello greeting='hi' %} Raises “'hello' received unexpected keyword argument 'greeting'” Also supplying a keyword argument a second time raises the wrong error message: #tag def hi(, greeting): return f'{greeting} world' {% hi greeting='hi' greeting='hello' %} Raises “'hi' received unexpected keyword argument 'greeting'” instead of "'hi' received multiple values for keyword argument 'greeting'" Same goes for inclusion tags (is the same code) I already have a fix ready, will push it after creating the ticket (that I have a ticket# for the commit). Is actually for all versions since the offending line is from 2.0…

if kwarg:             # The kwarg was successfully extracted             param, value = kwarg.popitem()            if param not in params and param not in unhandled_kwargs and varkw is None:            if param not in params and param not in kwonly and varkw is None:                 # An unexpected keyword argument was supplied                 raise TemplateSyntaxError(                     "'%s' received unexpected keyword argument '%s'" %


Question: Resetting primary key for a child model doesn't work. Description In the attached example code setting the primary key to None does not work (so that the existing object is overwritten on save()). The most important code fragments of the bug example: from import models class Item(models.Model): # uid = models.UUIDField(primarykey=True, default=uuid.uuid4, editable=False) uid = models.AutoField(primarykey=True, editable=False) f = models.BooleanField(default=False) def reset(self): = None self.f = False class Derived(Item): pass class SaveTestCase(TestCase): def setUp(self): self.derived = (f=True) # create the first object item = () obj1 = item.derived () () # the first object is overwritten def testftrue(self): obj = () self.assertTrue(obj.f) Django 2.1.2

return getattr(self, meta.pk.attname)      def _set_pk_val(self, value):<ins>        for parent_link in self._meta.parents.values():            if parent_link and parent_link != self._meta.pk:                setattr(self, parent_link.target_field.attname, value)</ins>         return setattr(self, self._meta.pk.attname, value)      pk = property(_get_pk_val, _set_pk_val)


Question: FileInput shouldn't display required attribute when initial data exists. Description (last modified by thenewguy) I think that ClearableFileInput.userequiredattribute() () should be moved to FileInput.userequiredattribute() so that required is not output on the html input element that represents FileInput when a file is already set (e.g. already saved on a model instance that is being edited). Maybe I am overlooking a use case where this is not desirable? I can not think of one.

def value_omitted_from_data(self, data, files, name):         return name not in files <ins>    def use_required_attribute(self, initial):        return super().use_required_attribute(initial) and not initial</ins>  FILE_INPUT_CONTRADICTION = object()


Question: FileInput shouldn't display required attribute when initial data exists. Description (last modified by thenewguy) I think that ClearableFileInput.userequiredattribute() () should be moved to FileInput.userequiredattribute() so that required is not output on the html input element that represents FileInput when a file is already set (e.g. already saved on a model instance that is being edited). Maybe I am overlooking a use case where this is not desirable? I can not think of one.

return False         return upload <del>    def use_required_attribute(self, initial):        return super().use_required_attribute(initial) and not initial</del>     def value_omitted_from_data(self, data, files, name):         return (             super().value_omitted_from_data(data, files, name) and


Question: admin.E130 (duplicate name attributes of actions) should specify which were duplicated. Description The fact that the name is used is somewhat an implementation detail, and there's no guarantee the user has enough of an understanding of python to know what that attribute is, let alone how to fix it. This just came up on IRC because a user had defined actions = [deleteselected] where deleteselected was a reference to their own callable, but shares the name of the base one (and by specifying the actions = they were assuming that they were wholesale replacing the actions list, where that may not be true for site-wide actions) so errored ... but they only had define a list of len(...) == 1 so how can there be a duplicate (is their thought process)? The error message should specify those names that occur 2(rather than just check len(...) vs len(set(...))), and ought ideally to explain where the duplicate comes from (ie: AdminSite-wide). Related ticket about E130: (+ those it references) but is about the replacement strategy rather than the error message itself.

<ins>import collections</ins> from itertools import chain  from django.apps import apps


Question: admin.E130 (duplicate name attributes of actions) should specify which were duplicated. Description The fact that the name is used is somewhat an implementation detail, and there's no guarantee the user has enough of an understanding of python to know what that attribute is, let alone how to fix it. This just came up on IRC because a user had defined actions = [deleteselected] where deleteselected was a reference to their own callable, but shares the name of the base one (and by specifying the actions = they were assuming that they were wholesale replacing the actions list, where that may not be true for site-wide actions) so errored ... but they only had define a list of len(...) == 1 so how can there be a duplicate (is their thought process)? The error message should specify those names that occur 2(rather than just check len(...) vs len(set(...))), and ought ideally to explain where the duplicate comes from (ie: AdminSite-wide). Related ticket about E130: (+ those it references) but is about the replacement strategy rather than the error message itself.

def _check_actions_uniqueness(self, obj):         """Check that every action has a unique __name__."""        names = [name for _, name, _ in obj._get_base_actions()]        if len(names) != len(set(names)):            return [checks.Error(                '__name__ attributes of actions defined in %s must be '                'unique.' % obj.__class__,                obj=obj.__class__,                id='admin.E130',            )]        return []        errors = []        names = collections.Counter(name for _, name, _ in obj._get_base_actions())        for name, count in names.items():            if count > 1:                errors.append(checks.Error(                    '__name__ attributes of actions defined in %s must be '                    'unique. Name %r is not unique.' % (                        obj.__class__.__name__,                        name,                    ),                    obj=obj.__class__,                    id='admin.E130',                ))        return errors   class InlineModelAdminChecks(BaseModelAdminChecks):


Question: Model.getFOOdisplay() does not work correctly with inherited choices. Description (last modified by Mariusz Felisiak) Given a base model with choices A containing 3 tuples Child Model inherits the base model overrides the choices A and adds 2 more tuples getfoodisplay does not work correctly for the new tuples Example: class A(models.Model): foochoice = [("A","output1"),("B","output2")] fieldfoo = models.CharField(maxlength=254,choices=foochoice) class Meta: abstract = True class B(A): foochoice = [("A","output1"),("B","output2"),("C","output3")] fieldfoo = models.CharField(maxlength=254,choices=foochoice) Upon invoking getfieldfoo_display() on instance of B , For value "A" and "B" the output works correctly i.e. returns "output1" / "output2" but for value "C" the method returns "C" and not "output3" which is the expected behaviour

if not getattr(cls, self.attname, None):                 setattr(cls, self.attname, self.descriptor_class(self))         if self.choices is not None:            if not hasattr(cls, 'get_%s_display' % self.name):            # Don't override a get_FOO_display() method defined explicitly on            # this class, but don't check methods derived from inheritance, to            # allow overriding inherited choices. For more complex inheritance            # structures users should override contribute_to_class().            if 'get_%s_display' % self.name not in cls.__dict__:                 setattr(                     cls,                     'get_%s_display' % self.name,


Question: translation.E004 shouldn't be raised on sublanguages when a base language is available. Description According to Django documentation: If a base language is available but the sublanguage specified is not, Django uses the base language. For example, if a user specifies de-at (Austrian German) but Django only has de available, Django uses de. However, when using Django 3.0.2, if my has LANGUAGECODE = "de-at" I get this error message: SystemCheckError: System check identified some issues: ERRORS: ?: (translation.E004) You have provided a value for the LANGUAGECODE setting that is not in the LANGUAGES setting. If using LANGUAGE_CODE = "es-ar" Django works fine (es-ar is one of the translations provided out of the box).

from django.conf import settings<ins>from django.utils.translation import get_supported_language_variant</ins> from django.utils.translation.trans_real import language_code_re  from . import Error, Tags, register


Question: translation.E004 shouldn't be raised on sublanguages when a base language is available. Description According to Django documentation: If a base language is available but the sublanguage specified is not, Django uses the base language. For example, if a user specifies de-at (Austrian German) but Django only has de available, Django uses de. However, when using Django 3.0.2, if my has LANGUAGECODE = "de-at" I get this error message: SystemCheckError: System check identified some issues: ERRORS: ?: (translation.E004) You have provided a value for the LANGUAGECODE setting that is not in the LANGUAGES setting. If using LANGUAGE_CODE = "es-ar" Django works fine (es-ar is one of the translations provided out of the box).

@register(Tags.translation) def check_language_settings_consistent(app_configs, **kwargs):     """Error if language settings are not consistent with each other."""    available_tags = {i for i, _ in settings.LANGUAGES} | {'en-us'}    if settings.LANGUAGE_CODE not in available_tags:    try:        get_supported_language_variant(settings.LANGUAGE_CODE)    except LookupError:         return [E004]    return []    else:        return []


Question: Raise a descriptive error on update()/delete() operations following (), intersection(), and difference(). Description (last modified by Joon Hwan ) bfilter() seems to merge but does not apply to the actual update q = () q = q.union(M.objects.afilter()) print(q) q = q.union(M.objects.bfilter()) print(q) q.update(name='foo') <QuerySet [<M: M object (675)]<QuerySet [<M: M object (675), <M: M object (773)]UPDATE "m" SET "name" = "foo" WHERE ORDER BY U0."id" ASC LIMIT 1); args=(...)

def delete(self):         """Delete the records in the current QuerySet."""<ins>        self._not_support_combined_queries('delete')</ins>         assert not self.query.is_sliced, \             "Cannot use 'limit' or 'offset' with delete."


Question: Raise a descriptive error on update()/delete() operations following (), intersection(), and difference(). Description (last modified by Joon Hwan ) bfilter() seems to merge but does not apply to the actual update q = () q = q.union(M.objects.afilter()) print(q) q = q.union(M.objects.bfilter()) print(q) q.update(name='foo') <QuerySet [<M: M object (675)]<QuerySet [<M: M object (675), <M: M object (773)]UPDATE "m" SET "name" = "foo" WHERE ORDER BY U0."id" ASC LIMIT 1); args=(...)

Update all elements in the current QuerySet, setting all the given         fields to the appropriate values.         """<ins>        self._not_support_combined_queries('update')</ins>         assert not self.query.is_sliced, \             "Cannot update a query once a slice has been taken."         self._for_write = True


Question: Enumeration Types are not usable in templates. Description (last modified by Mariusz Felisiak) The new enumeration types are great but can't be used in Django templates due to their being callable. For example this doesn't work: {% if student.yearinschool == YearInSchool.FRESHMAN %} This is because YearInSchool, being a class, is callable, and Django Templates always call callables with no arguments. The call fails because the required value argument is missing. The easy solution would be to declare donotcallintemplates = True on the various Choices classes.

# that is passed in as "self" as the value to use when looking up the         # label in the choices.         cls.label = property(lambda self: cls._value2label_map_.get(self.value))<ins>        cls.do_not_call_in_templates = True</ins>         return enum.unique(cls)      def __contains__(cls, member):


Question: Named groups in choices are not properly validated in case of non str typed values. Description In case of using typed choices and string value to store it (in my case it is multiple values stored in char field as JSON) it is possible to catch error while run makemigrations (checkchoices error): main.MultiValueFieldModel.multivaluefieldintegerwithgroupedchoices: (fields.E005) 'choices' must be an iterable containing (actual value, human readable name) tuples. Looking deeper into the django code, we see actual error message: 'int' object is not iterable and it happens in this block of code (): if self.maxlength is not None and groupchoices: choicemaxlength = max( choicemaxlength, (len(value) for value, in groupchoices if isinstance(value, str)), ) If we have CharField (any other with maxlength defined) and grouped choices with non str typed values like: choices=( ('one', ((1, 'One',), (11, 'Eleven',),),), ('two', ((2, 'Two',), (22, 'Twenty two',),),), ) we will have the situation, when max function receives only one integer value (choicemaxlength), because (len(value) for value, in groupchoices if isinstance(value, str)) will return empty generator, and that is why error 'int' object is not iterable raises (max function waits iterable if there is only one argument). Code block: choicemaxlength = max( choicemaxlength, (len(value) for value, in groupchoices if isinstance(value, str)), ) in this case works like: choicemaxlength = max( choicemaxlength, [], ) which is incorrect. The simples solution is to add one additional argument to max function, which will be usefull only in this partucular situation: choicemaxlength = max( choicemaxlength, 0, (len(value) for value, in group_choices if isinstance(value, str)), )

):                     break                 if self.max_length is not None and group_choices:                    choice_max_length = max(                    choice_max_length = max([                         choice_max_length,                         *(len(value) for value, _ in group_choices if isinstance(value, str)),                    )                    ])             except (TypeError, ValueError):                 # No groups, choices in the form [value, display]                 value, human_name = group_name, group_choices


Question: JSONField are not properly displayed in admin when they are readonly. Description JSONField values are displayed as dict when readonly in the admin. For example, {"foo": "bar"} would be displayed as {'foo': 'bar'}, which is not valid JSON. I believe the fix would be to add a special case in to call the prepare_value of the JSONField (not calling directly to take care of the InvalidJSONInput case).

return formats.number_format(value)     elif isinstance(field, models.FileField) and value:         return format_html('{}', value.url, value)    elif isinstance(field, models.JSONField) and value:        try:            return field.get_prep_value(value)        except TypeError:            return display_for_value(value, empty_value_display)     else:         return display_for_value(value, empty_value_display)


Question: makemigrations does not detect/like model name case changes Description Starting with class Evidence(models.Model): rubrictype = models.ForeignKey('Rubrictype') class Rubrictype(models.Model): typecode = models.CharField(maxlength=1) Make the initial migration: $ makemigrations Migrations for 'asmigrations': Create model Evidence Create model Rubrictype Add field rubrictype to evidence Change the name of Rubrictype to RubricType: class Evidence(models.Model): rubrictype = models.ForeignKey('RubricType') class RubricType(models.Model): typecode = models.CharField(maxlength=1) Generate the migration: $ makemigrations Migrations for 'asmigrations': Alter field rubrictype on evidence Django does not detect the name change on the RubricType model itself. No confirmation is requested for the name change and no operation is generated. The problem is that any subsequent makemigrations run will generate the same operation ad infinitum: $ makemigrations Migrations for 'asmigrations': Alter field rubrictype on evidence If instead the name is changed to RubricXype: class Evidence(models.Model): rubrictype = models.ForeignKey('RubricXype') class RubricXype(models.Model): typecode = models.CharField(maxlength=1) the corresponding migration becomes $ makemigrations Did you rename the asmigrations.Rubrictype model to RubricXype? [y/N] y Migrations for 'asmigrations': Rename model Rubrictype to RubricXype This migration generates a RenameModel operation only and any subsequent makemigrations runs will properly report "No changes detected". So it appears the change detector does not pick up on capitalization changes in model names. Trying to work around by adding a migrations.RenameModel( oldname='Rubrictype', newname='RubricType', ) to the auto generated operations results in a ValueError exception when makemigrations is run again: $ makemigrations Traceback (most recent call last): File "", line 10, in <moduleexecutefromcommandline() File "/home/svencoenye/developer/djangotest/lib/python2.7/site-", line 385, in executefromcommandline utility.execute() File "/home/svencoenye/developer/djangotest/lib/python2.7/site-", line 377, in execute self.fetchcommand(subcommand).runfromargv() File "/home/svencoenye/developer/djangotest/lib/python2.7/site-", line 288, in runfromargv self.execute(args, ) File "/home/svencoenye/developer/djangotest/lib/python2.7/site-", line 338, in execute output = (args, options) File "/home/svencoenye/developer/djangotest/lib/python2.7/site-", line 111, in handle convertapps=applabels or None, File "/home/svencoenye/developer/djangotest/lib/python2.7/site-", line 42, in changes changes = self.detectchanges(convertapps, graph) File "/home/svencoenye/developer/djangotest/lib/python2.7/site-", line 109, in detectchanges self.oldapps = (ignoreswappable=True) File "/home/svencoenye/developer/djangotest/lib/python2.7/site-", line 89, in render model=lookupmodel, ValueError: Lookup failed for model referenced by field asmigrations.Evidence.rubrictype: asmigrations.RubricType The sequence of the operations does not matter. Neither does substituting the RenameModel operation for the AlterField operation. (Looking at the next to last entry in the traceback, the autodetector seems to be looking for the new name in the oldapps state?) It is possible, however, to go the long way around and use two separate migrations: Rubrictype -RubricXype. RubricXype -RubricType works without getting the migration state stuck and does not throw an exception.

dependencies=dependencies,                             )                             self.renamed_models[app_label, model_name] = rem_model_name                            renamed_models_rel_key = '%s.%s' % (rem_model_state.app_label, rem_model_state.name)                            renamed_models_rel_key = '%s.%s' % (                                rem_model_state.app_label,                                rem_model_state.name_lower,                            )                             self.renamed_models_rel[renamed_models_rel_key] = '%s.%s' % (                                 model_state.app_label,                                model_state.name,                                model_state.name_lower,                             )                             self.old_model_keys.remove((rem_app_label, rem_model_name))                             self.old_model_keys.add((app_label, model_name))diff --git a/django/db/models/fields/related.py b/django/db/models/fields/related.py-- a/django/db/models/fields/related.py++ b/django/db/models/fields/related.py


Question: makemigrations does not detect/like model name case changes Description Starting with class Evidence(models.Model): rubrictype = models.ForeignKey('Rubrictype') class Rubrictype(models.Model): typecode = models.CharField(maxlength=1) Make the initial migration: $ makemigrations Migrations for 'asmigrations': Create model Evidence Create model Rubrictype Add field rubrictype to evidence Change the name of Rubrictype to RubricType: class Evidence(models.Model): rubrictype = models.ForeignKey('RubricType') class RubricType(models.Model): typecode = models.CharField(maxlength=1) Generate the migration: $ makemigrations Migrations for 'asmigrations': Alter field rubrictype on evidence Django does not detect the name change on the RubricType model itself. No confirmation is requested for the name change and no operation is generated. The problem is that any subsequent makemigrations run will generate the same operation ad infinitum: $ makemigrations Migrations for 'asmigrations': Alter field rubrictype on evidence If instead the name is changed to RubricXype: class Evidence(models.Model): rubrictype = models.ForeignKey('RubricXype') class RubricXype(models.Model): typecode = models.CharField(maxlength=1) the corresponding migration becomes $ makemigrations Did you rename the asmigrations.Rubrictype model to RubricXype? [y/N] y Migrations for 'asmigrations': Rename model Rubrictype to RubricXype This migration generates a RenameModel operation only and any subsequent makemigrations runs will properly report "No changes detected". So it appears the change detector does not pick up on capitalization changes in model names. Trying to work around by adding a migrations.RenameModel( oldname='Rubrictype', newname='RubricType', ) to the auto generated operations results in a ValueError exception when makemigrations is run again: $ makemigrations Traceback (most recent call last): File "", line 10, in <moduleexecutefromcommandline() File "/home/svencoenye/developer/djangotest/lib/python2.7/site-", line 385, in executefromcommandline utility.execute() File "/home/svencoenye/developer/djangotest/lib/python2.7/site-", line 377, in execute self.fetchcommand(subcommand).runfromargv() File "/home/svencoenye/developer/djangotest/lib/python2.7/site-", line 288, in runfromargv self.execute(args, ) File "/home/svencoenye/developer/djangotest/lib/python2.7/site-", line 338, in execute output = (args, options) File "/home/svencoenye/developer/djangotest/lib/python2.7/site-", line 111, in handle convertapps=applabels or None, File "/home/svencoenye/developer/djangotest/lib/python2.7/site-", line 42, in changes changes = self.detectchanges(convertapps, graph) File "/home/svencoenye/developer/djangotest/lib/python2.7/site-", line 109, in detectchanges self.oldapps = (ignoreswappable=True) File "/home/svencoenye/developer/djangotest/lib/python2.7/site-", line 89, in render model=lookupmodel, ValueError: Lookup failed for model referenced by field asmigrations.Evidence.rubrictype: asmigrations.RubricType The sequence of the operations does not matter. Neither does substituting the RenameModel operation for the AlterField operation. (Looking at the next to last entry in the traceback, the autodetector seems to be looking for the new name in the oldapps state?) It is possible, however, to go the long way around and use two separate migrations: Rubrictype -RubricXype. RubricXype -RubricType works without getting the migration state stuck and does not throw an exception.

if self.remote_field.parent_link:             kwargs['parent_link'] = self.remote_field.parent_link        # Work out string form of "to"         if isinstance(self.remote_field.model, str):            kwargs['to'] = self.remote_field.model            kwargs['to'] = self.remote_field.model.lower()         else:            kwargs['to'] = "%s.%s" % (                self.remote_field.model._meta.app_label,                self.remote_field.model._meta.object_name,            )            kwargs['to'] = self.remote_field.model._meta.label_lower         # If swappable is True, then see if we're actually pointing to the target         # of a swap.         swappable_setting = self.swappable_setting


Question: pk setup for MTI to parent get confused by multiple OneToOne references. Description class Document(models.Model): pass class Picking(Document): documentptr = models.OneToOneField(Document, ondelete=models.CASCADE, parentlink=True, relatedname='+') origin = models.OneToOneField(Document, relatedname='picking', ondelete=models.PROTECT) produces Add parentlink=True to class Picking(Document): origin = models.OneToOneField(Document, relatedname='picking', ondelete=models.PROTECT) documentptr = models.OneToOneField(Document, ondelete=models.CASCADE, parentlink=True, relatedname='+') Works First issue is that order seems to matter? Even if ordering is required "by design"(It shouldn't be we have explicit parentlink marker) shouldn't it look from top to bottom like it does with managers and other things?

continue             # Locate OneToOneField instances.             for field in base._meta.local_fields:                if isinstance(field, OneToOneField):                if isinstance(field, OneToOneField) and field.remote_field.parent_link:                     related = resolve_relation(new_class, field.remote_field.model)                     parent_links[make_model_tuple(related)] = field diff --git a/django/db/models/options.py b/django/db/models/options.py-- a/django/db/models/options.py++ b/django/db/models/options.py


Question: pk setup for MTI to parent get confused by multiple OneToOne references. Description class Document(models.Model): pass class Picking(Document): documentptr = models.OneToOneField(Document, ondelete=models.CASCADE, parentlink=True, relatedname='+') origin = models.OneToOneField(Document, relatedname='picking', ondelete=models.PROTECT) produces Add parentlink=True to class Picking(Document): origin = models.OneToOneField(Document, relatedname='picking', ondelete=models.PROTECT) documentptr = models.OneToOneField(Document, ondelete=models.CASCADE, parentlink=True, relatedname='+') Works First issue is that order seems to matter? Even if ordering is required "by design"(It shouldn't be we have explicit parentlink marker) shouldn't it look from top to bottom like it does with managers and other things?

from django.apps import apps from django.conf import settingsfrom django.core.exceptions import FieldDoesNotExist, ImproperlyConfiguredfrom django.core.exceptions import FieldDoesNotExist from django.db import connections from django.db.models import Manager from django.db.models.fields import AutoField


Question: pk setup for MTI to parent get confused by multiple OneToOne references. Description class Document(models.Model): pass class Picking(Document): documentptr = models.OneToOneField(Document, ondelete=models.CASCADE, parentlink=True, relatedname='+') origin = models.OneToOneField(Document, relatedname='picking', ondelete=models.PROTECT) produces Add parentlink=True to class Picking(Document): origin = models.OneToOneField(Document, relatedname='picking', ondelete=models.PROTECT) documentptr = models.OneToOneField(Document, ondelete=models.CASCADE, parentlink=True, relatedname='+') Works First issue is that order seems to matter? Even if ordering is required "by design"(It shouldn't be we have explicit parentlink marker) shouldn't it look from top to bottom like it does with managers and other things?

field = already_created[0]                 field.primary_key = True                 self.setup_pk(field)<del>                if not field.remote_field.parent_link:                    raise ImproperlyConfigured(                        'Add parent_link=True to %s.' % field,                    )</del>             else:                 auto = AutoField(verbose_name='ID', primary_key=True, auto_created=True)                 model.add_to_class('id', auto)


Question: Admin: Render foreign key models as links for readonly users Description In the admin UI, when viewing a model for which you have view only permission, foreign key / m2m fields are rendered as plaintext representation of the target object. It would be nicer to render those as links instead so that a readonly user can navigate through the relationships. The link should only be rendered if the user has permission to view the target model. django-developers discussion: !topic/django-developers/XFoaohDpqZE

from django import forms from django.contrib.admin.utils import (     display_for_field, flatten_fieldsets, help_text_for_field, label_for_field,    lookup_field,    lookup_field, quote, ) from django.core.exceptions import ObjectDoesNotExistfrom django.db.models import ManyToManyRelfrom django.db.models.fields.related import (    ForeignObjectRel, ManyToManyRel, OneToOneField,) from django.forms.utils import flatatt from django.template.defaultfilters import capfirst, linebreaksbrfrom django.urls import NoReverseMatch, reverse from django.utils.html import conditional_escape, format_html from django.utils.safestring import mark_safe from django.utils.translation import gettext, gettext_lazy as _


Question: Admin: Render foreign key models as links for readonly users Description In the admin UI, when viewing a model for which you have view only permission, foreign key / m2m fields are rendered as plaintext representation of the target object. It would be nicer to render those as links instead so that a readonly user can navigate through the relationships. The link should only be rendered if the user has permission to view the target model. django-developers discussion: !topic/django-developers/XFoaohDpqZE

label = self.field['label']         return format_html('{}{}', flatatt(attrs), capfirst(label), self.form.label_suffix)     def get_admin_url(self, remote_field, remote_obj):        url_name = 'admin:%s_%s_change' % (            remote_field.model._meta.app_label,            remote_field.model._meta.model_name,        )        try:            url = reverse(url_name, args=[quote(remote_obj.pk)])            return format_html('{}', url, remote_obj)        except NoReverseMatch:            return str(remote_obj)     def contents(self):         from django.contrib.admin.templatetags.admin_list import _boolean_icon         field, obj, model_admin = self.field['field'], self.form.instance, self.model_admin


Question: Admin: Render foreign key models as links for readonly users Description In the admin UI, when viewing a model for which you have view only permission, foreign key / m2m fields are rendered as plaintext representation of the target object. It would be nicer to render those as links instead so that a readonly user can navigate through the relationships. The link should only be rendered if the user has permission to view the target model. django-developers discussion: !topic/django-developers/XFoaohDpqZE

else:                 if isinstance(f.remote_field, ManyToManyRel) and value is not None:                     result_repr = ", ".join(map(str, value.all()))<ins>                elif (                    isinstance(f.remote_field, (ForeignObjectRel, OneToOneField)) and                    value is not None                ):                    result_repr = self.get_admin_url(f.remote_field, value)</ins>                 else:                     result_repr = display_for_field(value, f, self.empty_value_display)                 result_repr = linebreaksbr(result_repr)


Question: Add system check for the length of auth permissions codenames. Description I stumbled across this while performing some migrations on models with rather... descriptive names (my original model was dynamically created). Anyway, it looks like in cases where a model name is just under the 100 character limit, and is used, the codenames generated for the default permissions (ie. add, change, delete, view) can exceed the maximum 100 characters on the Permission model. As an example, the model below having a 98 character name allows migrations to be generated, but upon running the migration, a database error is raised when it tries to create permissions with codenames above 100 characters. The model: class SomeExcessivelyDescriptiveModelWithAnAbsolutelyRidiculousNameThatCouldntEverNeedToBeNearlyThisLong(models.Model): field1 = models.CharField(maxlength=25) The error: value too long for type character varying(100) While I'm aware that you can override the default permissions by setting Meta.defaultpermissions on the model (this is how I fixed my app), and that the majority of people would never need model names this long, I figured it might be worth adding validation for this case since it uses 's default behaviour. Also, the error message originally came up while using a flush command, which seemed counterintuitive.

Permission = apps.get_model('auth', 'Permission')     permission_name_max_length = Permission._meta.get_field('name').max_length<ins>    permission_codename_max_length = Permission._meta.get_field('codename').max_length</ins>     errors = []      for model in models:


Question: Add system check for the length of auth permissions codenames. Description I stumbled across this while performing some migrations on models with rather... descriptive names (my original model was dynamically created). Anyway, it looks like in cases where a model name is just under the 100 character limit, and is used, the codenames generated for the default permissions (ie. add, change, delete, view) can exceed the maximum 100 characters on the Permission model. As an example, the model below having a 98 character name allows migrations to be generated, but upon running the migration, a database error is raised when it tries to create permissions with codenames above 100 characters. The model: class SomeExcessivelyDescriptiveModelWithAnAbsolutelyRidiculousNameThatCouldntEverNeedToBeNearlyThisLong(models.Model): field1 = models.CharField(maxlength=25) The error: value too long for type character varying(100) While I'm aware that you can override the default permissions by setting Meta.defaultpermissions on the model (this is how I fixed my app), and that the majority of people would never need model names this long, I figured it might be worth adding validation for this case since it uses 's default behaviour. Also, the error message originally came up while using a flush command, which seemed counterintuitive.

id='auth.E007',                 )             )<ins>        # Check builtin permission codename length.        max_builtin_permission_codename_length = (            max(len(codename) for codename in builtin_permissions.keys())            if builtin_permissions else 0        )        if max_builtin_permission_codename_length > permission_codename_max_length:            model_name_max_length = permission_codename_max_length - (                max_builtin_permission_codename_length - len(opts.model_name)            )            errors.append(                checks.Error(                    "The name of model '%s.%s' must be at most %d characters "                    "for its builtin permission codenames to be at most %d "                    "characters." % (                        opts.app_label,                        opts.object_name,                        model_name_max_length,                        permission_codename_max_length,                    ),                    obj=model,                    id='auth.E011',                )            )</ins>         codenames = set()         for codename, name in opts.permissions:             # Check custom permission name length.


Question: Add system check for the length of auth permissions codenames. Description I stumbled across this while performing some migrations on models with rather... descriptive names (my original model was dynamically created). Anyway, it looks like in cases where a model name is just under the 100 character limit, and is used, the codenames generated for the default permissions (ie. add, change, delete, view) can exceed the maximum 100 characters on the Permission model. As an example, the model below having a 98 character name allows migrations to be generated, but upon running the migration, a database error is raised when it tries to create permissions with codenames above 100 characters. The model: class SomeExcessivelyDescriptiveModelWithAnAbsolutelyRidiculousNameThatCouldntEverNeedToBeNearlyThisLong(models.Model): field1 = models.CharField(maxlength=25) The error: value too long for type character varying(100) While I'm aware that you can override the default permissions by setting Meta.defaultpermissions on the model (this is how I fixed my app), and that the majority of people would never need model names this long, I figured it might be worth adding validation for this case since it uses 's default behaviour. Also, the error message originally came up while using a flush command, which seemed counterintuitive.

id='auth.E008',                     )                 )<ins>            # Check custom permission codename length.            if len(codename) > permission_codename_max_length:                errors.append(                    checks.Error(                        "The permission codenamed '%s' of model '%s.%s' is "                        "longer than %d characters." % (                            codename,                            opts.app_label,                            opts.object_name,                            permission_codename_max_length,                        ),                        obj=model,                        id='auth.E012',                    )                )</ins>             # Check custom permissions codename clashing.             if codename in builtin_permissions:                 errors.append(


Question: Detection of existing total ordering in admin changelist should take into account UniqueConstraints without conditions. Description I've been fiddling with db indexes lately to improve the performance of an admin view. Eventually I found this PR  which ensures the records displayed by ChangeList are deterministically ordered. Among other things, the code looks for the presence of a uniquetogether attribute on the Meta class of the model to determine if the model is already totally ordered. I think that getdeterministicordering() should check for the presence of UniqueConstraints as well, which currently aren’t considered by the discovery algorithm. I noticed the docs currently advise against using unique_together in favor of UniqueConstraint, suggesting the former may be deprecated in the future, so this fix would change that part of the code accordingly. I'm willing to submit a PR for this. Cheers, Fabio

break                 ordering_fields.add(field.attname)         else:            # No single total ordering field, try unique_together.            for field_names in self.lookup_opts.unique_together:            # No single total ordering field, try unique_together and total            # unique constraints.            constraint_field_names = (                *self.lookup_opts.unique_together,                *(                    constraint.fields                    for constraint in self.lookup_opts.total_unique_constraints                ),            )            for field_names in constraint_field_names:                 # Normalize attname references by using get_field().                 fields = [self.lookup_opts.get_field(field_name) for field_name in field_names]                 # Composite unique constraints containing a nullable columndiff --git a/django/db/models/base.py b/django/db/models/base.py-- a/django/db/models/base.py++ b/django/db/models/base.py


Question: Detection of existing total ordering in admin changelist should take into account UniqueConstraints without conditions. Description I've been fiddling with db indexes lately to improve the performance of an admin view. Eventually I found this PR  which ensures the records displayed by ChangeList are deterministically ordered. Among other things, the code looks for the presence of a uniquetogether attribute on the Meta class of the model to determine if the model is already totally ordered. I think that getdeterministicordering() should check for the presence of UniqueConstraints as well, which currently aren’t considered by the discovery algorithm. I noticed the docs currently advise against using unique_together in favor of UniqueConstraint, suggesting the former may be deprecated in the future, so this fix would change that part of the code accordingly. I'm willing to submit a PR for this. Cheers, Fabio

NOT_PROVIDED, ExpressionWrapper, IntegerField, Max, Value, ) from django.db.models.constants import LOOKUP_SEPfrom django.db.models.constraints import CheckConstraint, UniqueConstraintfrom django.db.models.constraints import CheckConstraint from django.db.models.deletion import CASCADE, Collector from django.db.models.fields.related import (     ForeignObjectRel, OneToOneField, lazy_related_operation, resolve_relation,


Question: Detection of existing total ordering in admin changelist should take into account UniqueConstraints without conditions. Description I've been fiddling with db indexes lately to improve the performance of an admin view. Eventually I found this PR  which ensures the records displayed by ChangeList are deterministically ordered. Among other things, the code looks for the presence of a uniquetogether attribute on the Meta class of the model to determine if the model is already totally ordered. I think that getdeterministicordering() should check for the presence of UniqueConstraints as well, which currently aren’t considered by the discovery algorithm. I noticed the docs currently advise against using unique_together in favor of UniqueConstraint, suggesting the former may be deprecated in the future, so this fix would change that part of the code accordingly. I'm willing to submit a PR for this. Cheers, Fabio

unique_checks = []          unique_togethers = [(self.__class__, self._meta.unique_together)]        constraints = [(self.__class__, self._meta.constraints)]        constraints = [(self.__class__, self._meta.total_unique_constraints)]         for parent_class in self._meta.get_parent_list():             if parent_class._meta.unique_together:                 unique_togethers.append((parent_class, parent_class._meta.unique_together))            if parent_class._meta.constraints:                constraints.append((parent_class, parent_class._meta.constraints))            if parent_class._meta.total_unique_constraints:                constraints.append(                    (parent_class, parent_class._meta.total_unique_constraints)                )          for model_class, unique_together in unique_togethers:             for check in unique_together:


Question: Detection of existing total ordering in admin changelist should take into account UniqueConstraints without conditions. Description I've been fiddling with db indexes lately to improve the performance of an admin view. Eventually I found this PR  which ensures the records displayed by ChangeList are deterministically ordered. Among other things, the code looks for the presence of a uniquetogether attribute on the Meta class of the model to determine if the model is already totally ordered. I think that getdeterministicordering() should check for the presence of UniqueConstraints as well, which currently aren’t considered by the discovery algorithm. I noticed the docs currently advise against using unique_together in favor of UniqueConstraint, suggesting the former may be deprecated in the future, so this fix would change that part of the code accordingly. I'm willing to submit a PR for this. Cheers, Fabio

for model_class, model_constraints in constraints:             for constraint in model_constraints:                if (isinstance(constraint, UniqueConstraint) and                        # Partial unique constraints can't be validated.                        constraint.condition is None and                        not any(name in exclude for name in constraint.fields)):                if not any(name in exclude for name in constraint.fields):                     unique_checks.append((model_class, constraint.fields))          # These are checks for the unique_for_<date/year/month>.diff --git a/django/db/models/options.py b/django/db/models/options.py-- a/django/db/models/options.py++ b/django/db/models/options.py


Question: Detection of existing total ordering in admin changelist should take into account UniqueConstraints without conditions. Description I've been fiddling with db indexes lately to improve the performance of an admin view. Eventually I found this PR  which ensures the records displayed by ChangeList are deterministically ordered. Among other things, the code looks for the presence of a uniquetogether attribute on the Meta class of the model to determine if the model is already totally ordered. I think that getdeterministicordering() should check for the presence of UniqueConstraints as well, which currently aren’t considered by the discovery algorithm. I noticed the docs currently advise against using unique_together in favor of UniqueConstraint, suggesting the former may be deprecated in the future, so this fix would change that part of the code accordingly. I'm willing to submit a PR for this. Cheers, Fabio

from django.conf import settings from django.core.exceptions import FieldDoesNotExist from django.db import connectionsfrom django.db.models import AutoField, Manager, OrderWrtfrom django.db.models import AutoField, Manager, OrderWrt, UniqueConstraint from django.db.models.query_utils import PathInfo from django.utils.datastructures import ImmutableList, OrderedSet from django.utils.functional import cached_property


Question: Detection of existing total ordering in admin changelist should take into account UniqueConstraints without conditions. Description I've been fiddling with db indexes lately to improve the performance of an admin view. Eventually I found this PR  which ensures the records displayed by ChangeList are deterministically ordered. Among other things, the code looks for the presence of a uniquetogether attribute on the Meta class of the model to determine if the model is already totally ordered. I think that getdeterministicordering() should check for the presence of UniqueConstraints as well, which currently aren’t considered by the discovery algorithm. I noticed the docs currently advise against using unique_together in favor of UniqueConstraint, suggesting the former may be deprecated in the future, so this fix would change that part of the code accordingly. I'm willing to submit a PR for this. Cheers, Fabio

self._get_fields_cache[cache_key] = fields         return fields <ins>    @cached_property    def total_unique_constraints(self):        """        Return a list of total unique constraints. Useful for determining set        of fields guaranteed to be unique for all rows.        """        return [            constraint            for constraint in self.constraints            if isinstance(constraint, UniqueConstraint) and constraint.condition is None        ]</ins>     @cached_property     def _property_names(self):         """Return a set of the names of the properties defined on the model."""


Question: Raising error about protected related objects can crash. Description (last modified by Matthias Kestenholz) ERROR: testprotectvia () Traceback (most recent call last): File "", line 59, in testPartExecutor yield File "", line 605, in run testMethod() File "", line 99, in testprotectvia p.delete() File "", line 941, in delete collector.collect([self], keepparents=keepparents) File "", line 300, in collect error.protectedobjects[0], TypeError: '' object is not subscriptable Pull request follows. (By the way, this came up while testing !msg/django-developers/WmgqJnQ6ioE/b52uACoPAgAJ / the JSONField GSoC pull request. Apart from this crash everything worked fine!)

try:                         field.remote_field.on_delete(self, field, sub_objs, self.using)                     except ProtectedError as error:                        key = "'%s.%s'" % (                            error.protected_objects[0].__class__.__name__,                            field.name,                        )                        key = "'%s.%s'" % (field.model.__name__, field.name)                         protected_objects[key] += error.protected_objects         if protected_objects:             raise ProtectedError(


Question: ModelForm RadioSelect widget for foreign keys should not present a blank option if blank=False on the model Description Unlike the select widget, where a blank option is idiomatic even for required fields, radioselect has an inherent unfilled state that makes the "-------" option look suspiciously like a valid choice. class TestRun(models.Model): datafile = models.ForeignKey(BatchData, ondelete=models.SETNULL, null=True, blank=False) class TestRunForm(ModelForm): class Meta: model = TestRun fields = ['datafile'] widgets = {'datafile': RadioSelect()} renders {{testrunform.datafile}} as <ul id="iddatafile"<li<label for="iddatafile0"<input checked="checked" id="iddatafile0" name="datafile" type="radio" value=""--------- </label</li<li<label for="iddatafile1"<input id="iddatafile1" name="datafile" type="radio" value="1"First Data File </label</li</ulInstead, there should be no checked option for RadioSelect's <inputtags when rendering a new form from a model if blank is not a valid selection.

'queryset': self.remote_field.model._default_manager.using(using),             'to_field_name': self.remote_field.field_name,             **kwargs,            'blank': self.blank,         })      def db_check(self, connection):diff --git a/django/forms/models.py b/django/forms/models.py-- a/django/forms/models.py++ b/django/forms/models.py


Question: ModelForm RadioSelect widget for foreign keys should not present a blank option if blank=False on the model Description Unlike the select widget, where a blank option is idiomatic even for required fields, radioselect has an inherent unfilled state that makes the "-------" option look suspiciously like a valid choice. class TestRun(models.Model): datafile = models.ForeignKey(BatchData, ondelete=models.SETNULL, null=True, blank=False) class TestRunForm(ModelForm): class Meta: model = TestRun fields = ['datafile'] widgets = {'datafile': RadioSelect()} renders {{testrunform.datafile}} as <ul id="iddatafile"<li<label for="iddatafile0"<input checked="checked" id="iddatafile0" name="datafile" type="radio" value=""--------- </label</li<li<label for="iddatafile1"<input id="iddatafile1" name="datafile" type="radio" value="1"First Data File </label</li</ulInstead, there should be no checked option for RadioSelect's <inputtags when rendering a new form from a model if blank is not a valid selection.

from django.forms.formsets import BaseFormSet, formset_factory from django.forms.utils import ErrorList from django.forms.widgets import (    HiddenInput, MultipleHiddenInput, SelectMultiple,    HiddenInput, MultipleHiddenInput, RadioSelect, SelectMultiple, ) from django.utils.text import capfirst, get_text_list from django.utils.translation import gettext, gettext_lazy as _


Question: ModelForm RadioSelect widget for foreign keys should not present a blank option if blank=False on the model Description Unlike the select widget, where a blank option is idiomatic even for required fields, radioselect has an inherent unfilled state that makes the "-------" option look suspiciously like a valid choice. class TestRun(models.Model): datafile = models.ForeignKey(BatchData, ondelete=models.SETNULL, null=True, blank=False) class TestRunForm(ModelForm): class Meta: model = TestRun fields = ['datafile'] widgets = {'datafile': RadioSelect()} renders {{testrunform.datafile}} as <ul id="iddatafile"<li<label for="iddatafile0"<input checked="checked" id="iddatafile0" name="datafile" type="radio" value=""--------- </label</li<li<label for="iddatafile1"<input id="iddatafile1" name="datafile" type="radio" value="1"First Data File </label</li</ulInstead, there should be no checked option for RadioSelect's <inputtags when rendering a new form from a model if blank is not a valid selection.

def __init__(self, queryset, *, empty_label="---------",                  required=True, widget=None, label=None, initial=None,                  help_text='', to_field_name=None, limit_choices_to=None,                 **kwargs):        if required and (initial is not None):            self.empty_label = None        else:            self.empty_label = empty_label                 blank=False, **kwargs):         # Call Field instead of ChoiceField __init__() because we don't need         # ChoiceField.__init__().         Field.__init__(             self, required=required, widget=widget, label=label,             initial=initial, help_text=help_text, **kwargs         )        if (            (required and initial is not None) or            (isinstance(self.widget, RadioSelect) and not blank)        ):            self.empty_label = None        else:            self.empty_label = empty_label         self.queryset = queryset         self.limit_choices_to = limit_choices_to   # limit the queryset later.         self.to_field_name = to_field_name


Question: {% include %} uses gettemplate where it could selecttemplate Description It'd be nice if the Include template tag was sensible enough to allow fallbacks by selecting the most appropriate template, as things like render/rendertoresponse/rendertostring do. It's tripped me up on more than one occasion, and it seems a trivial feature to support, from my limited testing. from django.template import Template, Context tmpl = Template('{% include var %}') ctx = Context({'var':''}) ctx [{'var': ''}] (ctx) ... some HTML output ... ({'var':['', '']}) {'var': ['', '']} (ctx) Traceback (most recent call last): File "<console", line 1, in <moduleFile "", line 140, in render return (context) File "", line 134, in render return (context) File "", line 823, in render bit = self.rendernode(node, context) File "", line 74, in rendernode return (context) File "", line 165, in render template = gettemplate(templatename) File "", line 145, in gettemplate template, origin = findtemplate(templatename) File "", line 138, in findtemplate raise TemplateDoesNotExist(name) TemplateDoesNotExist: ['', ''] The 'fix' is to change this line from gettemplate to selecttemplate, though this might now be slightly complicated by the recent changes in to allow for rendering of Template instances. Changing to selecttemplate on 1.4 yields the results I'd expect: from django.template import Template, Context tmpl = Template('{% include var %}') ctx = Context({'var':['', '']}) (ctx) ... some HTML output ... Both the above shell sessions assume is in INSTALLED_APPS.

template = self.template.resolve(context)         # Does this quack like a Template?         if not callable(getattr(template, 'render', None)):            # If not, try the cache and get_template().            template_name = template            # If not, try the cache and select_template().            template_name = template or ()            if isinstance(template_name, str):                template_name = (template_name,)            else:                template_name = tuple(template_name)             cache = context.render_context.dicts[0].setdefault(self, {})             template = cache.get(template_name)             if template is None:                template = context.template.engine.get_template(template_name)                template = context.template.engine.select_template(template_name)                 cache[template_name] = template         # Use the base.Template of a backends.django.Template.         elif hasattr(template, 'template'):


Question: Add secure default SECUREREFERRERPOLICY / Referrer-policy header Description the ability for the SECUREREFERRERPOLICY setting to set Referrer-Policy, released in Django 3.0. I propose we change the default for this to "same-origin" to make Django applications leak less information to third party sites. The main risk of breakage here would be linked websites breaking, if they depend on verification through the Referer header. This is a pretty fragile technique since it can be spoofed. Documentation:  The MDN support grid is out of date:

SECURE_HSTS_PRELOAD = False SECURE_HSTS_SECONDS = 0 SECURE_REDIRECT_EXEMPT = []SECURE_REFERRER_POLICY = NoneSECURE_REFERRER_POLICY = 'same-origin' SECURE_SSL_HOST = None SECURE_SSL_REDIRECT = False


Question: Possible data loss when using caching from async code. Description CacheHandler use instead of , hence it's a chance of data corruption if someone tries to use caching from async code. There is a potential race condition if two coroutines touch the same cache object at exactly the same time.

See docs/topics/cache.txt for information on the public API. """from threading import localfrom asgiref.local import Local  from django.conf import settings from django.core import signals


Question: Possible data loss when using caching from async code. Description CacheHandler use instead of , hence it's a chance of data corruption if someone tries to use caching from async code. There is a potential race condition if two coroutines touch the same cache object at exactly the same time.

Ensure only one instance of each alias exists per thread.     """     def __init__(self):        self._caches = local()        self._caches = Local()      def __getitem__(self, alias):         try:


Question: FileResponse with temporary file closing connection. Description (last modified by Oskar Persson) I think I might've found a regression in . When I run the following tests (in their defined order) against Postgres I get the error below. import tempfile from import getusermodel from import FileResponse from import TestCase User = getusermodel() class MyTests(TestCase): def setUp(self): = (username='user') def testfirst(self): with tempfile.TemporaryFile() as f: return FileResponse(f) def testsecond(self): pass Running tests... .E ERROR [0.003s]: testsecond (responses.testfileresponse.MyTests) Traceback (most recent call last): File "", line 238, in cursor return self.preparecursor(self.createcursor(name)) File "", line 26, in inner return func(args, kwargs) File "", line 231, in createcursor cursor = () psycopg2.InterfaceError: connection already closed The above exception was the direct cause of the following exception: Traceback (most recent call last): File "", line 19, in setUp = (username='user') File "", line 82, in managermethod return getattr(self.getqueryset(), name)(args, kwargs) File "", line 433, in create (forceinsert=True, ) File "", line 66, in save super().save(args, kwargs) File "", line 746, in save forceupdate=forceupdate, updatefields=updatefields) File "", line 784, in savebase forceupdate, using, updatefields, File "", line 887, in savetable results = self.doinsert(cls.basemanager, using, fields, returningfields, raw) File "", line 926, in doinsert using=using, raw=raw, File "", line 82, in managermethod return getattr(self.getqueryset(), name)(args, kwargs) File "", line 1204, in insert return query.getcompiler(using=using).executesql(returningfields) File "", line 1382, in executesql with () as cursor: File "", line 26, in inner return func(args, kwargs) File "", line 260, in cursor return () File "", line 238, in cursor return self.preparecursor(self.createcursor(name)) File "", line 90, in exit raise djexcvalue.withtraceback(traceback) from excvalue File "", line 238, in cursor return self.preparecursor(self.createcursor(name)) File "", line 26, in inner return func(args, kwargs) File "", line 231, in create_cursor cursor = () connection already closed

# Setup default url resolver for this thread         set_urlconf(settings.ROOT_URLCONF)         response = self._middleware_chain(request)        response._closable_objects.append(request)        response._resource_closers.append(request.close)         if response.status_code >= 400:             log_response(                 '%s: %s', response.reason_phrase, request.path,diff --git a/django/core/handlers/wsgi.py b/django/core/handlers/wsgi.py-- a/django/core/handlers/wsgi.py++ b/django/core/handlers/wsgi.py


Question: FileResponse with temporary file closing connection. Description (last modified by Oskar Persson) I think I might've found a regression in . When I run the following tests (in their defined order) against Postgres I get the error below. import tempfile from import getusermodel from import FileResponse from import TestCase User = getusermodel() class MyTests(TestCase): def setUp(self): = (username='user') def testfirst(self): with tempfile.TemporaryFile() as f: return FileResponse(f) def testsecond(self): pass Running tests... .E ERROR [0.003s]: testsecond (responses.testfileresponse.MyTests) Traceback (most recent call last): File "", line 238, in cursor return self.preparecursor(self.createcursor(name)) File "", line 26, in inner return func(args, kwargs) File "", line 231, in createcursor cursor = () psycopg2.InterfaceError: connection already closed The above exception was the direct cause of the following exception: Traceback (most recent call last): File "", line 19, in setUp = (username='user') File "", line 82, in managermethod return getattr(self.getqueryset(), name)(args, kwargs) File "", line 433, in create (forceinsert=True, ) File "", line 66, in save super().save(args, kwargs) File "", line 746, in save forceupdate=forceupdate, updatefields=updatefields) File "", line 784, in savebase forceupdate, using, updatefields, File "", line 887, in savetable results = self.doinsert(cls.basemanager, using, fields, returningfields, raw) File "", line 926, in doinsert using=using, raw=raw, File "", line 82, in managermethod return getattr(self.getqueryset(), name)(args, kwargs) File "", line 1204, in insert return query.getcompiler(using=using).executesql(returningfields) File "", line 1382, in executesql with () as cursor: File "", line 26, in inner return func(args, kwargs) File "", line 260, in cursor return () File "", line 238, in cursor return self.preparecursor(self.createcursor(name)) File "", line 90, in exit raise djexcvalue.withtraceback(traceback) from excvalue File "", line 238, in cursor return self.preparecursor(self.createcursor(name)) File "", line 26, in inner return func(args, kwargs) File "", line 231, in create_cursor cursor = () connection already closed

]         start_response(status, response_headers)         if getattr(response, 'file_to_stream', None) is not None and environ.get('wsgi.file_wrapper'):            # If `wsgi.file_wrapper` is used the WSGI server does not call            # .close on the response, but on the file wrapper. Patch it to use            # response.close instead which takes care of closing all files.            response.file_to_stream.close = response.close             response = environ['wsgi.file_wrapper'](response.file_to_stream, response.block_size)         return response diff --git a/django/http/response.py b/django/http/response.py-- a/django/http/response.py++ b/django/http/response.py


Question: FileResponse with temporary file closing connection. Description (last modified by Oskar Persson) I think I might've found a regression in . When I run the following tests (in their defined order) against Postgres I get the error below. import tempfile from import getusermodel from import FileResponse from import TestCase User = getusermodel() class MyTests(TestCase): def setUp(self): = (username='user') def testfirst(self): with tempfile.TemporaryFile() as f: return FileResponse(f) def testsecond(self): pass Running tests... .E ERROR [0.003s]: testsecond (responses.testfileresponse.MyTests) Traceback (most recent call last): File "", line 238, in cursor return self.preparecursor(self.createcursor(name)) File "", line 26, in inner return func(args, kwargs) File "", line 231, in createcursor cursor = () psycopg2.InterfaceError: connection already closed The above exception was the direct cause of the following exception: Traceback (most recent call last): File "", line 19, in setUp = (username='user') File "", line 82, in managermethod return getattr(self.getqueryset(), name)(args, kwargs) File "", line 433, in create (forceinsert=True, ) File "", line 66, in save super().save(args, kwargs) File "", line 746, in save forceupdate=forceupdate, updatefields=updatefields) File "", line 784, in savebase forceupdate, using, updatefields, File "", line 887, in savetable results = self.doinsert(cls.basemanager, using, fields, returningfields, raw) File "", line 926, in doinsert using=using, raw=raw, File "", line 82, in managermethod return getattr(self.getqueryset(), name)(args, kwargs) File "", line 1204, in insert return query.getcompiler(using=using).executesql(returningfields) File "", line 1382, in executesql with () as cursor: File "", line 26, in inner return func(args, kwargs) File "", line 260, in cursor return () File "", line 238, in cursor return self.preparecursor(self.createcursor(name)) File "", line 90, in exit raise djexcvalue.withtraceback(traceback) from excvalue File "", line 238, in cursor return self.preparecursor(self.createcursor(name)) File "", line 26, in inner return func(args, kwargs) File "", line 231, in create_cursor cursor = () connection already closed

# the header (required for working with legacy systems) and the header         # value. Both the name of the header and its value are ASCII strings.         self._headers = {}        self._closable_objects = []        self._resource_closers = []         # This parameter is set by the handler. It's necessary to preserve the         # historical behavior of request_finished.         self._handler_class = None


Question: FileResponse with temporary file closing connection. Description (last modified by Oskar Persson) I think I might've found a regression in . When I run the following tests (in their defined order) against Postgres I get the error below. import tempfile from import getusermodel from import FileResponse from import TestCase User = getusermodel() class MyTests(TestCase): def setUp(self): = (username='user') def testfirst(self): with tempfile.TemporaryFile() as f: return FileResponse(f) def testsecond(self): pass Running tests... .E ERROR [0.003s]: testsecond (responses.testfileresponse.MyTests) Traceback (most recent call last): File "", line 238, in cursor return self.preparecursor(self.createcursor(name)) File "", line 26, in inner return func(args, kwargs) File "", line 231, in createcursor cursor = () psycopg2.InterfaceError: connection already closed The above exception was the direct cause of the following exception: Traceback (most recent call last): File "", line 19, in setUp = (username='user') File "", line 82, in managermethod return getattr(self.getqueryset(), name)(args, kwargs) File "", line 433, in create (forceinsert=True, ) File "", line 66, in save super().save(args, kwargs) File "", line 746, in save forceupdate=forceupdate, updatefields=updatefields) File "", line 784, in savebase forceupdate, using, updatefields, File "", line 887, in savetable results = self.doinsert(cls.basemanager, using, fields, returningfields, raw) File "", line 926, in doinsert using=using, raw=raw, File "", line 82, in managermethod return getattr(self.getqueryset(), name)(args, kwargs) File "", line 1204, in insert return query.getcompiler(using=using).executesql(returningfields) File "", line 1382, in executesql with () as cursor: File "", line 26, in inner return func(args, kwargs) File "", line 260, in cursor return () File "", line 238, in cursor return self.preparecursor(self.createcursor(name)) File "", line 90, in exit raise djexcvalue.withtraceback(traceback) from excvalue File "", line 238, in cursor return self.preparecursor(self.createcursor(name)) File "", line 26, in inner return func(args, kwargs) File "", line 231, in create_cursor cursor = () connection already closed

# The WSGI server must call this method upon completion of the request.     # See http://blog.dscpl.com.au/2012/10/obligations-for-calling-close-on.html    # When wsgi.file_wrapper is used, the WSGI server instead calls close()    # on the file-like object. Django ensures this method is called in this    # case by replacing self.file_to_stream.close() with a wrapped version.     def close(self):        for closable in self._closable_objects:        for closer in self._resource_closers:             try:                closable.close()                closer()             except Exception:                 pass        # Free resources that were still referenced.        self._resource_closers.clear()         self.closed = True         signals.request_finished.send(sender=self._handler_class)


Question: FileResponse with temporary file closing connection. Description (last modified by Oskar Persson) I think I might've found a regression in . When I run the following tests (in their defined order) against Postgres I get the error below. import tempfile from import getusermodel from import FileResponse from import TestCase User = getusermodel() class MyTests(TestCase): def setUp(self): = (username='user') def testfirst(self): with tempfile.TemporaryFile() as f: return FileResponse(f) def testsecond(self): pass Running tests... .E ERROR [0.003s]: testsecond (responses.testfileresponse.MyTests) Traceback (most recent call last): File "", line 238, in cursor return self.preparecursor(self.createcursor(name)) File "", line 26, in inner return func(args, kwargs) File "", line 231, in createcursor cursor = () psycopg2.InterfaceError: connection already closed The above exception was the direct cause of the following exception: Traceback (most recent call last): File "", line 19, in setUp = (username='user') File "", line 82, in managermethod return getattr(self.getqueryset(), name)(args, kwargs) File "", line 433, in create (forceinsert=True, ) File "", line 66, in save super().save(args, kwargs) File "", line 746, in save forceupdate=forceupdate, updatefields=updatefields) File "", line 784, in savebase forceupdate, using, updatefields, File "", line 887, in savetable results = self.doinsert(cls.basemanager, using, fields, returningfields, raw) File "", line 926, in doinsert using=using, raw=raw, File "", line 82, in managermethod return getattr(self.getqueryset(), name)(args, kwargs) File "", line 1204, in insert return query.getcompiler(using=using).executesql(returningfields) File "", line 1382, in executesql with () as cursor: File "", line 26, in inner return func(args, kwargs) File "", line 260, in cursor return () File "", line 238, in cursor return self.preparecursor(self.createcursor(name)) File "", line 90, in exit raise djexcvalue.withtraceback(traceback) from excvalue File "", line 238, in cursor return self.preparecursor(self.createcursor(name)) File "", line 26, in inner return func(args, kwargs) File "", line 231, in create_cursor cursor = () connection already closed

# Ensure we can never iterate on "value" more than once.         self._iterator = iter(value)         if hasattr(value, 'close'):            self._closable_objects.append(value)            self._resource_closers.append(value.close)      def __iter__(self):         return self.streaming_content


Question: FileResponse with temporary file closing connection. Description (last modified by Oskar Persson) I think I might've found a regression in . When I run the following tests (in their defined order) against Postgres I get the error below. import tempfile from import getusermodel from import FileResponse from import TestCase User = getusermodel() class MyTests(TestCase): def setUp(self): = (username='user') def testfirst(self): with tempfile.TemporaryFile() as f: return FileResponse(f) def testsecond(self): pass Running tests... .E ERROR [0.003s]: testsecond (responses.testfileresponse.MyTests) Traceback (most recent call last): File "", line 238, in cursor return self.preparecursor(self.createcursor(name)) File "", line 26, in inner return func(args, kwargs) File "", line 231, in createcursor cursor = () psycopg2.InterfaceError: connection already closed The above exception was the direct cause of the following exception: Traceback (most recent call last): File "", line 19, in setUp = (username='user') File "", line 82, in managermethod return getattr(self.getqueryset(), name)(args, kwargs) File "", line 433, in create (forceinsert=True, ) File "", line 66, in save super().save(args, kwargs) File "", line 746, in save forceupdate=forceupdate, updatefields=updatefields) File "", line 784, in savebase forceupdate, using, updatefields, File "", line 887, in savetable results = self.doinsert(cls.basemanager, using, fields, returningfields, raw) File "", line 926, in doinsert using=using, raw=raw, File "", line 82, in managermethod return getattr(self.getqueryset(), name)(args, kwargs) File "", line 1204, in insert return query.getcompiler(using=using).executesql(returningfields) File "", line 1382, in executesql with () as cursor: File "", line 26, in inner return func(args, kwargs) File "", line 260, in cursor return () File "", line 238, in cursor return self.preparecursor(self.createcursor(name)) File "", line 90, in exit raise djexcvalue.withtraceback(traceback) from excvalue File "", line 238, in cursor return self.preparecursor(self.createcursor(name)) File "", line 26, in inner return func(args, kwargs) File "", line 231, in create_cursor cursor = () connection already closed

self.filename = filename         super().__init__(*args, **kwargs)     def _wrap_file_to_stream_close(self, filelike):        """        Wrap the file-like close() with a version that calls        FileResponse.close().        """        closing = False        filelike_close = getattr(filelike, 'close', lambda: None)        def file_wrapper_close():            nonlocal closing            # Prevent an infinite loop since FileResponse.close() tries to            # close the objects in self._closable_objects.            if closing:                return            closing = True            try:                filelike_close()            finally:                self.close()        filelike.close = file_wrapper_close     def _set_streaming_content(self, value):         if not hasattr(value, 'read'):             self.file_to_stream = None             return super()._set_streaming_content(value)          self.file_to_stream = filelike = value        # Add to closable objects before wrapping close(), since the filelike        # might not have close().         if hasattr(filelike, 'close'):            self._closable_objects.append(filelike)        self._wrap_file_to_stream_close(filelike)            self._resource_closers.append(filelike.close)         value = iter(lambda: filelike.read(self.block_size), b'')         self.set_headers(filelike)         super()._set_streaming_content(value)


Question: Calling a form method htmloutput modifies the dict for NONFIELDERRORS if there are hidden field with errors Description Each time the htmloutput method of a form is called, it appends the errors of the hidden field errors to the NONFIELDERRORS (all) entry. This happen for example when the form methods asp() astable() asul() are called multiple time, or any other method that themselves call one of them. For example, a test form with an hidden input field that add errors during the clean call. Python 3.6.5 (default, Apr 25 2018, 14:26:36) Type 'copyright', 'credits' or 'license' for more information IPython 6.4.0 -- An enhanced Interactive Python. Type '?' for help. In [1]: import django In [2]: django.version Out[2]: '2.1.7' In [3]: from django import forms ...: In [4]: class TestForm(forms.Form): ...: hiddeninput = forms.CharField(widget=forms.HiddenInput) ...: ...: def clean(self): ...: self.adderror(None, 'Form error') ...: self.adderror('hiddeninput', 'Hidden input error') ...: In [5]: testform = TestForm({}) In [6]: Out[6]: {'hiddeninput': ['This field is required.', 'Hidden input error'], 'all': ['Form error']} In [7]: print(testform.astable()) <tr<td colspan="2"<ul class="errorlist nonfield"<liForm error</li<li(Hidden field hiddeninput) This field is required.</li<li(Hidden field hiddeninput) Hidden input error</li</ul<input type="hidden" name="hiddeninput" id="idhiddeninput"</td</trIn [8]: Out[8]: {'hiddeninput': ['This field is required.', 'Hidden input error'], 'all': ['Form error', '(Hidden field hiddeninput) This field is required.', '(Hidden field hiddeninput) Hidden input error']} In [9]: print(testform.astable()) <tr<td colspan="2"<ul class="errorlist nonfield"<liForm error</li<li(Hidden field hiddeninput) This field is required.</li<li(Hidden field hiddeninput) Hidden input error</li<li(Hidden field hiddeninput) This field is required.</li<li(Hidden field hiddeninput) Hidden input error</li</ul<input type="hidden" name="hiddeninput" id="idhiddeninput"</td</trIn [10]: Out[10]: {'hiddeninput': ['This field is required.', 'Hidden input error'], 'all': ['Form error', '(Hidden field hiddeninput) This field is required.', '(Hidden field hiddeninput) Hidden input error', '(Hidden field hiddeninput) This field is required.', '(Hidden field hiddeninput) Hidden input error']} In [11]: testform.nonfielderrors() Out[11]: ['Form error', '(Hidden field hiddeninput) This field is required.', '(Hidden field hiddeninput) Hidden input error', '(Hidden field hiddeninput) This field is required.', '(Hidden field hiddeninput) Hidden input error'] This bug affects probably also version 2.2. A simple fix would be to use a copy of the error list before adding the hidden field errors in the file --- 2019-03-17 18:59:04. +0100 +++ 2019-03-17 19:00:08. +0100 -194,7 +194,7 def htmloutput(self, normalrow, errorrow, rowender, helptexthtml, errorsonseparaterow): "Output HTML. Used by astable(), asul(), asp()." toperrors = self.nonfielderrors() # Errors that should be displayed above all fields. toperrors = self.nonfielderrors().copy() # Errors that should be displayed above all fields. output, hidden_fields = [], [] for name, field in ():

def _html_output(self, normal_row, error_row, row_ender, help_text_html, errors_on_separate_row):         "Output HTML. Used by as_table(), as_ul(), as_p()."        top_errors = self.non_field_errors()  # Errors that should be displayed above all fields.        # Errors that should be displayed above all fields.        top_errors = self.non_field_errors().copy()         output, hidden_fields = [], []          for name, field in self.fields.items():diff --git a/django/forms/utils.py b/django/forms/utils.py-- a/django/forms/utils.py++ b/django/forms/utils.py


Question: Calling a form method htmloutput modifies the dict for NONFIELDERRORS if there are hidden field with errors Description Each time the htmloutput method of a form is called, it appends the errors of the hidden field errors to the NONFIELDERRORS (all) entry. This happen for example when the form methods asp() astable() asul() are called multiple time, or any other method that themselves call one of them. For example, a test form with an hidden input field that add errors during the clean call. Python 3.6.5 (default, Apr 25 2018, 14:26:36) Type 'copyright', 'credits' or 'license' for more information IPython 6.4.0 -- An enhanced Interactive Python. Type '?' for help. In [1]: import django In [2]: django.version Out[2]: '2.1.7' In [3]: from django import forms ...: In [4]: class TestForm(forms.Form): ...: hiddeninput = forms.CharField(widget=forms.HiddenInput) ...: ...: def clean(self): ...: self.adderror(None, 'Form error') ...: self.adderror('hiddeninput', 'Hidden input error') ...: In [5]: testform = TestForm({}) In [6]: Out[6]: {'hiddeninput': ['This field is required.', 'Hidden input error'], 'all': ['Form error']} In [7]: print(testform.astable()) <tr<td colspan="2"<ul class="errorlist nonfield"<liForm error</li<li(Hidden field hiddeninput) This field is required.</li<li(Hidden field hiddeninput) Hidden input error</li</ul<input type="hidden" name="hiddeninput" id="idhiddeninput"</td</trIn [8]: Out[8]: {'hiddeninput': ['This field is required.', 'Hidden input error'], 'all': ['Form error', '(Hidden field hiddeninput) This field is required.', '(Hidden field hiddeninput) Hidden input error']} In [9]: print(testform.astable()) <tr<td colspan="2"<ul class="errorlist nonfield"<liForm error</li<li(Hidden field hiddeninput) This field is required.</li<li(Hidden field hiddeninput) Hidden input error</li<li(Hidden field hiddeninput) This field is required.</li<li(Hidden field hiddeninput) Hidden input error</li</ul<input type="hidden" name="hiddeninput" id="idhiddeninput"</td</trIn [10]: Out[10]: {'hiddeninput': ['This field is required.', 'Hidden input error'], 'all': ['Form error', '(Hidden field hiddeninput) This field is required.', '(Hidden field hiddeninput) Hidden input error', '(Hidden field hiddeninput) This field is required.', '(Hidden field hiddeninput) Hidden input error']} In [11]: testform.nonfielderrors() Out[11]: ['Form error', '(Hidden field hiddeninput) This field is required.', '(Hidden field hiddeninput) Hidden input error', '(Hidden field hiddeninput) This field is required.', '(Hidden field hiddeninput) Hidden input error'] This bug affects probably also version 2.2. A simple fix would be to use a copy of the error list before adding the hidden field errors in the file --- 2019-03-17 18:59:04. +0100 +++ 2019-03-17 19:00:08. +0100 -194,7 +194,7 def htmloutput(self, normalrow, errorrow, rowender, helptexthtml, errorsonseparaterow): "Output HTML. Used by astable(), asul(), asp()." toperrors = self.nonfielderrors() # Errors that should be displayed above all fields. toperrors = self.nonfielderrors().copy() # Errors that should be displayed above all fields. output, hidden_fields = [], [] for name, field in ():

def as_data(self):         return ValidationError(self.data).error_list <ins>    def copy(self):        copy = super().copy()        copy.error_class = self.error_class        return copy</ins>     def get_json_data(self, escape_html=False):         errors = []         for error in self.as_data():


Question: fails to restore objects due to ordering constraints Description I hit this problem in a fairly complex projet and haven't had the time to write a minimal reproduction case. I think it can be understood just by inspecting the code so I'm going to describe it while I have it in mind. Setting serializedrollback = True on a TransactionTestCase triggers rollback emulation. In practice, for each database: BaseDatabaseCreation.createtestdb calls connection.testserializedcontents = connection.creation.serializedbtostring() TransactionTestCase.fixturesetup calls connection.creation.deserializedbfromstring(connection.testserializedcontents) (The actual code isn't written that way; it's equivalent but the symmetry is less visible.) serializedbtostring orders models with serializers.sortdependencies and serializes them. The sorting algorithm only deals with natural keys. It doesn't do anything to order models referenced by foreign keys before models containing said foreign keys. That wouldn't be possible in general because circular foreign keys are allowed. deserializedbfromstring deserializes and saves models without wrapping in a transaction. This can result in integrity errors if an instance containing a foreign key is saved before the instance it references. I'm suggesting to fix it as follows: diff --git index .. --- +++ -4,7 +4,7 import time from import apps from import settings from import serializers -from import router +from import router, transaction from import StringIO from import input -128,8 +128,9 class BaseDatabaseCreation(object): the serializedbtostring method. """ data = StringIO(data) for obj in serializers.deserialize("json", data, ): () with (): for obj in serializers.deserialize("json", data, ): () def getdatabasedisplaystr(self, verbosity, databasename): """ Note that loaddata doesn't have this problem because it wraps everything in a transaction: def handle(self, fixturelabels, *options): # ... with (): self.loaddata(fixturelabels) # ... This suggest that the transaction was just forgotten in the implementation of deserializedbfromstring. It should be possible to write a deterministic test for this bug because the order in which serializedbto_string serializes models depends on the app registry, and the app registry uses OrderedDict to store apps and models in a deterministic order.

from django.conf import settings from django.core import serializers from django.db import router<ins>from django.db.transaction import atomic</ins>  # The prefix to put on the default database name when creating # the test database.


Question: fails to restore objects due to ordering constraints Description I hit this problem in a fairly complex projet and haven't had the time to write a minimal reproduction case. I think it can be understood just by inspecting the code so I'm going to describe it while I have it in mind. Setting serializedrollback = True on a TransactionTestCase triggers rollback emulation. In practice, for each database: BaseDatabaseCreation.createtestdb calls connection.testserializedcontents = connection.creation.serializedbtostring() TransactionTestCase.fixturesetup calls connection.creation.deserializedbfromstring(connection.testserializedcontents) (The actual code isn't written that way; it's equivalent but the symmetry is less visible.) serializedbtostring orders models with serializers.sortdependencies and serializes them. The sorting algorithm only deals with natural keys. It doesn't do anything to order models referenced by foreign keys before models containing said foreign keys. That wouldn't be possible in general because circular foreign keys are allowed. deserializedbfromstring deserializes and saves models without wrapping in a transaction. This can result in integrity errors if an instance containing a foreign key is saved before the instance it references. I'm suggesting to fix it as follows: diff --git index .. --- +++ -4,7 +4,7 import time from import apps from import settings from import serializers -from import router +from import router, transaction from import StringIO from import input -128,8 +128,9 class BaseDatabaseCreation(object): the serializedbtostring method. """ data = StringIO(data) for obj in serializers.deserialize("json", data, ): () with (): for obj in serializers.deserialize("json", data, ): () def getdatabasedisplaystr(self, verbosity, databasename): """ Note that loaddata doesn't have this problem because it wraps everything in a transaction: def handle(self, fixturelabels, *options): # ... with (): self.loaddata(fixturelabels) # ... This suggest that the transaction was just forgotten in the implementation of deserializedbfromstring. It should be possible to write a deterministic test for this bug because the order in which serializedbto_string serializes models depends on the app registry, and the app registry uses OrderedDict to store apps and models in a deterministic order.

the serialize_db_to_string() method.         """         data = StringIO(data)        for obj in serializers.deserialize("json", data, using=self.connection.alias):            obj.save()        # Load data in a transaction to handle forward references and cycles.        with atomic(using=self.connection.alias):            # Disable constraint checks, because some databases (MySQL) doesn't            # support deferred checks.            with self.connection.constraint_checks_disabled():                for obj in serializers.deserialize('json', data, using=self.connection.alias):                    obj.save()            # Manually check for any invalid keys that might have been added,            # because constraint checks were disabled.            self.connection.check_constraints()      def _get_database_display_str(self, verbosity, database_name):         """


Question: Serialization dependency sorting disallows circular references unneccesarily. Description The core.serialization.sortdependencies() function takes a list of apps and/or models, and resolves this into a sorted flat list of models, ready to be serialized in that order. This function is intended to make natural foreign keys work, by serializing models referenced by a natural key before the referencing models. When deserializing, this guarantees that natural key references can be resolved, because there are no "forward references". Furthermore, when a circular reference using natural keys is present, this function raises an exception (e.g. "Can't resolve dependencies for someapp.SomeModel in serialized app list") and prevents serialization from completing, since there is no way to guarantee a model ordering that will have no forward references. Note that this ordering is only needed when natural keys are involved, since data is intended to be loaded in a transaction without constraint checks, so numerical foreign keys can be in the wrong order, as long as all referenced data is present at the end of the transaction. This does not work with natural keys, since those are resolved by Python code that needs the referenced objects present in the database to resolve them. However, this sorting is not actually strictly necessary in all cases where it is applied. When circular references are involved, this then actually prevents serialization for no good reason. In particular, this is the case: When running dumpdata without natural keys enabled (which is the default). Even though natural keys might be defined in the models (which causes the sorting and exception), no natural keys will be present in the dumped data, so no ordering is needed. When dumping data intended for loading with loaddata (which I think is the primary usecase for dumpdata?). loaddata will (since 17 months ago in v2.2, see ) automatically handle forward references by deferring setting fields that reference natural keys that are not yet. In this case, sorting is still useful, to prevent forward references where possible, but when there are circular references, it is acceptable to ignore some dependencies rather than preventing serialization from happening alltogether. When serializing data for tests for serializedrollback=True (in ). This is a serialization that does not use natural keys, so no ordering is needed at all. Note that this serialization happens always (unlike deserialization only happens with serializedrollback=True), so AFAIU this effectively prevents any tests from working on a database with circular references with natural keys defined. The fix for these issues seems to be rather simple: For dumpdata without usenaturalforeignkeys, skip the ordering and just serialize all models in arbitrary order. AFAICS usenaturalprimarykeys is not relevant here, since that only controls omitting the numerical primary key. For dumpdata with usenaturalforeignkeys, do the ordering but do not bail out when there are circular references (instead just ignore some dependencies and produce a best-effort ordering). For test database serialization, also skip the ordering and serialize in arbitrary order. Note that this would remove two of the three calls to sortdependencies() and allow loops in the last remaining instance. This means that sort_dependencies could be modified to allow loops unconditionally, or we could add an argument and default to disallowing loops in case any code outside of django is using this function? Note that is a related, but different issue, concerning the deserialization of data in testcases. I've been working on fixing this and that related issue today and have a basic version working, with testcases (which proved to be quite a challenge, since testing the test subsystem is a bit tricky...). I'll do some additional testing and cleanup and submit a PR soon. Also note that the circular-reference exception was already disabled for self-referencing models in . The fix for that issue simply ignores self-referencing models for sorting, without taking any additional measures to sort instances to prevent problems in deserialization (this code was when the deferred deserialization did not exist yet), so I wonder how much value this exception still has.

Collate the objects to be serialized. If count_only is True, just             count the number of objects to be serialized.             """            models = serializers.sort_dependencies(app_list.items())            if use_natural_foreign_keys:                models = serializers.sort_dependencies(app_list.items(), allow_cycles=True)            else:                # There is no need to sort dependencies when natural foreign                # keys are not used.                models = []                for (app_config, model_list) in app_list.items():                    if model_list is None:                        models.extend(app_config.get_models())                    else:                        models.extend(model_list)             for model in models:                 if model in excluded_models:                     continuediff --git a/django/core/serializers/__init__.py b/django/core/serializers/__init__.py-- a/django/core/serializers/__init__.py++ b/django/core/serializers/__init__.py