Question: ManyToManyField does not respect the PositiveBigIntegerField in m2m intermediate table. Description When creating a relation between 2 models using PositiveBigIntegerField on Postgresql, the relation table is created using regular ints as the column type. This in turn leads to out of bound error when using large numbers for ids. from import models class Node(models.Model): id = models.PositiveBigIntegerField(primarykey=True) point = models.PointField() class Relation(models.Model): id = models.PositiveBigIntegerField(primarykey=True) nodes = models.ManyToManyField(Node) The generated table will look like this: Column Description -------------+---------+-----------+----------+------------------------------------------------+---------+--------------+------------- id relationid nodeid As you can see, the PositiveBigInteger is not respected and a regular int is set

})  class PositiveSmallIntegerField(PositiveIntegerRelDbTypeMixin, IntegerField):class PositiveSmallIntegerField(PositiveIntegerRelDbTypeMixin, SmallIntegerField):     description = _("Positive small integer")      def get_internal_type(self):


Question: ManyToManyField does not respect the PositiveBigIntegerField in m2m intermediate table. Description When creating a relation between 2 models using PositiveBigIntegerField on Postgresql, the relation table is created using regular ints as the column type. This in turn leads to out of bound error when using large numbers for ids. from import models class Node(models.Model): id = models.PositiveBigIntegerField(primarykey=True) point = models.PointField() class Relation(models.Model): id = models.PositiveBigIntegerField(primarykey=True) nodes = models.ManyToManyField(Node) The generated table will look like this: Column Description -------------+---------+-----------+----------+------------------------------------------------+---------+--------------+------------- id relationid nodeid As you can see, the PositiveBigInteger is not respected and a regular int is set

})  <del>class SmallIntegerField(IntegerField):    description = _("Small integer")    def get_internal_type(self):        return "SmallIntegerField"</del> class TextField(Field):     description = _("Text")


Question: BaseFormSet.isvalid() produces ValidationError when there is no management form Description I was torn between reporting this as a bug or a feature request, but then I thought if I make it a feature request it will most likely break a lot of Django apps and hence I guess that means it's more of a bug... Anyway so the line in question is (in Django version 1.6.1): for i in range(0, self.totalformcount()): the self.totalformcount() executes this line (in Django version 1.6.1): return min(self.managementform.cleaneddata[TOTALFORMCOUNT], self.absolutemax) ..which then raises this exception (in Django version 1.6.1): raise ValidationError( ('ManagementForm data is missing or has been tampered with'), code='missingmanagementform', ) That stack trace occurs if/when a user submits a formset after stripping out the management form hidden fields. I have been using Django for a few years now and have never come across an exception being raised by a form/formset isvalid() call before. So my point is that I believe this exception should never be allowed to leave the BaseFormSet.isvalid() call, because it is an oddball behaviour compared to the rest of the isvalid() implementations. I.e. I believe there should be a check in BaseFormSet.isvalid() which checks for the presence of a valid management form (first) and returns False if it is not present, as opposed to raising an exception. Yes I could wrap the isvalid() call in a try/catch, but I believe this is an unnecessary hack caused by a bad design deviation of the implementation of the BaseFormSet.is_valid() method. I didn't bother creating a patch and test cases, because I have a feeling this will get rejected or something like that, but I just thought I should bring this up, as I can't find mention of it anywhere and it seems important to me.

from django.utils.functional import cached_property from django.utils.html import html_safe from django.utils.safestring import mark_safefrom django.utils.translation import gettext as _, ngettextfrom django.utils.translation import gettext_lazy as _, ngettext  __all__ = ('BaseFormSet', 'formset_factory', 'all_valid')


Question: BaseFormSet.isvalid() produces ValidationError when there is no management form Description I was torn between reporting this as a bug or a feature request, but then I thought if I make it a feature request it will most likely break a lot of Django apps and hence I guess that means it's more of a bug... Anyway so the line in question is (in Django version 1.6.1): for i in range(0, self.totalformcount()): the self.totalformcount() executes this line (in Django version 1.6.1): return min(self.managementform.cleaneddata[TOTALFORMCOUNT], self.absolutemax) ..which then raises this exception (in Django version 1.6.1): raise ValidationError( ('ManagementForm data is missing or has been tampered with'), code='missingmanagementform', ) That stack trace occurs if/when a user submits a formset after stripping out the management form hidden fields. I have been using Django for a few years now and have never come across an exception being raised by a form/formset isvalid() call before. So my point is that I believe this exception should never be allowed to leave the BaseFormSet.isvalid() call, because it is an oddball behaviour compared to the rest of the isvalid() implementations. I.e. I believe there should be a check in BaseFormSet.isvalid() which checks for the presence of a valid management form (first) and returns False if it is not present, as opposed to raising an exception. Yes I could wrap the isvalid() call in a try/catch, but I believe this is an unnecessary hack caused by a bad design deviation of the implementation of the BaseFormSet.is_valid() method. I didn't bother creating a patch and test cases, because I have a feeling this will get rejected or something like that, but I just thought I should bring this up, as I can't find mention of it anywhere and it seems important to me.

self.base_fields[MAX_NUM_FORM_COUNT] = IntegerField(required=False, widget=HiddenInput)         super().__init__(*args, **kwargs) <ins>    def clean(self):        cleaned_data = super().clean()        # When the management form is invalid, we don't know how many forms        # were submitted.        cleaned_data.setdefault(TOTAL_FORM_COUNT, 0)        cleaned_data.setdefault(INITIAL_FORM_COUNT, 0)        return cleaned_data</ins>  @html_safe class BaseFormSet:


Question: BaseFormSet.isvalid() produces ValidationError when there is no management form Description I was torn between reporting this as a bug or a feature request, but then I thought if I make it a feature request it will most likely break a lot of Django apps and hence I guess that means it's more of a bug... Anyway so the line in question is (in Django version 1.6.1): for i in range(0, self.totalformcount()): the self.totalformcount() executes this line (in Django version 1.6.1): return min(self.managementform.cleaneddata[TOTALFORMCOUNT], self.absolutemax) ..which then raises this exception (in Django version 1.6.1): raise ValidationError( ('ManagementForm data is missing or has been tampered with'), code='missingmanagementform', ) That stack trace occurs if/when a user submits a formset after stripping out the management form hidden fields. I have been using Django for a few years now and have never come across an exception being raised by a form/formset isvalid() call before. So my point is that I believe this exception should never be allowed to leave the BaseFormSet.isvalid() call, because it is an oddball behaviour compared to the rest of the isvalid() implementations. I.e. I believe there should be a check in BaseFormSet.isvalid() which checks for the presence of a valid management form (first) and returns False if it is not present, as opposed to raising an exception. Yes I could wrap the isvalid() call in a try/catch, but I believe this is an unnecessary hack caused by a bad design deviation of the implementation of the BaseFormSet.is_valid() method. I didn't bother creating a patch and test cases, because I have a feeling this will get rejected or something like that, but I just thought I should bring this up, as I can't find mention of it anywhere and it seems important to me.

A collection of instances of the same Form class.     """     ordering_widget = NumberInput    default_error_messages = {        'missing_management_form': _(            'ManagementForm data is missing or has been tampered with. Missing fields: '            '%(field_names)s. You may need to file a bug report if the issue persists.'        ),    }      def __init__(self, data=None, files=None, auto_id='id_%s', prefix=None,                 initial=None, error_class=ErrorList, form_kwargs=None):                 initial=None, error_class=ErrorList, form_kwargs=None,                 error_messages=None):         self.is_bound = data is not None or files is not None         self.prefix = prefix or self.get_default_prefix()         self.auto_id = auto_id


Question: BaseFormSet.isvalid() produces ValidationError when there is no management form Description I was torn between reporting this as a bug or a feature request, but then I thought if I make it a feature request it will most likely break a lot of Django apps and hence I guess that means it's more of a bug... Anyway so the line in question is (in Django version 1.6.1): for i in range(0, self.totalformcount()): the self.totalformcount() executes this line (in Django version 1.6.1): return min(self.managementform.cleaneddata[TOTALFORMCOUNT], self.absolutemax) ..which then raises this exception (in Django version 1.6.1): raise ValidationError( ('ManagementForm data is missing or has been tampered with'), code='missingmanagementform', ) That stack trace occurs if/when a user submits a formset after stripping out the management form hidden fields. I have been using Django for a few years now and have never come across an exception being raised by a form/formset isvalid() call before. So my point is that I believe this exception should never be allowed to leave the BaseFormSet.isvalid() call, because it is an oddball behaviour compared to the rest of the isvalid() implementations. I.e. I believe there should be a check in BaseFormSet.isvalid() which checks for the presence of a valid management form (first) and returns False if it is not present, as opposed to raising an exception. Yes I could wrap the isvalid() call in a try/catch, but I believe this is an unnecessary hack caused by a bad design deviation of the implementation of the BaseFormSet.is_valid() method. I didn't bother creating a patch and test cases, because I have a feeling this will get rejected or something like that, but I just thought I should bring this up, as I can't find mention of it anywhere and it seems important to me.

self._errors = None         self._non_form_errors = None <ins>        messages = {}        for cls in reversed(type(self).__mro__):            messages.update(getattr(cls, 'default_error_messages', {}))        if error_messages is not None:            messages.update(error_messages)        self.error_messages = messages</ins>     def __str__(self):         return self.as_table()


Question: BaseFormSet.isvalid() produces ValidationError when there is no management form Description I was torn between reporting this as a bug or a feature request, but then I thought if I make it a feature request it will most likely break a lot of Django apps and hence I guess that means it's more of a bug... Anyway so the line in question is (in Django version 1.6.1): for i in range(0, self.totalformcount()): the self.totalformcount() executes this line (in Django version 1.6.1): return min(self.managementform.cleaneddata[TOTALFORMCOUNT], self.absolutemax) ..which then raises this exception (in Django version 1.6.1): raise ValidationError( ('ManagementForm data is missing or has been tampered with'), code='missingmanagementform', ) That stack trace occurs if/when a user submits a formset after stripping out the management form hidden fields. I have been using Django for a few years now and have never come across an exception being raised by a form/formset isvalid() call before. So my point is that I believe this exception should never be allowed to leave the BaseFormSet.isvalid() call, because it is an oddball behaviour compared to the rest of the isvalid() implementations. I.e. I believe there should be a check in BaseFormSet.isvalid() which checks for the presence of a valid management form (first) and returns False if it is not present, as opposed to raising an exception. Yes I could wrap the isvalid() call in a try/catch, but I believe this is an unnecessary hack caused by a bad design deviation of the implementation of the BaseFormSet.is_valid() method. I didn't bother creating a patch and test cases, because I have a feeling this will get rejected or something like that, but I just thought I should bring this up, as I can't find mention of it anywhere and it seems important to me.

"""Return the ManagementForm instance for this FormSet."""         if self.is_bound:             form = ManagementForm(self.data, auto_id=self.auto_id, prefix=self.prefix)            if not form.is_valid():                raise ValidationError(                    _(                        'ManagementForm data is missing or has been tampered '                        'with. Missing fields: %(field_names)s'                    ) % {                        'field_names': ', '.join(                            form.add_prefix(field_name) for field_name in form.errors                        ),                    },                    code='missing_management_form',                )            form.full_clean()         else:             form = ManagementForm(auto_id=self.auto_id, prefix=self.prefix, initial={                 TOTAL_FORM_COUNT: self.total_form_count(),


Question: BaseFormSet.isvalid() produces ValidationError when there is no management form Description I was torn between reporting this as a bug or a feature request, but then I thought if I make it a feature request it will most likely break a lot of Django apps and hence I guess that means it's more of a bug... Anyway so the line in question is (in Django version 1.6.1): for i in range(0, self.totalformcount()): the self.totalformcount() executes this line (in Django version 1.6.1): return min(self.managementform.cleaneddata[TOTALFORMCOUNT], self.absolutemax) ..which then raises this exception (in Django version 1.6.1): raise ValidationError( ('ManagementForm data is missing or has been tampered with'), code='missingmanagementform', ) That stack trace occurs if/when a user submits a formset after stripping out the management form hidden fields. I have been using Django for a few years now and have never come across an exception being raised by a form/formset isvalid() call before. So my point is that I believe this exception should never be allowed to leave the BaseFormSet.isvalid() call, because it is an oddball behaviour compared to the rest of the isvalid() implementations. I.e. I believe there should be a check in BaseFormSet.isvalid() which checks for the presence of a valid management form (first) and returns False if it is not present, as opposed to raising an exception. Yes I could wrap the isvalid() call in a try/catch, but I believe this is an unnecessary hack caused by a bad design deviation of the implementation of the BaseFormSet.is_valid() method. I didn't bother creating a patch and test cases, because I have a feeling this will get rejected or something like that, but I just thought I should bring this up, as I can't find mention of it anywhere and it seems important to me.

if not self.is_bound:  # Stop further processing.             return<ins>        if not self.management_form.is_valid():            error = ValidationError(                self.error_messages['missing_management_form'],                params={                    'field_names': ', '.join(                        self.management_form.add_prefix(field_name)                        for field_name in self.management_form.errors                    ),                },                code='missing_management_form',            )            self._non_form_errors.append(error)</ins>         for i, form in enumerate(self.forms):             # Empty forms are unchanged forms beyond those with initial data.             if not form.has_changed() and i >= self.initial_form_count():


Question: Add warning in makemessages command if the localecode with flag is not correct Description (last modified by Sanyam Khurana) Hey Calude, What about normalizing the directory name to something that would just work. For example, No matter, if the developer is doing all these: python makemessages -l zhcn python makemessages -l zhCN python makemessages -l ZHCN python makemessages -l ZH-CN etc. we, just normalize the directory name to zhCN and it would work. I'm still about to read the code of makemessages command and probably if there are any more checks than just this, then we'll have to figure out another way all together.

# Build po files for each selected locale             for locale in locales:<ins>                if '-' in locale:                    self.stdout.write(                        'invalid locale %s, did you mean %s?' % (                            locale,                            locale.replace('-', '_'),                        ),                    )                    continue</ins>                 if self.verbosity > 0:                     self.stdout.write('processing locale %s' % locale)                 for potfile in potfiles:


Question: Unclear error raised by makemessages when /locale does not exist under app directory Description I got this error message: CommandError: Unable to find a locale path to store translations for file I had to google the error message and got to this:  It would be better if the error message was simple and obvious so you could solve the problem yourself directly instead. I thought the error message would be better if it said 'We are looking for a directory called "locale"" but after that change the command fails silently anyway. Maybe LOCALE_PATHS should also include the directory where is by default? Or some other error message should exist...

if locale_dir is NO_LOCALE_DIR:                 file_path = os.path.normpath(build_files[0].path)                 raise CommandError(                    'Unable to find a locale path to store translations for '                    'file %s' % file_path                    "Unable to find a locale path to store translations for "                    "file %s. Make sure the 'locale' directory exist in an "                    "app or LOCALE_PATHS setting is set." % file_path                 )             for build_file in build_files:                 msgs = build_file.postprocess_messages(msgs)


Question: Aggregation over subquery annotation GROUP BY produces wrong results Description (last modified by Christian Klus) Starting in django 3.0.7, specifically after patch I noticed some of my more complex queries returning incorrect results. I think I've narrowed it down to a simpler test case: Example query: ().annotate( pubyear=TruncYear('pubdate') ).orderby().values('pubyear').annotate( totalpages=Sum('pages'), toprating=Subquery( ( pubdateyear=OuterRef('pubyear') ).orderby('rating').values('rating')[:1] ) ).values('pubyear', 'totalpages', 'toprating') Generated SQL on 3.0.6: SELECT djangodatetrunc('year', "aggregationregressbook"."pubdate") AS "pubyear", SUM("aggregationregressbook"."pages") AS "totalpages", ( SELECT U0."rating" FROM "aggregationregressbook" U0 WHERE djangodateextract('year', U0."pubdate") = djangodatetrunc('year', "aggregationregressbook"."pubdate") ORDER BY U0."rating" ASC LIMIT 1 ) AS "toprating" FROM "aggregationregressbook" GROUP BY djangodatetrunc('year', "aggregationregressbook"."pubdate"), "toprating" Generated SQL on current master: SELECT djangodatetrunc('year', "aggregationregressbook"."pubdate", NULL, NULL) AS "pubyear", SUM("aggregationregressbook"."pages") AS "totalpages", ( SELECT U0."rating" FROM "aggregationregressbook" U0 WHERE djangodateextract('year', U0."pubdate") = djangodatetrunc('year', "aggregationregressbook"."pubdate", NULL, NULL) ORDER BY U0."rating" ASC LIMIT 1 ) AS "toprating" FROM "aggregationregressbook" GROUP BY djangodatetrunc('year', "aggregationregressbook"."pubdate", NULL, NULL), ( SELECT U0."rating" FROM "aggregationregressbook" U0 WHERE djangodateextract('year', U0."pubdate") = djangodatetrunc('year', "aggregationregressbook"."pubdate", NULL, NULL) ORDER BY U0."rating" ASC LIMIT 1 ), "aggregationregressbook"."pubdate" I see two separate issues here: "aggregationregressbook"."pubdate" is being to the group by clause incorrectly (this issue probably predates the patch mentioned above) Even though the subquery is in the select statement, the alias is not being used and instead the subquery is reevaluated. This nearly doubles the cost of one of my queries that is experiencing this problem. I don't know much about the ORM internals, but here is my naive patch for the second issue: diff --git index .. --- +++ -2220,7 +2220,7 class Query(BaseExpression): # the selected fields anymore. groupby = [] for expr in self.groupby: if isinstance(expr, Ref) and not in fieldnames: if isinstance(expr, Ref) and not in fieldnames + annotationnames: expr = self.annotations[] (expr) self.groupby = tuple(groupby) I'd appreciate anyone with deeper knowlege of the ORM to chime in and let me know if I'm on the right track. Tests are passing locally. The resulting query on master: SELECT djangodatetrunc('year', "aggregationregressbook"."pubdate", NULL, NULL) AS "pubyear", SUM("aggregationregressbook"."pages") AS "totalpages", ( SELECT U0."rating" FROM "aggregationregressbook" U0 WHERE djangodateextract('year', U0."pubdate") = djangodatetrunc('year', "aggregationregressbook"."pubdate", NULL, NULL) ORDER BY U0."rating" ASC LIMIT 1 ) AS "toprating" FROM "aggregationregressbook" GROUP BY djangodatetrunc('year', "aggregationregressbook"."pubdate", NULL, NULL), "top_rating"

field_names.append(f)             self.set_extra_mask(extra_names)             self.set_annotation_mask(annotation_names)            selected = frozenset(field_names + extra_names + annotation_names)         else:             field_names = [f.attname for f in self.model._meta.concrete_fields]            selected = frozenset(field_names)         # Selected annotations must be known before setting the GROUP BY         # clause.         if self.group_by is True:


Question: Aggregation over subquery annotation GROUP BY produces wrong results Description (last modified by Christian Klus) Starting in django 3.0.7, specifically after patch I noticed some of my more complex queries returning incorrect results. I think I've narrowed it down to a simpler test case: Example query: ().annotate( pubyear=TruncYear('pubdate') ).orderby().values('pubyear').annotate( totalpages=Sum('pages'), toprating=Subquery( ( pubdateyear=OuterRef('pubyear') ).orderby('rating').values('rating')[:1] ) ).values('pubyear', 'totalpages', 'toprating') Generated SQL on 3.0.6: SELECT djangodatetrunc('year', "aggregationregressbook"."pubdate") AS "pubyear", SUM("aggregationregressbook"."pages") AS "totalpages", ( SELECT U0."rating" FROM "aggregationregressbook" U0 WHERE djangodateextract('year', U0."pubdate") = djangodatetrunc('year', "aggregationregressbook"."pubdate") ORDER BY U0."rating" ASC LIMIT 1 ) AS "toprating" FROM "aggregationregressbook" GROUP BY djangodatetrunc('year', "aggregationregressbook"."pubdate"), "toprating" Generated SQL on current master: SELECT djangodatetrunc('year', "aggregationregressbook"."pubdate", NULL, NULL) AS "pubyear", SUM("aggregationregressbook"."pages") AS "totalpages", ( SELECT U0."rating" FROM "aggregationregressbook" U0 WHERE djangodateextract('year', U0."pubdate") = djangodatetrunc('year', "aggregationregressbook"."pubdate", NULL, NULL) ORDER BY U0."rating" ASC LIMIT 1 ) AS "toprating" FROM "aggregationregressbook" GROUP BY djangodatetrunc('year', "aggregationregressbook"."pubdate", NULL, NULL), ( SELECT U0."rating" FROM "aggregationregressbook" U0 WHERE djangodateextract('year', U0."pubdate") = djangodatetrunc('year', "aggregationregressbook"."pubdate", NULL, NULL) ORDER BY U0."rating" ASC LIMIT 1 ), "aggregationregressbook"."pubdate" I see two separate issues here: "aggregationregressbook"."pubdate" is being to the group by clause incorrectly (this issue probably predates the patch mentioned above) Even though the subquery is in the select statement, the alias is not being used and instead the subquery is reevaluated. This nearly doubles the cost of one of my queries that is experiencing this problem. I don't know much about the ORM internals, but here is my naive patch for the second issue: diff --git index .. --- +++ -2220,7 +2220,7 class Query(BaseExpression): # the selected fields anymore. groupby = [] for expr in self.groupby: if isinstance(expr, Ref) and not in fieldnames: if isinstance(expr, Ref) and not in fieldnames + annotationnames: expr = self.annotations[] (expr) self.groupby = tuple(groupby) I'd appreciate anyone with deeper knowlege of the ORM to chime in and let me know if I'm on the right track. Tests are passing locally. The resulting query on master: SELECT djangodatetrunc('year', "aggregationregressbook"."pubdate", NULL, NULL) AS "pubyear", SUM("aggregationregressbook"."pages") AS "totalpages", ( SELECT U0."rating" FROM "aggregationregressbook" U0 WHERE djangodateextract('year', U0."pubdate") = djangodatetrunc('year', "aggregationregressbook"."pubdate", NULL, NULL) ORDER BY U0."rating" ASC LIMIT 1 ) AS "toprating" FROM "aggregationregressbook" GROUP BY djangodatetrunc('year', "aggregationregressbook"."pubdate", NULL, NULL), "top_rating"

# the selected fields anymore.             group_by = []             for expr in self.group_by:                if isinstance(expr, Ref) and expr.refs not in field_names:                if isinstance(expr, Ref) and expr.refs not in selected:                     expr = self.annotations[expr.refs]                 group_by.append(expr)             self.group_by = tuple(group_by)


Question: Support for list arguments inside exclusive required groups Description (last modified by Mark Gajdosik) Are there any plans to add support for the following? from import BaseCommand class Command(BaseCommand): def addarguments(self, parser): group = parser.addmutuallyexclusivegroup(required=True) group.addargument('--foo', nargs='+', type=int) def handle(self, args, *options): pass When calling the above using callcommand: callcommand('callcommandtest', foo=[1, 2, 3]) callcommand('callcommandtest', '--foo=1', '--foo=2', '--foo=3') I can't think of any workarounds other than setting type=str (somehow, that works fine) and coercing manually. Thank you!

for s_opt in parser._actions if s_opt.option_strings     }     arg_options = {opt_mapping.get(key, key): value for key, value in options.items()}    parse_args = [str(a) for a in args]    parse_args = []    for arg in args:        if isinstance(arg, (list, tuple)):            parse_args += map(str, arg)        else:            parse_args.append(str(arg))      def get_actions(parser):         # Parser actions and actions from sub-parser choices.


Question: Support for list arguments inside exclusive required groups Description (last modified by Mark Gajdosik) Are there any plans to add support for the following? from import BaseCommand class Command(BaseCommand): def addarguments(self, parser): group = parser.addmutuallyexclusivegroup(required=True) group.addargument('--foo', nargs='+', type=int) def handle(self, args, *options): pass When calling the above using callcommand: callcommand('callcommandtest', foo=[1, 2, 3]) callcommand('callcommandtest', '--foo=1', '--foo=2', '--foo=3') I can't think of any workarounds other than setting type=str (somehow, that works fine) and coercing manually. Thank you!

}     # Any required arguments which are passed in via **options must be passed     # to parse_args().    parse_args += [        min(opt.option_strings)        if isinstance(opt, (_AppendConstAction, _CountAction, _StoreConstAction))        else '{}={}'.format(min(opt.option_strings), arg_options[opt.dest])        for opt in parser_actions if (    for opt in parser_actions:        if (             opt.dest in options and             (opt.required or opt in mutually_exclusive_required_options)        )    ]        ):            parse_args.append(min(opt.option_strings))            if isinstance(opt, (_AppendConstAction, _CountAction, _StoreConstAction)):                continue            value = arg_options[opt.dest]            if isinstance(value, (list, tuple)):                parse_args += map(str, value)            else:                parse_args.append(str(value))     defaults = parser.parse_args(args=parse_args)     defaults = dict(defaults._get_kwargs(), **arg_options)     # Raise an error if any unknown options were passed.


Question: ManagementUtility instantiates CommandParser without passing already-computed prog argument Description ManagementUtility goes to the trouble to parse the program name from the argv it's passed rather than from def init(self, argv=None): = argv or [:] self.progname = ([0]) if self.progname == '': self.progname = 'python -m django' But then when it needs to parse --pythonpath and --settings, it uses the program name from parser = CommandParser(usage='%(prog)s subcommand [options] [args]', addhelp=False, allowabbrev=False) Above "%(prog)s" refers to [0]. Instead, it should refer to self.progname. This can fixed as follows: parser = CommandParser( prog=self.progname, usage='%(prog)s subcommand [options] [args]', addhelp=False, allowabbrev=False) I'm aware that executefromcommandline is a private API, but it'd be really convenient for me if it worked properly in my weird embedded environment where [0] is incorrectly None. If passing my own argv to executefromcommand_line avoided all the ensuing exceptions, I wouldn't have to modify [0] globally as I'm doing in the meantime.

# Preprocess options to extract --settings and --pythonpath.         # These options could affect the commands that are available, so they         # must be processed early.        parser = CommandParser(usage='%(prog)s subcommand [options] [args]', add_help=False, allow_abbrev=False)        parser = CommandParser(            prog=self.prog_name,            usage='%(prog)s subcommand [options] [args]',            add_help=False,            allow_abbrev=False,        )         parser.add_argument('--settings')         parser.add_argument('--pythonpath')         parser.add_argument('args', nargs='*')  # catch-all


Question: shell command crashes when passing (with -c) the python code with functions. Description The examples below use Python 3.7 and Django 2.2.16, but I checked that the code is the same on master and works the same in Python 3.8. Here's how python -c works: $ python -c <<EOF " import django def f(): print(django.version) f()" EOF 16 Here's how python -m django shell -c works (paths shortened for clarify): $ python -m django shell -c <<EOF " import django def f(): print(django.version) f()" EOF Traceback (most recent call last): File "{sys.baseprefix}", line 193, in runmoduleasmain "main", modspec) File "{sys.baseprefix}", line 85, in runcode exec(code, runglobals) File "{}/lib/python3.7/site-", line 9, in <modulemanagement.executefromcommandline() File "{}/lib/python3.7/site-", line 381, in executefromcommandline utility.execute() File "{}/lib/python3.7/site-", line 375, in execute self.fetchcommand(subcommand).runfromargv() File "{}/lib/python3.7/site-", line 323, in runfromargv self.execute(args, cmdoptions) File "{}/lib/python3.7/site-", line 364, in execute output = (args, options) File "{}/lib/python3.7/site-", line 86, in handle exec(options['command']) File "<string", line 5, in <moduleFile "<string", line 4, in f NameError: name 'django' is not defined The problem is in the usage of exec: def handle(self, **options): # Execute the command and exit. if options['command']: exec(options['command']) return # Execute stdin if it has anything to read and exit. # Not supported on Windows due to () limitations. if sys.platform != 'win32' and not () and ([], [], [], 0)[0]: exec(()) return exec should be passed a dictionary containing a minimal set of globals. This can be done by just passing a new, empty dictionary as the second argument of exec.

def handle(self, **options):         # Execute the command and exit.         if options['command']:            exec(options['command'])            exec(options['command'], globals())             return          # Execute stdin if it has anything to read and exit.         # Not supported on Windows due to select.select() limitations.         if sys.platform != 'win32' and not sys.stdin.isatty() and select.select([sys.stdin], [], [], 0)[0]:            exec(sys.stdin.read())            exec(sys.stdin.read(), globals())             return          available_shells = [options['interface']] if options['interface'] else self.shells


Question: Dabase creation backend should use basemanager to serialize database Description class Service(Model): objects = CustomManagerThatFillterOutSomeRecords() class CustomManagerThatFillterOutSomeRecords(Manager): def getqueryset(self): return super().getqueryset().exclude(pk=1) class TestService(TransactionTestCase): serializedrollback = True def testsomething(self): pass Assume we have a migration that creates few records of Service. from import callcommand from import migrations def loadfixtures(, *_): callcommand('loaddata', '') class Migration(migrations.Migration): dependencies = [] operations = [ migrations.RunPython( loadfixtures, , ) ] Then TestService will fail as serializedbtostring by default use defaultmanager that is CustomManagerThatFillterOutSomeRecords. Here is proposed fix:

model._meta.can_migrate(self.connection) and                             router.allow_migrate_model(self.connection.alias, model)                         ):                            queryset = model._default_manager.using(                            queryset = model._base_manager.using(                                 self.connection.alias,                             ).order_by(model._meta.pk.name)                             yield from queryset.iterator()


Question: Augment () optimizations to .union().exists(). Description (last modified by Simon Charette) The method performs optimization by clearing the select clause, dropping ordering, and limiting the number of results to 1 if possible. A similar optimization can be applied for combined queries when using () as some query planers (e.g. MySQL) are not smart enough to prune changes down into combined queries. For example, given (otherauthors).exists() the currently generated is SELECT 1 FROM (SELECT FROM authors WHERE ... ORDER BY ... UNION SELECT FROM authors WHERE ... ORDER BY ...) LIMIT 1; But some planers won't be smart enough to realize that both * and ORDER BY are not necessary and fetch all matching rows. In order to help them we should generate the following SQL SELECT 1 FROM (SELECT 1 FROM authors WHERE ... LIMIT 1 UNION SELECT 1 FROM authors WHERE ... LIMIT 1) LIMIT 1; This can already be done manually through filteredauthors.orderby().values(Value(1))[:1].union(otherauthors.orderby().values(Value(1))[:1]).exists() but that involves a lot of boilerplate. Note that the optimization is only possible in this form for union and not for intersection and difference since they require both sets to be unaltered.

def external_aliases(self):         return self.query.external_aliases     def as_sql(self, compiler, connection, template=None, **extra_context):    def as_sql(self, compiler, connection, template=None, query=None, **extra_context):         connection.ops.check_expression_support(self)         template_params = {**self.extra, **extra_context}        subquery_sql, sql_params = self.query.as_sql(compiler, connection)        query = query or self.query        subquery_sql, sql_params = query.as_sql(compiler, connection)         template_params['subquery'] = subquery_sql[1:-1]          template = template or template_params.get('template', self.template)


Question: Augment () optimizations to .union().exists(). Description (last modified by Simon Charette) The method performs optimization by clearing the select clause, dropping ordering, and limiting the number of results to 1 if possible. A similar optimization can be applied for combined queries when using () as some query planers (e.g. MySQL) are not smart enough to prune changes down into combined queries. For example, given (otherauthors).exists() the currently generated is SELECT 1 FROM (SELECT FROM authors WHERE ... ORDER BY ... UNION SELECT FROM authors WHERE ... ORDER BY ...) LIMIT 1; But some planers won't be smart enough to realize that both * and ORDER BY are not necessary and fetch all matching rows. In order to help them we should generate the following SQL SELECT 1 FROM (SELECT 1 FROM authors WHERE ... LIMIT 1 UNION SELECT 1 FROM authors WHERE ... LIMIT 1) LIMIT 1; This can already be done manually through filteredauthors.orderby().values(Value(1))[:1].union(otherauthors.orderby().values(Value(1))[:1]).exists() but that involves a lot of boilerplate. Note that the optimization is only possible in this form for union and not for intersection and difference since they require both sets to be unaltered.

def __init__(self, queryset, negated=False, **kwargs):         self.negated = negated         super().__init__(queryset, **kwargs)<del>        self.query = self.query.exists()</del>      def __invert__(self):         clone = self.copy()


Question: Augment () optimizations to .union().exists(). Description (last modified by Simon Charette) The method performs optimization by clearing the select clause, dropping ordering, and limiting the number of results to 1 if possible. A similar optimization can be applied for combined queries when using () as some query planers (e.g. MySQL) are not smart enough to prune changes down into combined queries. For example, given (otherauthors).exists() the currently generated is SELECT 1 FROM (SELECT FROM authors WHERE ... ORDER BY ... UNION SELECT FROM authors WHERE ... ORDER BY ...) LIMIT 1; But some planers won't be smart enough to realize that both * and ORDER BY are not necessary and fetch all matching rows. In order to help them we should generate the following SQL SELECT 1 FROM (SELECT 1 FROM authors WHERE ... LIMIT 1 UNION SELECT 1 FROM authors WHERE ... LIMIT 1) LIMIT 1; This can already be done manually through filteredauthors.orderby().values(Value(1))[:1].union(otherauthors.orderby().values(Value(1))[:1]).exists() but that involves a lot of boilerplate. Note that the optimization is only possible in this form for union and not for intersection and difference since they require both sets to be unaltered.

return clone      def as_sql(self, compiler, connection, template=None, **extra_context):        sql, params = super().as_sql(compiler, connection, template, **extra_context)        query = self.query.exists(using=connection.alias)        sql, params = super().as_sql(            compiler,            connection,            template=template,            query=query,            **extra_context,        )         if self.negated:             sql = 'NOT {}'.format(sql)         return sql, paramsdiff --git a/django/db/models/sql/query.py b/django/db/models/sql/query.py-- a/django/db/models/sql/query.py++ b/django/db/models/sql/query.py


Question: Augment () optimizations to .union().exists(). Description (last modified by Simon Charette) The method performs optimization by clearing the select clause, dropping ordering, and limiting the number of results to 1 if possible. A similar optimization can be applied for combined queries when using () as some query planers (e.g. MySQL) are not smart enough to prune changes down into combined queries. For example, given (otherauthors).exists() the currently generated is SELECT 1 FROM (SELECT FROM authors WHERE ... ORDER BY ... UNION SELECT FROM authors WHERE ... ORDER BY ...) LIMIT 1; But some planers won't be smart enough to realize that both * and ORDER BY are not necessary and fetch all matching rows. In order to help them we should generate the following SQL SELECT 1 FROM (SELECT 1 FROM authors WHERE ... LIMIT 1 UNION SELECT 1 FROM authors WHERE ... LIMIT 1) LIMIT 1; This can already be done manually through filteredauthors.orderby().values(Value(1))[:1].union(otherauthors.orderby().values(Value(1))[:1]).exists() but that involves a lot of boilerplate. Note that the optimization is only possible in this form for union and not for intersection and difference since they require both sets to be unaltered.

def has_filters(self):         return self.where     def exists(self):    def exists(self, using, limit=True):         q = self.clone()         if not q.distinct:             if q.group_by is True:


Question: Augment () optimizations to .union().exists(). Description (last modified by Simon Charette) The method performs optimization by clearing the select clause, dropping ordering, and limiting the number of results to 1 if possible. A similar optimization can be applied for combined queries when using () as some query planers (e.g. MySQL) are not smart enough to prune changes down into combined queries. For example, given (otherauthors).exists() the currently generated is SELECT 1 FROM (SELECT FROM authors WHERE ... ORDER BY ... UNION SELECT FROM authors WHERE ... ORDER BY ...) LIMIT 1; But some planers won't be smart enough to realize that both * and ORDER BY are not necessary and fetch all matching rows. In order to help them we should generate the following SQL SELECT 1 FROM (SELECT 1 FROM authors WHERE ... LIMIT 1 UNION SELECT 1 FROM authors WHERE ... LIMIT 1) LIMIT 1; This can already be done manually through filteredauthors.orderby().values(Value(1))[:1].union(otherauthors.orderby().values(Value(1))[:1]).exists() but that involves a lot of boilerplate. Note that the optimization is only possible in this form for union and not for intersection and difference since they require both sets to be unaltered.

# SELECT clause which is about to be cleared.                 q.set_group_by(allow_aliases=False)             q.clear_select_clause()        if q.combined_queries and q.combinator == 'union':            limit_combined = connections[using].features.supports_slicing_ordering_in_compound            q.combined_queries = tuple(                combined_query.exists(using, limit=limit_combined)                for combined_query in q.combined_queries            )         q.clear_ordering(True)        q.set_limits(high=1)        if limit:            q.set_limits(high=1)         q.add_extra({'a': 1}, None, None, None, None, None)         q.set_extra_mask(['a'])         return q      def has_results(self, using):        q = self.exists()        q = self.exists(using)         compiler = q.get_compiler(using=using)         return compiler.has_results()


Question: dateformat.y() doesn't support years < 1000. Description (last modified by Sam) When using the the dateformat of django with a date before 999 (or 99 and 9 for similar matters) and the format character "y" no leading zero will be printed. This is not consistent with the way the python datetime module and PHP handle that character "y" in format strings: django (version 3.1): import datetime from import dateformat (datetime.datetime(123, 4, 5, 6, 7), "y") '3' python (version 3.8): import datetime datetime.datetime(123, 4, 5, 6, 7).strftime("%y") '23' php (version 7.4): echo date("y", strtotime("0123-04-05 06:07:00")) 23 I have a pull-request ready for this:

return self.data.isocalendar()[1]      def y(self):        "Year, 2 digits; e.g. '99'"        return str(self.data.year)[2:]        """Year, 2 digits with leading zeros; e.g. '99'."""        return '%02d' % (self.data.year % 100)      def Y(self):         "Year, 4 digits; e.g. '1999'"


Question: Allow cache.getorset() to cache a None result Description (last modified by Phill Tornroth) getorset docstring says "If the key does not exist, add the key and set it to the default value." -- but that's not quite what it does. It will perform a set if the key doesn't exist, or if the cached value is None. I think in order to be doing what it says on the tin it'd need to be: if self.has_key(key, version=version): return (key, version=version) else: if callable(default): default = default() (key, default, timeout=timeout, version=version) # Fetch the value again to avoid a race condition if another # caller a value between the first get() and the add() # above. return (key, default, version=version) I'd find this useful in cases where None was an expensive result to arrive at. If there's spiritual alignment with the suggestion, I'm happy to prepare and submit a change with tests.

class BaseCache:<ins>    _missing_key = object()</ins>     def __init__(self, params):         timeout = params.get('timeout', params.get('TIMEOUT', 300))         if timeout is not None:


Question: Allow cache.getorset() to cache a None result Description (last modified by Phill Tornroth) getorset docstring says "If the key does not exist, add the key and set it to the default value." -- but that's not quite what it does. It will perform a set if the key doesn't exist, or if the cached value is None. I think in order to be doing what it says on the tin it'd need to be: if self.has_key(key, version=version): return (key, version=version) else: if callable(default): default = default() (key, default, timeout=timeout, version=version) # Fetch the value again to avoid a race condition if another # caller a value between the first get() and the add() # above. return (key, default, version=version) I'd find this useful in cases where None was an expensive result to arrive at. If there's spiritual alignment with the suggestion, I'm happy to prepare and submit a change with tests.

"""         d = {}         for k in keys:            val = self.get(k, version=version)            if val is not None:            val = self.get(k, self._missing_key, version=version)            if val is not self._missing_key:                 d[k] = val         return d


Question: Allow cache.getorset() to cache a None result Description (last modified by Phill Tornroth) getorset docstring says "If the key does not exist, add the key and set it to the default value." -- but that's not quite what it does. It will perform a set if the key doesn't exist, or if the cached value is None. I think in order to be doing what it says on the tin it'd need to be: if self.has_key(key, version=version): return (key, version=version) else: if callable(default): default = default() (key, default, timeout=timeout, version=version) # Fetch the value again to avoid a race condition if another # caller a value between the first get() and the add() # above. return (key, default, version=version) I'd find this useful in cases where None was an expensive result to arrive at. If there's spiritual alignment with the suggestion, I'm happy to prepare and submit a change with tests.

Return the value of the key stored or retrieved.         """        val = self.get(key, version=version)        if val is None:        val = self.get(key, self._missing_key, version=version)        if val is self._missing_key:             if callable(default):                 default = default()            if default is not None:                self.add(key, default, timeout=timeout, version=version)                # Fetch the value again to avoid a race condition if another                # caller added a value between the first get() and the add()                # above.                return self.get(key, default, version=version)            self.add(key, default, timeout=timeout, version=version)            # Fetch the value again to avoid a race condition if another caller            # added a value between the first get() and the add() above.            return self.get(key, default, version=version)         return val      def has_key(self, key, version=None):         """         Return True if the key is in the cache and has not expired.         """        return self.get(key, version=version) is not None        return self.get(key, self._missing_key, version=version) is not self._missing_key      def incr(self, key, delta=1, version=None):         """         Add delta to value in the cache. If the key does not exist, raise a         ValueError exception.         """        value = self.get(key, version=version)        if value is None:        value = self.get(key, self._missing_key, version=version)        if value is self._missing_key:             raise ValueError("Key '%s' not found" % key)         new_value = value + delta         self.set(key, new_value, version=version)


Question: Allow cache.getorset() to cache a None result Description (last modified by Phill Tornroth) getorset docstring says "If the key does not exist, add the key and set it to the default value." -- but that's not quite what it does. It will perform a set if the key doesn't exist, or if the cached value is None. I think in order to be doing what it says on the tin it'd need to be: if self.has_key(key, version=version): return (key, version=version) else: if callable(default): default = default() (key, default, timeout=timeout, version=version) # Fetch the value again to avoid a race condition if another # caller a value between the first get() and the add() # above. return (key, default, version=version) I'd find this useful in cases where None was an expensive result to arrive at. If there's spiritual alignment with the suggestion, I'm happy to prepare and submit a change with tests.

if version is None:             version = self.version         value = self.get(key, version=version)        if value is None:        value = self.get(key, self._missing_key, version=version)        if value is self._missing_key:             raise ValueError("Key '%s' not found" % key)          self.set(key, value, version=version + delta)diff --git a/django/core/cache/backends/memcached.py b/django/core/cache/backends/memcached.py-- a/django/core/cache/backends/memcached.py++ b/django/core/cache/backends/memcached.py


Question: Allow cache.getorset() to cache a None result Description (last modified by Phill Tornroth) getorset docstring says "If the key does not exist, add the key and set it to the default value." -- but that's not quite what it does. It will perform a set if the key doesn't exist, or if the cached value is None. I think in order to be doing what it says on the tin it'd need to be: if self.has_key(key, version=version): return (key, version=version) else: if callable(default): default = default() (key, default, timeout=timeout, version=version) # Fetch the value again to avoid a race condition if another # caller a value between the first get() and the add() # above. return (key, default, version=version) I'd find this useful in cases where None was an expensive result to arrive at. If there's spiritual alignment with the suggestion, I'm happy to prepare and submit a change with tests.

class MemcachedCache(BaseMemcachedCache):     "An implementation of a cache binding using python-memcached"<ins>    # python-memcached doesn't support default values in get().    # https://github.com/linsomniac/python-memcached/issues/159    _missing_key = None</ins>     def __init__(self, server, params):         warnings.warn(             'MemcachedCache is deprecated in favor of PyMemcacheCache and '


Question: Improve error messages from forgetting to call .asview() on a CBV Description (last modified by Angus Holder) We can detect early-on that the user has forgotten to call .asview() on their CBV when passing it into path(). For: urlpatterns = [ path('home', HomeView) ] The error currently happens only when you first load the route (rather than when constructing the routes), and looks like Internal Server Error: / Traceback (most recent call last): File "C:\Users\Angus.virtualenvs\django-WBTbdxDv\lib\site-packages\django\core\handlers\", line 47, in inner response = getresponse(request) File "C:\Users\Angus.virtualenvs\django-WBTbdxDv\lib\site-packages\django\core\handlers\", line 179, in getresponse response = wrappedcallback(request, callbackargs, *callbackkwargs) TypeError: init() takes 1 positional argument but 2 were given Which is especially hard to work out given that the traceback doesn't even include any of the user's own code, and this is an error that's easy for beginners to run into when first using CBVs. My PR changes it to fail early, inside the call to (), with a clear error: URL route 'foo' should pass in 'EmptyCBView.as_view()' instead of 'EmptyCBView' Pull request:

def _path(route, view, kwargs=None, name=None, Pattern=None):<ins>    from django.views import View</ins>     if isinstance(view, (list, tuple)):         # For include(...) processing.         pattern = Pattern(route, is_endpoint=False)


Question: Improve error messages from forgetting to call .asview() on a CBV Description (last modified by Angus Holder) We can detect early-on that the user has forgotten to call .asview() on their CBV when passing it into path(). For: urlpatterns = [ path('home', HomeView) ] The error currently happens only when you first load the route (rather than when constructing the routes), and looks like Internal Server Error: / Traceback (most recent call last): File "C:\Users\Angus.virtualenvs\django-WBTbdxDv\lib\site-packages\django\core\handlers\", line 47, in inner response = getresponse(request) File "C:\Users\Angus.virtualenvs\django-WBTbdxDv\lib\site-packages\django\core\handlers\", line 179, in getresponse response = wrappedcallback(request, callbackargs, *callbackkwargs) TypeError: init() takes 1 positional argument but 2 were given Which is especially hard to work out given that the traceback doesn't even include any of the user's own code, and this is an error that's easy for beginners to run into when first using CBVs. My PR changes it to fail early, inside the call to (), with a clear error: URL route 'foo' should pass in 'EmptyCBView.as_view()' instead of 'EmptyCBView' Pull request:

elif callable(view):         pattern = Pattern(route, name=name, is_endpoint=True)         return URLPattern(pattern, view, kwargs, name)    elif isinstance(view, View):        view_cls_name = view.__class__.__name__        raise TypeError(            f'view must be a callable, pass {view_cls_name}.as_view(), not '            f'{view_cls_name}().'        )     else:         raise TypeError('view must be a callable or a list/tuple in the case of include().') diff --git a/django/urls/resolvers.py b/django/urls/resolvers.py-- a/django/urls/resolvers.py++ b/django/urls/resolvers.py


Question: Improve error messages from forgetting to call .asview() on a CBV Description (last modified by Angus Holder) We can detect early-on that the user has forgotten to call .asview() on their CBV when passing it into path(). For: urlpatterns = [ path('home', HomeView) ] The error currently happens only when you first load the route (rather than when constructing the routes), and looks like Internal Server Error: / Traceback (most recent call last): File "C:\Users\Angus.virtualenvs\django-WBTbdxDv\lib\site-packages\django\core\handlers\", line 47, in inner response = getresponse(request) File "C:\Users\Angus.virtualenvs\django-WBTbdxDv\lib\site-packages\django\core\handlers\", line 179, in getresponse response = wrappedcallback(request, callbackargs, *callbackkwargs) TypeError: init() takes 1 positional argument but 2 were given Which is especially hard to work out given that the traceback doesn't even include any of the user's own code, and this is an error that's easy for beginners to run into when first using CBVs. My PR changes it to fail early, inside the call to (), with a clear error: URL route 'foo' should pass in 'EmptyCBView.as_view()' instead of 'EmptyCBView' Pull request:

def check(self):         warnings = self._check_pattern_name()         warnings.extend(self.pattern.check())<ins>        warnings.extend(self._check_callback())</ins>         return warnings      def _check_pattern_name(self):


Question: Improve error messages from forgetting to call .asview() on a CBV Description (last modified by Angus Holder) We can detect early-on that the user has forgotten to call .asview() on their CBV when passing it into path(). For: urlpatterns = [ path('home', HomeView) ] The error currently happens only when you first load the route (rather than when constructing the routes), and looks like Internal Server Error: / Traceback (most recent call last): File "C:\Users\Angus.virtualenvs\django-WBTbdxDv\lib\site-packages\django\core\handlers\", line 47, in inner response = getresponse(request) File "C:\Users\Angus.virtualenvs\django-WBTbdxDv\lib\site-packages\django\core\handlers\", line 179, in getresponse response = wrappedcallback(request, callbackargs, *callbackkwargs) TypeError: init() takes 1 positional argument but 2 were given Which is especially hard to work out given that the traceback doesn't even include any of the user's own code, and this is an error that's easy for beginners to run into when first using CBVs. My PR changes it to fail early, inside the call to (), with a clear error: URL route 'foo' should pass in 'EmptyCBView.as_view()' instead of 'EmptyCBView' Pull request:

else:             return [] <ins>    def _check_callback(self):        from django.views import View        view = self.callback        if inspect.isclass(view) and issubclass(view, View):            return [Error(                'Your URL pattern %s has an invalid view, pass %s.as_view() '                'instead of %s.' % (                    self.pattern.describe(),                    view.__name__,                    view.__name__,                ),                id='urls.E009',            )]        return []</ins>     def resolve(self, path):         match = self.pattern.match(path)         if match:


Question: Misleading error when loading a database backend fails with ImportError Description For example, in the following case, djangocockroachdb is installed but has an import error itself. Traceback (most recent call last): File "", line 110, in loadbackend return importmodule('' % backendname) File "", line 127, in importmodule return bootstrap.gcdimport(name[level:], package, level) File "<frozen importlib.bootstrap", line 1030, in gcdimport File "<frozen importlib.bootstrap", line 1007, in findandload File "<frozen importlib.bootstrap", line 972, in findandloadunlocked File "<frozen importlib.bootstrap", line 228, in callwithframesremoved File "<frozen importlib.bootstrap", line 1030, in gcdimport File "<frozen importlib.bootstrap", line 1007, in findandload File "<frozen importlib.bootstrap", line 986, in findandloadunlocked File "<frozen importlib.bootstrap", line 680, in loadunlocked File "<frozen importlib.bootstrapexternal", line 790, in execmodule File "<frozen importlib.bootstrap", line 228, in callwithframesremoved File "/home/tim/code/cockroach-", line 2, in <modulefrom .lookups import patchlookups File "/home/tim/code/cockroach-", line 1, in <modulefrom import HasKeyLookup, KeyTransform ModuleNotFoundError: No module named '' The above exception was the direct cause of the following exception: Traceback (most recent call last): File "", line 10, in <moduleexecutefromcommandline() File "", line 381, in executefromcommandline utility.execute() File "", line 357, in execute () File "", line 24, in setup apps.populate(settings.INSTALLEDAPPS) File "", line 114, in populate appconfig.importmodels() File "", line 211, in importmodels self.modelsmodule = importmodule(modelsmodulename) File "", line 127, in importmodule return bootstrap.gcdimport(name[level:], package, level) File "<frozen importlib.bootstrap", line 1030, in gcdimport File "<frozen importlib.bootstrap", line 1007, in findandload File "<frozen importlib.bootstrap", line 986, in findandloadunlocked File "<frozen importlib.bootstrap", line 680, in loadunlocked File "<frozen importlib.bootstrapexternal", line 790, in execmodule File "<frozen importlib.bootstrap", line 228, in callwithframesremoved File "", line 12, in <moduleclass Question(models.Model): File "", line 117, in new newclass.addtoclass('meta', Options(meta, applabel)) File "", line 321, in addtoclass value.contributetoclass(cls, name) File "", line 204, in contributetoclass self.dbtable = truncatename(self.dbtable, ()) File "", line 28, in getattr return getattr(connections[DEFAULTDBALIAS], item) File "", line 201, in getitem backend = loadbackend(db['ENGINE']) File "", line 121, in loadbackend raise ImproperlyConfigured( 'djangocockroachdb' isn't an available database backend. Try using '', where XXX is one of: 'mysql', 'oracle', 'postgresql', 'sqlite3' I'm not sure there's anything that could be done besides rephrasing the error message to say something like this: 'djangocockroachdb' isn't an available database backend or couldn't be imported. Check the above exception. In this case, "Try using..." isn't so helpful either. Perhaps we could only include that if backend_name begins with django..

if backend_name not in ['django.db.backends.%s' % b for b in builtin_backends]:             backend_reprs = map(repr, sorted(builtin_backends))             raise ImproperlyConfigured(                "%r isn't an available database backend.\n"                "Try using 'django.db.backends.XXX', where XXX is one of:\n"                "%r isn't an available database backend or couldn't be "                "imported. Check the above exception. To use one of the "                "built-in backends, use 'django.db.backends.XXX', where XXX "                "is one of:\n"                 "    %s" % (backend_name, ", ".join(backend_reprs))             ) from e_user         else:


Question: Aggregating when grouping on an ExpressionWrapper omits the expression from the group by Description I ran into this with Postgres on Django 3.1.3, I'm not sure what other versions it exists on. print( Fred.objects.annotate( bobidisnull=ExpressionWrapper( Q(bobid=None), outputfield=BooleanField() ) ).values( "bobidisnull" ).annotate( idcount=Count("id", distinct=True) ).values( "bobidisnull", "idcount" ).query ) SELECT "mainfred"."bobid" IS NULL AS "bobidisnull", COUNT(DISTINCT "mainfred"."id") AS "idcount" FROM "mainfred" GROUP BY "mainfred"."bobid" On the last line there the group by has dropped the "IS NULL"

return [self.expression]      def get_group_by_cols(self, alias=None):        expression = self.expression.copy()        expression.output_field = self.output_field        return expression.get_group_by_cols(alias=alias)        if isinstance(self.expression, Expression):            expression = self.expression.copy()            expression.output_field = self.output_field            return expression.get_group_by_cols(alias=alias)        # For non-expressions e.g. an SQL WHERE clause, the entire        # `expression` must be included in the GROUP BY clause.        return super().get_group_by_cols()      def as_sql(self, compiler, connection):         return compiler.compile(self.expression)


Question: Error with values with JSON path lookup in Sqlite when value is numeric. Description (last modified by Gordon Wrigley) This Django 3.1.3, I only see this with Sqlite, it works fine with MySQL and Postgres. When I do a path lookup with values like ("myjsonfieldposition") if there is an integer, float or bool in "position" then I get a JSON decode error. Strings, nones, dicts and lists all work, fetching the top level dict works and filtering on the path lookup works. TypeError: the JSON object must be str, bytes or bytearray, not float .....venv\databrowser\lib\site-packages\django\db\models\ in iter self.fetchall() .....venv\databrowser\lib\site-packages\django\db\models\ in fetchall self.resultcache = list(self.iterableclass(self)) .....venv\databrowser\lib\site-packages\django\db\models\ in iter for row in compiler.resultsiter(chunkedfetch=self.chunkedfetch, chunksize=self.chunksize): .....venv\databrowser\lib\site-packages\django\db\models\sql\ in applyconverters value = converter(value, expression, connection) .....venv\databrowser\lib\site-packages\django\db\models\fields\ in fromdb_value return (value, cls=self.decoder)

def from_db_value(self, value, expression, connection):         if value is None:             return value<ins>        # Some backends (SQLite at least) extract non-string values in their        # SQL datatypes.        if isinstance(expression, KeyTransform) and not isinstance(value, str):            return value</ins>         try:             return json.loads(value, cls=self.decoder)         except json.JSONDecodeError:


Question: django-admin runserver and getchildarguments() crashes on Windows and Python < 3.8. Description "django-admin runserver" fails with the following error: Traceback (most recent call last): File "c:\users\someone\appdata\local\programs\python\python37\lib\", line 193, in runmoduleasmain "main", modspec) File "c:\users\someone\appdata\local\programs\python\python37\lib\", line 85, in runcode exec(code, runglobals) File "C:\Users\someone\AppData\Local\Programs\Python\Python37\Scripts\django-\", line 7, in <moduleFile "c:\users\someone\appdata\local\programs\python\python37\lib\site-packages\django\core\management\", line 401, in executefromcommandline utility.execute() File "c:\users\someone\appdata\local\programs\python\python37\lib\site-packages\django\core\management\", line 395, in execute self.fetchcommand(subcommand).runfromargv() File "c:\users\someone\appdata\local\programs\python\python37\lib\site-packages\django\core\management\", line 330, in runfromargv self.execute(args, cmdoptions) File "c:\users\someone\appdata\local\programs\python\python37\lib\site-packages\django\core\management\commands\", line 61, in execute super().execute(args, options) File "c:\users\someone\appdata\local\programs\python\python37\lib\site-packages\django\core\management\", line 371, in execute output = (args, options) File "c:\users\someone\appdata\local\programs\python\python37\lib\site-packages\django\core\management\commands\", line 96, in handle (options) File "c:\users\someone\appdata\local\programs\python\python37\lib\site-packages\django\core\management\commands\", line 103, in run autoreload.runwithreloader(self.innerrun, options) File "c:\users\someone\appdata\local\programs\python\python37\lib\site-packages\django\utils\", line 616, in runwithreloader exitcode = restartwithreloader() File "c:\users\someone\appdata\local\programs\python\python37\lib\site-packages\django\utils\", line 244, in restartwithreloader p = (args, env=newenviron, closefds=False) File "c:\users\someone\appdata\local\programs\python\python37\lib\", line 488, in run with Popen(popenargs, kwargs) as process: File "c:\users\someone\appdata\local\programs\python\python37\lib\", line 800, in init restoresignals, startnewsession) File "c:\users\someone\appdata\local\programs\python\python37\lib\", line 1148, in executechild args = list2cmdline(args) File "c:\users\someone\appdata\local\programs\python\python37\lib\", line 555, in list2cmdline needquote = (" " in arg) or ("\t" in arg) or not arg TypeError: argument of type 'WindowsPath' is not iterable Environment: Windows 10 Python: 3.7.6 Django: 3.1.3 Steps to reproduce: django-admin startproject mysite set PYTHONPATH=<thelocaldirdjango-admin runserver --settings=mysite.settings Apparently returns WindowsPath("C:\Users\someone\AppData\Local\Programs\Python\Python37\Scripts\django-") as the first argument, and subprocess.Popen expects a string and is not able to use Path. The following monkey-patch fixes the error: def getchildargumentsoverride(): args = autoreload.getchildarguments() for i, arg in enumerate(args): if isinstance(arg, Path): args[i] = str(arg) return args autoreload.getchildarguments = autoreload.getchildarguments autoreload.getchildarguments = getchildargumentsoverride

exe_entrypoint = py_script.with_suffix('.exe')         if exe_entrypoint.exists():             # Should be executed directly, ignoring sys.executable.            return [exe_entrypoint, *sys.argv[1:]]            # TODO: Remove str() when dropping support for PY37.            # args parameter accepts path-like on Windows from Python 3.8.            return [str(exe_entrypoint), *sys.argv[1:]]         script_entrypoint = py_script.with_name('%s-script.py' % py_script.name)         if script_entrypoint.exists():             # Should be executed as usual.            return [*args, script_entrypoint, *sys.argv[1:]]            # TODO: Remove str() when dropping support for PY37.            # args parameter accepts path-like on Windows from Python 3.8.            return [*args, str(script_entrypoint), *sys.argv[1:]]         raise RuntimeError('Script %s does not exist.' % py_script)     else:         args += sys.argv


Question: Add a 'durable' flag to () Description As discussed on Twitter following David Seddon's blog post. () guarantees the wrapped database operations is atomic - at the end of its wrapped block, operations within that block will all be applied, or all rolled back. In some situations it's also useful to guarantee that the wrapped operations are durable - at the end of the wrapped block, all operations have definitely been committed. atomic() doesn't guarantee this at the moment since its use may be wrapped by another atomic() higher in the stack, delaying the commit. David Seddon's technique to guarantee durability is to wrap atomic() to first check if an atomic() is already active for the current connection (in other words, connection.get_autocommit() returns False), and raise an error if so. On Twitter, Ian Foote suggested adding a durable flag to atomic() to add this behaviour to Django, to which Aymeric Augustin said it sounds like a good idea.

Since database connections are thread-local, this is thread-safe.     An atomic block can be tagged as durable. In this case, raise a    RuntimeError if it's nested within another atomic block. This guarantees    that database changes in a durable block are committed to the database when    the block exists without error.     This is a private API.     """    # This private flag is provided only to disable the durability checks in    # TestCase.    _ensure_durability = True     def __init__(self, using, savepoint):    def __init__(self, using, savepoint, durable):         self.using = using         self.savepoint = savepoint        self.durable = durable      def __enter__(self):         connection = get_connection(self.using)         if self.durable and self._ensure_durability and connection.in_atomic_block:            raise RuntimeError(                'A durable atomic block cannot be nested within another '                'atomic block.'            )         if not connection.in_atomic_block:             # Reset state when entering an outermost atomic block.             connection.commit_on_exit = True


Question: Add a 'durable' flag to () Description As discussed on Twitter following David Seddon's blog post. () guarantees the wrapped database operations is atomic - at the end of its wrapped block, operations within that block will all be applied, or all rolled back. In some situations it's also useful to guarantee that the wrapped operations are durable - at the end of the wrapped block, all operations have definitely been committed. atomic() doesn't guarantee this at the moment since its use may be wrapped by another atomic() higher in the stack, delaying the commit. David Seddon's technique to guarantee durability is to wrap atomic() to first check if an atomic() is already active for the current connection (in other words, connection.get_autocommit() returns False), and raise an error if so. On Twitter, Ian Foote suggested adding a durable flag to atomic() to add this behaviour to Django, to which Aymeric Augustin said it sounds like a good idea.

connection.in_atomic_block = False  def atomic(using=None, savepoint=True):def atomic(using=None, savepoint=True, durable=False):     # Bare decorator: @atomic -- although the first argument is called     # `using`, it's actually the function being decorated.     if callable(using):        return Atomic(DEFAULT_DB_ALIAS, savepoint)(using)        return Atomic(DEFAULT_DB_ALIAS, savepoint, durable)(using)     # Decorator: @atomic(...) or context manager: with atomic(...): ...     else:        return Atomic(using, savepoint)        return Atomic(using, savepoint, durable)   def _non_atomic_requests(view, using):


Question: Use Admin Inline verbosename as default for Inline verbosenameplural Description Django allows specification of a verbosename and a verbosenameplural for Inline classes in admin views. However, verbosenameplural for an Inline is not currently based on a specified verbosename. Instead, it continues to be based on the model name, or an a verbosename specified in the model's Meta class. This was confusing to me initially (I didn't understand why I had to specify both name forms for an Inline if I wanted to overrule the default name), and seems inconsistent with the approach for a model's Meta class (which does automatically base the plural form on a specified verbosename). I propose that verbosenameplural for an Inline class should by default be based on the verbosename for an Inline if that is specified. I have written a patch to implement this, including tests. Would be happy to submit that.

self.opts = self.model._meta         self.has_registered_model = admin_site.is_registered(self.model)         super().__init__()        if self.verbose_name_plural is None:            if self.verbose_name is None:                self.verbose_name_plural = self.model._meta.verbose_name_plural            else:                self.verbose_name_plural = format_lazy('{}s', self.verbose_name)         if self.verbose_name is None:             self.verbose_name = self.model._meta.verbose_name        if self.verbose_name_plural is None:            self.verbose_name_plural = self.model._meta.verbose_name_plural      @property     def media(self):


Question: SQLite3 bad filepath raising as JSON extension error. Description Applying migrations on an installation using spatialite backend (probably sqlite too) where the path to the database file does not exist causes a confusing JSON field error. Take this following DATABASES configuration. Note that the dev.sqlite3 file does not exist. DATABASES = { 'default': { 'ENGINE': '', 'NAME': '/home/user/app/dev.sqlite3', } } When running migrate the following exception is raised SystemCheckError: System check identified some issues: ERRORS: rumble_common.RumbleEvent: (fields.E180) SQLite does not support JSONFields. We assert the JSON extension is present with the following code block, as per import sqlite3 conn = sqlite3.connect(':memory:') cursor = () cursor.execute('SELECT JSON(\'{"a": "b"}\')') The issue is resolved by changing DATABASES['default']['NAME'] to point to the right path. The issue is that the error is misleading, and leads developers down the wrong rabbit hole Hope thats descriptive enough, thanks all! Extra: Django 3.1.3 Debian Buster Docker environment

@cached_property     def supports_json_field(self):        try:            with self.connection.cursor() as cursor, transaction.atomic():                cursor.execute('SELECT JSON(\'{"a": "b"}\')')        except OperationalError:            return False        with self.connection.cursor() as cursor:            try:                with transaction.atomic(self.connection.alias):                    cursor.execute('SELECT JSON(\'{"a": "b"}\')')            except OperationalError:                return False         return True      can_introspect_json_field = property(operator.attrgetter('supports_json_field'))


Question: Add a hook to customize the admin's formsets parameters Description New feature that adds a method on InlineModelAdmin for providing initial data for the inline formset. By default there is no implementation, although one could be implemented to use GET parameters like getchangeforminitial_data, but it wouldn't be trivial due to the list nature of formset initial data.

"admin/object_history.html"         ], context) <ins>    def get_formset_kwargs(self, request, obj, inline, prefix):        formset_params = {            'instance': obj,            'prefix': prefix,            'queryset': inline.get_queryset(request),        }        if request.method == 'POST':            formset_params.update({                'data': request.POST.copy(),                'files': request.FILES,                'save_as_new': '_saveasnew' in request.POST            })        return formset_params</ins>     def _create_formsets(self, request, obj, change):         "Helper function to generate formsets for add/change_view."         formsets = []


Question: Add a hook to customize the admin's formsets parameters Description New feature that adds a method on InlineModelAdmin for providing initial data for the inline formset. By default there is no implementation, although one could be implemented to use GET parameters like getchangeforminitial_data, but it wouldn't be trivial due to the list nature of formset initial data.

prefixes[prefix] = prefixes.get(prefix, 0) + 1             if prefixes[prefix] != 1 or not prefix:                 prefix = "%s-%s" % (prefix, prefixes[prefix])            formset_params = {                'instance': obj,                'prefix': prefix,                'queryset': inline.get_queryset(request),            }            if request.method == 'POST':                formset_params.update({                    'data': request.POST.copy(),                    'files': request.FILES,                    'save_as_new': '_saveasnew' in request.POST                })            formset_params = self.get_formset_kwargs(request, obj, inline, prefix)             formset = FormSet(**formset_params)              def user_deleted_form(request, obj, formset, index):


Question: Set disabled prop on ReadOnlyPasswordHashField Description Currently the defines a cleanpassword method that returns the initial password value to prevent (accidental) changes to the password value. It is also documented that custom forms for the User model need to define this method:  A while ago the forms.Field base class gained the disabled argument to: [disable] a form field using the disabled HTML attribute so that it won’t be editable by users. Even if a user tampers with the field’s value submitted to the server, it will be ignored in favor of the value from the form’s initial data. It seems to me that this property could be set to True be default on the ReadOnlyPasswordHashField used to display the password hash. This way the cleanpassword is no longer necessary and the potential pitfall when using the ReadOnlyPasswordHashField without implementing clean_password is removed.

def __init__(self, *args, **kwargs):         kwargs.setdefault("required", False)        kwargs.setdefault('disabled', True)         super().__init__(*args, **kwargs)     def bound_data(self, data, initial):        # Always return initial because the widget doesn't        # render an input field.        return initial    def has_changed(self, initial, data):        return False  class UsernameField(forms.CharField):     def to_python(self, value):


Question: Set disabled prop on ReadOnlyPasswordHashField Description Currently the defines a cleanpassword method that returns the initial password value to prevent (accidental) changes to the password value. It is also documented that custom forms for the User model need to define this method:  A while ago the forms.Field base class gained the disabled argument to: [disable] a form field using the disabled HTML attribute so that it won’t be editable by users. Even if a user tampers with the field’s value submitted to the server, it will be ignored in favor of the value from the form’s initial data. It seems to me that this property could be set to True be default on the ReadOnlyPasswordHashField used to display the password hash. This way the cleanpassword is no longer necessary and the potential pitfall when using the ReadOnlyPasswordHashField without implementing clean_password is removed.

if user_permissions:             user_permissions.queryset = user_permissions.queryset.select_related('content_type') <del>    def clean_password(self):        # Regardless of what the user provides, return the initial value.        # This is done here, rather than on the field, because the        # field does not have access to the initial value        return self.initial.get('password')</del>  class AuthenticationForm(forms.Form):     """


Question: ConnectionAbortedError should be treated as a broken pipe error in the development server Description People using Windows have been complaining about very long stack traces in the development server for a long time:   These happen under normal interaction with the development server using Chrome. I have fixed one issue in CPython in wsgiref, but I belive that we should also fix isbrokenpipe_error in

def is_broken_pipe_error():     exc_type, _, _ = sys.exc_info()    return issubclass(exc_type, BrokenPipeError)    return issubclass(exc_type, (        BrokenPipeError,        ConnectionAbortedError,        ConnectionResetError,    ))   class WSGIServer(simple_server.WSGIServer):


Question: Deprecate MemcachedCache. Description python-memcached is not maintained anymore (see python-memcached) and it makes difficulties in fixing some issues (e.g. ). Moreover we a cache backend for pymemcache () so we have a good builtin alternative. I think it's time to deprecate the backend in Django 3.2 and remove it in Django 4.1.

def close_caches(**kwargs):    # Some caches -- python-memcached in particular -- need to do a cleanup at the    # end of a request cycle. If not implemented in a particular backend    # cache.close is a no-op    # Some caches need to do a cleanup at the end of a request cycle. If not    # implemented in a particular backend cache.close() is a no-op.     for cache in caches.all():         cache.close() diff --git a/django/core/cache/backends/memcached.py b/django/core/cache/backends/memcached.py-- a/django/core/cache/backends/memcached.py++ b/django/core/cache/backends/memcached.py


Question: Deprecate MemcachedCache. Description python-memcached is not maintained anymore (see python-memcached) and it makes difficulties in fixing some issues (e.g. ). Moreover we a cache backend for pymemcache () so we have a good builtin alternative. I think it's time to deprecate the backend in Django 3.2 and remove it in Django 4.1.

import pickle import re import timeimport warnings  from django.core.cache.backends.base import (     DEFAULT_TIMEOUT, BaseCache, InvalidCacheKey, memcache_key_warnings, )from django.utils.deprecation import RemovedInDjango41Warning from django.utils.functional import cached_property


Question: Deprecate MemcachedCache. Description python-memcached is not maintained anymore (see python-memcached) and it makes difficulties in fixing some issues (e.g. ). Moreover we a cache backend for pymemcache () so we have a good builtin alternative. I think it's time to deprecate the backend in Django 3.2 and remove it in Django 4.1.

class MemcachedCache(BaseMemcachedCache):     "An implementation of a cache binding using python-memcached"     def __init__(self, server, params):<ins>        warnings.warn(            'MemcachedCache is deprecated in favor of PyMemcacheCache and '            'PyLibMCCache.',            RemovedInDjango41Warning, stacklevel=2,        )</ins>         # python-memcached ≥ 1.45 returns None for a nonexistent key in         # incr/decr(), python-memcached < 1.45 raises ValueError.         import memcache


Question: Using isnull=True on a KeyTransform should not match JSON null on SQLite and Oracle Description The KeyTransformIsNull lookup borrows the logic from HasKey for isnull=False, which is correct. If isnull=True, the query should only match objects that do not have the key. The query is correct for MariaDB, MySQL, and PostgreSQL. However, on SQLite and Oracle, the query also matches objects that have the key with the value null, which is incorrect. To confirm, edit tests.modelfields.testjsonfield.TestQuerying.testisnullkey. For the first assertion, change self.assertSequenceEqual( (valueaisnull=True), [:3] + [5:], ) to self.assertSequenceEqual( (valuejisnull=True), [:4] + [5:], ) The test previously only checks with valuea which could not catch this behavior because the value is not JSON null.

class KeyTransformIsNull(lookups.IsNull):     # key__isnull=False is the same as has_key='key'     def as_oracle(self, compiler, connection):        sql, params = HasKey(            self.lhs.lhs,            self.lhs.key_name,        ).as_oracle(compiler, connection)         if not self.rhs:            return HasKey(self.lhs.lhs, self.lhs.key_name).as_oracle(compiler, connection)        return super().as_sql(compiler, connection)            return sql, params        # Column doesn't have a key or IS NULL.        lhs, lhs_params, _ = self.lhs.preprocess_lhs(compiler, connection)        return '(NOT %s OR %s IS NULL)' % (sql, lhs), tuple(params) + tuple(lhs_params)      def as_sqlite(self, compiler, connection):        template = 'JSON_TYPE(%s, %%s) IS NULL'         if not self.rhs:            return HasKey(self.lhs.lhs, self.lhs.key_name).as_sqlite(compiler, connection)        return super().as_sql(compiler, connection)            template = 'JSON_TYPE(%s, %%s) IS NOT NULL'        return HasKey(self.lhs.lhs, self.lhs.key_name).as_sql(            compiler,            connection,            template=template,        )   class KeyTransformIn(lookups.In):


Question: Log exceptions handled in Signal.sendrobust() Description As pointed out by Haki Benita on Twitter, by default Signal.sendrobust() doesn't have any log messages for exceptions raised in receivers. Since Django logs exceptions in other similar situations, such as missing template variables, I think it would be worth adding a logger.exception() call in the except clause of send_robust() . Users would then see such exceptions in their error handling tools, e.g. Sentry, and be able to figure out what action to take from there. Ultimately any expected exception should be caught with a try in the receiver function.

from django.utils.deprecation import RemovedInDjango40Warning from django.utils.inspect import func_accepts_kwargs <ins>logger = logging.getLogger('django.dispatch')</ins>  def _make_id(target):     if hasattr(target, '__func__'):


Question: Log exceptions handled in Signal.sendrobust() Description As pointed out by Haki Benita on Twitter, by default Signal.sendrobust() doesn't have any log messages for exceptions raised in receivers. Since Django logs exceptions in other similar situations, such as missing template variables, I think it would be worth adding a logger.exception() call in the except clause of send_robust() . Users would then see such exceptions in their error handling tools, e.g. Sentry, and be able to figure out what action to take from there. Ultimately any expected exception should be caught with a try in the receiver function.

try:                 response = receiver(signal=self, sender=sender, **named)             except Exception as err:<ins>                logger.error(                    'Error calling %s in Signal.send_robust() (%s)',                    receiver.__qualname__,                    err,                    exc_info=err,                )</ins>                 responses.append((receiver, err))             else:                 responses.append((receiver, response))


Question: squashmigrations optimizer crashes when fields' names are swapped using a temporary name Description If you rename fields using a pattern like a-c; b-a; c-b (such as if previously DateTimeFields using autonow and autonowadd had been mixed-up) and then attempt to squashmigrations with an optimization barrier between the CreateModel and RenameFields, the migration optimizer will attempt to create a CreateModel operation object with two fields using the same name and fail. I'll attach a migration file that triggers the failure. I believe the root cause of this issue is that allows itself to optimize through (i.e be moved to the right of, I may have gotten this terminology wrong) other RenameField operations that reference oldname.

),             ]         # Skip `FieldOperation.reduce` as we want to run `references_field`        # against self.new_name.        # against self.old_name and self.new_name.         return (             super(FieldOperation, self).reduce(operation, app_label) or            not operation.references_field(self.model_name, self.new_name, app_label)            not (                operation.references_field(self.model_name, self.old_name, app_label) or                operation.references_field(self.model_name, self.new_name, app_label)            )         )


Question: prefetchrelatedobjects() does not work for reused model instances. Description (last modified by Dennis Kliban) Our project processes instances in a stream. In some cases the instances are repeated. In these cases, we discovered that prefetchrelatedobjects() does not set the toattr on all of the instances if the first instance in the list already has it set. When Django determines that the very first instance in the list isfetched[0], it does not call into the the prefetchonelevel()[1]. The attributed specified in the toattr parameter is only set in the prefetchonelevel() method[2]. isfetched is set by looking for the to_attr attribute on the instance[3]. [0]  [1]  [2]  [3]

"prefetching - this is an invalid parameter to "                                  "prefetch_related()." % lookup.prefetch_through)             if prefetcher is not None and not is_fetched:                obj_list, additional_lookups = prefetch_one_level(obj_list, prefetcher, lookup, level)            obj_to_fetch = None            if prefetcher is not None:                obj_to_fetch = [obj for obj in obj_list if not is_fetched(obj)]            if obj_to_fetch:                obj_list, additional_lookups = prefetch_one_level(                    obj_to_fetch,                    prefetcher,                    lookup,                    level,                )                 # We need to ensure we don't keep adding lookups from the                 # same relationships to stop infinite recursion. So, if we                 # are already on an automatically added lookup, don't add


Question: prefetchrelatedobjects() does not work for reused model instances. Description (last modified by Dennis Kliban) Our project processes instances in a stream. In some cases the instances are repeated. In these cases, we discovered that prefetchrelatedobjects() does not set the toattr on all of the instances if the first instance in the list already has it set. When Django determines that the very first instance in the list isfetched[0], it does not call into the the prefetchonelevel()[1]. The attributed specified in the toattr parameter is only set in the prefetchonelevel() method[2]. isfetched is set by looking for the to_attr attribute on the instance[3]. [0]  [1]  [2]  [3]

(the object with get_prefetch_queryset (or None),      the descriptor object representing this relationship (or None),      a boolean that is False if the attribute was not found at all,     a boolean that is True if the attribute has already been fetched)     a function that takes an instance and returns a boolean that is True if     the attribute has already been fetched for that instance)     """    def has_to_attr_attribute(instance):        return hasattr(instance, to_attr)     prefetcher = None    is_fetched = False    is_fetched = has_to_attr_attribute      # For singly related objects, we have to avoid getting the attribute     # from the object, as this will trigger the query. So we first try


Question: prefetchrelatedobjects() does not work for reused model instances. Description (last modified by Dennis Kliban) Our project processes instances in a stream. In some cases the instances are repeated. In these cases, we discovered that prefetchrelatedobjects() does not set the toattr on all of the instances if the first instance in the list already has it set. When Django determines that the very first instance in the list isfetched[0], it does not call into the the prefetchonelevel()[1]. The attributed specified in the toattr parameter is only set in the prefetchonelevel() method[2]. isfetched is set by looking for the to_attr attribute on the instance[3]. [0]  [1]  [2]  [3]

# get_prefetch_queryset() method.             if hasattr(rel_obj_descriptor, 'get_prefetch_queryset'):                 prefetcher = rel_obj_descriptor                if rel_obj_descriptor.is_cached(instance):                    is_fetched = True                is_fetched = rel_obj_descriptor.is_cached             else:                 # descriptor doesn't support prefetching, so we go ahead and get                 # the attribute on the instance rather than the class to


Question: prefetchrelatedobjects() does not work for reused model instances. Description (last modified by Dennis Kliban) Our project processes instances in a stream. In some cases the instances are repeated. In these cases, we discovered that prefetchrelatedobjects() does not set the toattr on all of the instances if the first instance in the list already has it set. When Django determines that the very first instance in the list isfetched[0], it does not call into the the prefetchonelevel()[1]. The attributed specified in the toattr parameter is only set in the prefetchonelevel() method[2]. isfetched is set by looking for the to_attr attribute on the instance[3]. [0]  [1]  [2]  [3]

# Special case cached_property instances because hasattr                     # triggers attribute computation and assignment.                     if isinstance(getattr(instance.__class__, to_attr, None), cached_property):                        is_fetched = to_attr in instance.__dict__                    else:                        is_fetched = hasattr(instance, to_attr)                        def has_cached_property(instance):                            return to_attr in instance.__dict__                        is_fetched = has_cached_property                 else:                    is_fetched = through_attr in instance._prefetched_objects_cache                    def in_prefetched_cache(instance):                        return through_attr in instance._prefetched_objects_cache                    is_fetched = in_prefetched_cache     return prefetcher, rel_obj_descriptor, attr_found, is_fetched


Question: squashmigrations does not unset model options when optimizing CreateModel and AlterModelOptions Description When an operation resembling AlterModelOptions(name="testmodel", options={}) is squashed into the corresponding CreateModel operation, model options are not cleared on the resulting new CreateModel operation object. () sets the new options as options={self.options, operation.options} in this case ( line 144 on commit ), with no logic to remove options not found in operation.options as is found in AlterModelOptions.stateforwards(). I believe this issue still exists on the master branch based on my reading of the code, but I've only tested against 2.2.

),             ]         elif isinstance(operation, AlterModelOptions) and self.name_lower == operation.name_lower:            options = {**self.options, **operation.options}            for key in operation.ALTER_OPTION_KEYS:                if key not in operation.options:                    options.pop(key, None)             return [                 CreateModel(                     self.name,                     fields=self.fields,                    options={**self.options, **operation.options},                    options=options,                     bases=self.bases,                     managers=self.managers,                 ),


Question: parseduration() ISO string sign is ignored when the timedelta only has days Description I'm pretty sure that this is a bug even though I'm not an expert on the ISO 8601 standard. The sign of a timedelta string will be ignored by if the input string only contains days. Compare the following (notice the minus signs): In [4]: timedelta(days=-1) Out[4]: datetime.timedelta(days=-1) In [5]: td = timedelta(days=-1) In [6]: durationisostring(td) Out[6]: '-P1DT00H00M00S' In [7]: parseduration(durationisostring(td)) Out[7]: datetime.timedelta(days=1) # <-- Why is this 1 and not -1? In [8]: td = timedelta(days=-1, microseconds=1) In [9]: durationisostring(td) Out[9]: '-P0DT23H59M59.999999S' In [10]: parseduration(durationisostring(td)) Out[10]: datetime.timedelta(days=-1, microseconds=1) I guess the problem is in line 147 that reads return days + sign * datetime.timedelta(kw). However, if datetime.timedelta(kw) ends up being zero (timedelta(0)) then the sign multiplication ends up in zero, not -0. This is just a preliminary quick look though and maybe the problem is something else.

kw['microseconds'] = '-' + kw['microseconds']         kw = {k: float(v.replace(',', '.')) for k, v in kw.items() if v is not None}         days = datetime.timedelta(kw.pop('days', .0) or .0)<ins>        if match.re == iso8601_duration_re:            days *= sign</ins>         return days + sign * datetime.timedelta(**kw)


Question: add filter is unable to concatenate strings with lazy string Description If you try to concatenate a string with a lazy string with the add template filter, the result is always the empty string because the add filter generates an exception (TypeError: can only concatenate str (not "proxy") to str).

return str(self) % rhs             return self.__cast() % rhs <ins>        def __add__(self, other):            return self.__cast() + other        def __radd__(self, other):            return other + self.__cast()</ins>         def __deepcopy__(self, memo):             # Instances of this class are effectively immutable. It's just a             # collection of functions. So we don't need to do anything


Question: Add support for fixtures compression in dumpdata Description Since loaddata command support compressed fixtures [1] loading, I would propose to add fixtures compression support in dumpdata command. [1]

<ins>import gzipimport os</ins> import warnings  from django.apps import apps


Question: Add support for fixtures compression in dumpdata Description Since loaddata command support compressed fixtures [1] loading, I would propose to add fixtures compression support in dumpdata command. [1]

from django.core.management.utils import parse_apps_and_model_labels from django.db import DEFAULT_DB_ALIAS, router <ins>try:    import bz2    has_bz2 = Trueexcept ImportError:    has_bz2 = Falsetry:    import lzma    has_lzma = Trueexcept ImportError:    has_lzma = False</ins>  class ProxyModelWarning(Warning):     pass