Question: class VendorAlias(object):     def __init__(self):    def __init__(self, package_names):        self._package_names = package_names         self._vendor_name = __name__         self._vendor_pkg = self._vendor_name + "."        self._vendor_pkgs = [            self._vendor_pkg + name for name in self._package_names        ]      def find_module(self, fullname, path=None):         if fullname.startswith(self._vendor_pkg):

Issue with pyinstaller Can't seem to wrap my head around this. having a problem compiling a windows exe in python 2.7.9 that uses the requests library and can't find anything on google about the specific error. My script runs fine from the interpreter but when i use pyinstaller, i get :ImportError: No module named '' I can also compile windows executables that don't use requests just fine.


Question: )             ) <ins>        if not (name == self._vendor_name or                any(name.startswith(pkg) for pkg in self._vendor_pkgs)):            raise ImportError(                "Cannot import %s, must be one of %s." % (                    name, self._vendor_pkgs                )            )</ins>         # Check to see if we already have this item in sys.modules, if we do         # then simply return that.         if name in sys.modules:


Question: return module  sys.meta_path.append(VendorAlias())sys.meta_path.append(VendorAlias(["urllib3", "chardet"]))


Question: def prepare_method(self, method):         """Prepares the given HTTP method."""         self.method = method        if self.method is not None:            self.method = self.method.upper()        if self.method is None:            raise ValueError('Request method cannot be "None"')        self.method = to_native_string(self.method).upper()      def prepare_url(self, url, params):         """Prepares the given HTTP URL."""diff --git a/requests/sessions.py b/requests/sessions.py-- a/requests/sessions.py++ b/requests/sessions.py

Prepared requests containing binary files will not send when unicodeliterals is imported The above program works perfectly in python3, and in python2 when is not imported. If the request isn't prepared it works without a problem unfortunately, I require both prepared requests and in my project. The exception raised is: Traceback (most recent call last): File "", line 15, in (main()) File "", line 12, in main requests.Session().send(prepared) File "", line 573, in send r = (request, kwargs) File "", line 370, in send timeout=timeout File "", line 544, in urlopen body=body, headers=headers) File "", line 349, in makerequest conn.request(method, url, httplibrequestkw) File "", line 1001, in request self.sendrequest(method, url, body, headers) File "", line 1035, in sendrequest self.endheaders(body) File "", line 997, in endheaders self.sendoutput(messagebody) File "", line 848, in sendoutput msg += message_body UnicodeDecodeError: 'ascii' codec can't decode byte 0xff in position 109: ordinal not in range(128)


Question: :param cert: (optional) if String, path to ssl client cert file (.pem).             If Tuple, ('cert', 'key') pair.         """<del>        method = to_native_string(method)</del>         # Create the Request.         req = Request(             method = method.upper(),


Question: from .utils import (DEFAULT_CA_BUNDLE_PATH, get_encoding_from_headers,                     prepend_scheme_if_needed, get_auth_from_url, urldefragauth) from .structures import CaseInsensitiveDict<ins>from .packages.urllib3.exceptions import ClosedPoolError</ins> from .packages.urllib3.exceptions import ConnectTimeoutError from .packages.urllib3.exceptions import HTTPError as _HTTPError from .packages.urllib3.exceptions import MaxRetryError

urllib3 exceptions passing through requests API I don't know if it's a design goal of requests to hide urllib3's exceptions and wrap them around requests.exceptions types. (If it's not IMHO it should be, but that's another discussion) If it is, I have at least two of them passing through that I have to catch in addition to requests' exceptions. They are requests.packages.urllib3.exceptions.DecodeError and requests.packages.urllib3.exceptions.TimeoutError (this one I get when a proxy timeouts) Thanks!


Question: raise ConnectionError(e, request=request) <ins>        except ClosedPoolError as e:            raise ConnectionError(e, request=request)</ins>         except _ProxyError as e:             raise ProxyError(e)


Question: from datetime import datetime  from .auth import _basic_auth_strfrom .compat import cookielib, OrderedDict, urljoin, urlparsefrom .compat import cookielib, OrderedDict, urljoin, urlparse, is_py3, str from .cookies import (     cookiejar_from_dict, extract_cookies_to_jar, RequestsCookieJar, merge_cookies) from .models import Request, PreparedRequest, DEFAULT_REDIRECT_LIMIT

.htaccesss redirect to non ASCII folder does not work Hello, I have the following setup on a shared hoster: Apache 2.2.15 A Japanese language . (.minna; xn--q9jyb4c) IDN domain. A blog which is in the subfolder (blog) A redirect in the .htaccess file like this: So I usually open the domain http://test. and the server redirects to http://test./. This works fine in Firefox etc. With requests, I get the following error (Python 3.4 with Requests 2.7.0 on a Japanese Ubuntu 15.04): So I guess the request lib gets a redirect from a server with Japanese characters, but then fails to convert the characters correctly. If I do directly it works, only the redirect does not.


Question: parsed = urlparse(location_url)             location_url = parsed.geturl()             # On Python 3, the location header was decoded using Latin 1, but            # urlparse in requote_uri will encode it with UTF-8 before quoting.            # Because of this insanity, we need to fix it up ourselves by            # sending the URL back to bytes ourselves.            if is_py3 and isinstance(location_url, str):                location_url = location_url.encode('latin1')             # Facilitate relative 'location' headers, as allowed by RFC 7231.             # (e.g. '/path/to/resource' instead of 'http://domain.tld/path/to/resource')             # Compliant with RFC3986, we percent encode the url.diff --git a/requests/utils.py b/requests/utils.py-- a/requests/utils.py++ b/requests/utils.py


Question: from .compat import parse_http_list as _parse_list_header from .compat import (quote, urlparse, bytes, str, OrderedDict, unquote, is_py2,                      builtin_str, getproxies, proxy_bypass, urlunparse,                     basestring)                     basestring, is_py3) from .cookies import RequestsCookieJar, cookiejar_from_dict from .structures import CaseInsensitiveDict from .exceptions import InvalidURL, FileModeWarning


Question: """Un-escape any percent-escape sequences in a URI that are unreserved     characters. This leaves all reserved, illegal and non-ASCII bytes encoded.     """    parts = uri.split('%')    # This convert function is used to optionally convert the output of `chr`.    # In Python 3, `chr` returns a unicode string, while in Python 2 it returns    # a bytestring. Here we deal with that by optionally converting.    def convert(is_bytes, c):        if is_py2 and not is_bytes:            return c.decode('ascii')        elif is_py3 and is_bytes:            return c.encode('ascii')        else:            return c    # Handle both bytestrings and unicode strings.    is_bytes = isinstance(uri, bytes)    splitchar = u'%'    base = u''    if is_bytes:        splitchar = splitchar.encode('ascii')        base = base.encode('ascii')    parts = uri.split(splitchar)     for i in range(1, len(parts)):         h = parts[i][0:2]         if len(h) == 2 and h.isalnum():


Question: raise InvalidURL("Invalid percent-escape sequence: '%s'" % h)              if c in UNRESERVED_SET:                parts[i] = c + parts[i][2:]                parts[i] = convert(is_bytes, c) + parts[i][2:]             else:                parts[i] = '%' + parts[i]                parts[i] = splitchar + parts[i]         else:            parts[i] = '%' + parts[i]    return ''.join(parts)            parts[i] = splitchar + parts[i]    return base.join(parts)   def requote_uri(uri):


Question: """Prepares the given HTTP method."""         self.method = method         if self.method is not None:            self.method = self.method.upper()            self.method = to_native_string(self.method.upper())      def prepare_url(self, url, params):         """Prepares the given HTTP URL."""diff --git a/requests/sessions.py b/requests/sessions.py-- a/requests/sessions.py++ b/requests/sessions.py

TypeError after 2.7.0 -2.8.0 upgrade (cannot make memory view...) I'm running into this traceback after upgrading to 2.8.0 from 2.7.0: The problem goes away after downgrading to 2.7.0. A full traceback can be found . Here are the versions of relevant packages: And a full list of packages: Any ideas?


Question: :param cert: (optional) if String, path to ssl client cert file (.pem).             If Tuple, ('cert', 'key') pair.         """<del>        method = to_native_string(method)</del>         # Create the Request.         req = Request(             method = method.upper(),


Question: def super_len(o):    total_length = 0    current_position = 0     if hasattr(o, '__len__'):        return len(o)        total_length = len(o)    elif hasattr(o, 'len'):        total_length = o.len     if hasattr(o, 'len'):        return o.len    elif hasattr(o, 'getvalue'):        # e.g. BytesIO, cStringIO.StringIO        total_length = len(o.getvalue())     if hasattr(o, 'fileno'):    elif hasattr(o, 'fileno'):         try:             fileno = o.fileno()         except io.UnsupportedOperation:             pass         else:            filesize = os.fstat(fileno).st_size            total_length = os.fstat(fileno).st_size              # Having used fstat to determine the file length, we need to             # confirm that this file was opened up in binary mode.

Post request hangs in certain cases when body is a StringIO This is related to a report for the . The following hangs: After a cursory look, it looks like the request isn't fully formed so the server doesn't attempt to send a response which leaves the client hanging. If we call , this works. A bit more counterintuitively, this also works:


Question: FileModeWarning                 )             return filesize    if hasattr(o, 'tell'):        current_position = o.tell()     if hasattr(o, 'getvalue'):        # e.g. BytesIO, cStringIO.StringIO        return len(o.getvalue())    return max(0, total_length - current_position)   def get_netrc_auth(url, raise_errors=False):


Question: """          if isinstance(data, (str, bytes)):            return to_native_string(data)            return data         elif hasattr(data, 'read'):             return data         elif hasattr(data, '__iter__'):

Request with binary payload fails due to calling tonativestring Introduced with This works with 2.8.1, but not with 2.9.


Question: if isinstance(fragment, str):                 fragment = fragment.encode('utf-8') <ins>        if isinstance(params, (str, bytes)):            params = to_native_string(params)</ins>         enc_params = self._encode_params(params)         if enc_params:             if query:


Question: def stream_decode_response_unicode(iterator, r):     """Stream decodes a iterator."""    encoding = r.encoding     if r.encoding is None:        for item in iterator:            yield item        return    if encoding is None:        encoding = r.apparent_encoding    try:        decoder = codecs.getincrementaldecoder(encoding)(errors='replace')    except (LookupError, TypeError):        # A LookupError is raised if the encoding was not found which could        # indicate a misspelling or similar mistake.        #        # A TypeError can be raised if encoding is None        raise UnicodeError("Unable to decode contents with encoding %s." % encoding)     decoder = codecs.getincrementaldecoder(r.encoding)(errors='replace')     for chunk in iterator:         rv = decoder.decode(chunk)         if rv:

Uncertain about content/text vs itercontent(decodeunicode=True/False) When requesting an application/json document, I'm seeing returning bytes, whereas returns unicode. My understanding was that both should return a unicode object. In essence, I thought "itercontent" was equivalent to "itertext" when decode_unicode was True. Have I misunderstood something? I can provide an example if needed. For reference, I'm using python 3.5.1 and requests 2.10.0. Thanks!


Question: raise RuntimeError(                     'The content for this response was already consumed')             if self.status_code == 0:            if self.status_code == 0 or self.raw is None:                 self._content = None             else:                 self._content = bytes().join(self.iter_content(CONTENT_CHUNK_SIZE)) or bytes()

AttributeError: 'NoneType' object has no attribute 'read' Hello :) After a recent upgrade for our project to 2.12.1 we encounter an exception in our test suites which seems to be caused by . Build: Relevant part: happens on Windows and Linux. Thanks in advance :)


Question: url = url.lstrip()          # Don't do any URL preparation for non-HTTP schemes like `mailto`,        # `data`, `http+unix` etc to work around exceptions from `url_parse`,        # which handles RFC 3986 only.        if ':' in url and not url.lower().startswith(('http://', 'https://')):        # `data` etc to work around exceptions from `url_parse`, which        # handles RFC 3986 only.        if ':' in url and not url.lower().startswith('http'):             self.url = url             return

Requests v2.12.2 does not add parameters to URLs with schemes it does not understand. Follows from . When we patched to ignore all unrecognised schemes instead of just those that didn't begin , we stopped handling parameters for those URLs. This may break some more benign uses such as , which wanted to add parameters to their URLs. While this is inline with our intended policies (we do not understand URLs that have schemes we don't know anything about), this, along with the IDNA behaviour in v2.12.{0,1} will probably have broken a whole bunch of people using these non-standard URL schemes. We should consider whether Session objects should have a registry of places to look for URL preparation based on scheme. This will allow people to opt-in to the HTTP-like processing of URLs, as well as to register their own.


Question: # I don't like it either. Just look the other way. :)  for package in ('urllib3', 'idna', 'chardet'):    __import__(package)    locals()[package] = __import__(package)     # This traversal is apparently necessary such that the identities are     # preserved (requests.packages.urllib3.* is urllib3.*)     for mod in list(sys.modules):

AttributeError: module 'requests.packages' has no attribute 'urllib3' This seems to have broken requests.packages. requests.packages.urllib3 to be the urllib3 package AttributeError: module 'requests.packages' has no attribute 'urllib3' $ python -m


Question: from urllib3.poolmanager import PoolManager, proxy_from_url from urllib3.response import HTTPResponse<ins>from urllib3.util import parse_url</ins> from urllib3.util import Timeout as TimeoutSauce from urllib3.util.retry import Retry from urllib3.exceptions import ClosedPoolError

Misleading exception with invalid protocol in proxy variable When the value of or variable(s) accidentally miss one '/' in the protocol, a traceback is thrown to the user which doesn't pin point that the issue is with the proxy configuration. A better exception An exception which doesn't pin point exactly what went wrong. I am not sure what is the correct place to fix this. Should the fix/check be in requests, urllib3, or urlparse?


Question: from .structures import CaseInsensitiveDict from .cookies import extract_cookies_to_jar from .exceptions import (ConnectionError, ConnectTimeout, ReadTimeout, SSLError,                         ProxyError, RetryError, InvalidSchema)                         ProxyError, RetryError, InvalidSchema, InvalidProxyURL) from .auth import _basic_auth_str  try:


Question: if proxy:             proxy = prepend_scheme_if_needed(proxy, 'http')            proxy_url = parse_url(proxy)            if not proxy_url.host:                raise InvalidProxyURL("Please check proxy URL. It is malformed"                                      " and could be missing the host.")             proxy_manager = self.proxy_manager_for(proxy)             conn = proxy_manager.connection_from_url(url)         else:diff --git a/requests/exceptions.py b/requests/exceptions.py-- a/requests/exceptions.py++ b/requests/exceptions.py


Question: """The header value provided was somehow invalid."""  <ins>class InvalidProxyURL(InvalidURL):    """The proxy URL provided is invalid."""</ins> class ChunkedEncodingError(RequestException):     """The server declared chunked encoding but sent an invalid chunk."""


Question: return to_native_string(location, 'utf8')         return None <ins>    def should_strip_auth(self, old_url, new_url):        """Decide whether Authorization header should be removed when redirecting"""        old_parsed = urlparse(old_url)        new_parsed = urlparse(new_url)        if old_parsed.hostname != new_parsed.hostname:            return True        # Special case: allow http -> https redirect when using the standard        # ports. This isn't specified by RFC 7235, but is kept to avoid        # breaking backwards compatibility with older versions of requests        # that allowed any redirects on the same host.        if (old_parsed.scheme == 'http' and old_parsed.port in (80, None)                and new_parsed.scheme == 'https' and new_parsed.port in (443, None)):            return False        # Standard case: root URI must match        return old_parsed.port != new_parsed.port or old_parsed.scheme != new_parsed.scheme</ins>     def resolve_redirects(self, resp, req, stream=False, timeout=None,                           verify=True, cert=None, proxies=None, yield_requests=False, **adapter_kwargs):         """Receives a Response. Returns a generator of Responses or Requests."""

Should Authorization header be cleared in https -http redirect? This may be considered intentional behaviour (in which case feel free to close this), but if a request is made to an https endpoint with authorization and it redirects to http on the same host, the Authorization header is not stripped and will be exposed on the wire. rebuild_auth would strip the Authorization header if the scheme is changed from https to http. The credentials that were intended to be sent over TLS were transmitted in plaintext with the redirected request. Run an HTTPS server on localhost:4443 that replies with a 302 redirect to , and a plain HTTP server (or netcat) on localhost:8000. Then run The basic auth credentials are sent in plaintext to (the is just because I had a self-signed cert). Here's the code I used for the SSL server:


Question: headers = prepared_request.headers         url = prepared_request.url         if 'Authorization' in headers:        if 'Authorization' in headers and self.should_strip_auth(response.request.url, url):             # If we get redirected to a new host, we should strip out any             # authentication headers.            original_parsed = urlparse(response.request.url)            redirect_parsed = urlparse(url)            if (original_parsed.hostname != redirect_parsed.hostname):                del headers['Authorization']            del headers['Authorization']          # .netrc might have more auth for us on our new host.         new_auth = get_netrc_auth(url) if self.trust_env else None


Question: #: is a response.         self.request = None <ins>        #: If there was an error in the processing of content,        #: then save the error that would return the same error when you re-appeal.        self._error = None</ins>     def __enter__(self):         return self

Accessing response.content twice removes forgets read error I had a hard debugging time today because an error in the response stream is only reported when accessing for the first time. This is especially irritating when running code in a debugger. If accessing the first time raises an exception I would expect that accessing again would also raise an exception (ideally the same). Instead after raising on the first get, getting again returns an empty string. Here is a patch with a new test case for this: . Basically, it boils down to this: Output of my test case: $ python -m Edit: Oops, I used which actually called into system python 2.7. Here comes the real data: Thanks for looking into this!


Question: try:                     for chunk in self.raw.stream(chunk_size, decode_content=True):                         yield chunk                 except ProtocolError as e:                    raise ChunkedEncodingError(e)                    self._error = ChunkedEncodingError(e)                 except DecodeError as e:                    raise ContentDecodingError(e)                    self._error = ContentDecodingError(e)                 except ReadTimeoutError as e:                    raise ConnectionError(e)                    self._error = ConnectionError(e)                finally:                    # if we had an error - throw the saved error                    if self._error:                        raise self._error             else:                 # Standard file-like object.                 while True:


Question: else:                 self._content = b''.join(self.iter_content(CONTENT_CHUNK_SIZE)) or b'' <ins>        # if we had an error - throw the saved error        if self._error is not None:            raise self._error</ins>         self._content_consumed = True         # don't need to release the connection; that's been handled by urllib3         # since we exhausted the data.


Question: host = self._get_idna_encoded_host(host)             except UnicodeError:                 raise InvalidURL('URL has an invalid label.')        elif host.startswith(u'*'):        elif host.startswith((u'*', u'.')):             raise InvalidURL('URL has an invalid label.')          # Carefully reconstruct the network location

Getting raises UnicodeError Attempting to get e.g. results in a . It seems like the intention so far has been to raise instead (see e.g. ). I see there was some hesitation in fixing a similar issue () and would like to add that even catching the error just to rethrow as a requests exception would be beneficial. Based on PR : $ python -m


Question: if not netloc:         netloc, path = path, netloc <ins>    if auth:        # parse_url doesn't provide the netloc with auth        # so we'll add it ourselves.        netloc = '@'.join([auth, netloc])</ins>     if scheme is None:         scheme = new_scheme     if path is None:

Proxy authentication bug <!-- Summary. --When using proxies in python 3.8.12, I get an error 407. Using any other version of python works fine. I am assuming it could be to do with this <!-- What you expected. --I should get a status of 200. <!-- What happened instead. --I get a status code of 407.


Question: if not scheme in SCHEMAS:             raise InvalidSchema("Invalid scheme %r" % scheme)         netloc = netloc.encode('idna').decode('utf-8')        try:            netloc = netloc.encode('idna').decode('utf-8')        except UnicodeError:            raise InvalidURL('URL has an invalid label.')          if not path:             path = '/'

Catch UnicodeError coming from Here's a Python 2.6 Requests 0.13.1 traceback. Looks like the call to needs a try/except.


Question: def register_hook(self, event, hook):         """Properly register a hook."""        self.hooks[event].append(hook)        if isinstance(hook, (list, tuple, set)):            self.hooks[event].extend(hook)        else:            self.hooks[event].append(hook)      def deregister_hook(self, event, hook):         """Deregister a previously registered hook.

Allow lists in the dict values of the hooks argument Currently the Request class has a .registerhook() method but it parses the dictionary it expects from it's hooks argument weirdly: the argument can only specify one hook function per hook. If you pass in a list of hook functions per hook the code in () will wrap the list in a list which then fails when the hooks are consumed (since a list is not callable). This is especially annoying since you can not use multiple hooks from a session. The only way to get multiple hooks now is to create the request object without sending it, then call .registerhook() multiple times and then finally call .send(). This would all be much easier if () parsed the hooks parameter in a way that it accepts lists as it's values.


Question: data = np.timedelta64(getattr(data, "value", data), "ns")      # we don't want nested self-described arrays    data = getattr(data, "values", data)    if isinstance(data, (pd.Series, pd.Index, pd.DataFrame)):        data = data.values      if isinstance(data, np.ma.MaskedArray):         mask = np.ma.getmaskarray(data)

Variable.setitem coercing types on objects with a values property The current behavior prevents storing objects inside arrays of even when only performing non-broadcasted assignments if the RHS has a property. Many libraries produce objects with a property that gets coerced as a result. The use case I had in prior versions was to store instances from the curve fitting library , when fitting had be performed over an axis of a or . Ideally: Breaking changed introduced going from -as a result of , namely the change on line <detailsINSTALLED VERSIONS commit: None python: python-bits: 64 OS: Darwin OS-release: 16.7.0 machine: x8664 processor: i386 byteorder: little LCALL: None LANG: enUS.UTF-8 LOCALE: enUS.UTF-8 xarray: 0.10.1 pandas: 0.20.3 numpy: 1.13.1 scipy: 0.19.1 netCDF4: 1.3.0 h5netcdf: None h5py: 2.7.0 Nio: None zarr: None bottleneck: None cyordereddict: None dask: 0.15.2 distributed: None matplotlib: 2.0.2 cartopy: None seaborn: 0.8.1 setuptools: 38.4.0 pip: 9.0.1 conda: None pytest: 3.3.2 IPython: 6.1.0 sphinx: None </detailsThank you for your help! If I can be brought to better understand any constraints to adjacent issues, I can consider drafting a fix for this.


Question: },         )     def weighted(self, weights):        """        Weighted operations.        Parameters        ----------        weights : DataArray            An array of weights associated with the values in this Dataset.            Each value in the data contributes to the reduction operation            according to its associated weight.        Notes        -----        ``weights`` must be a DataArray and cannot contain missing values.        Missing values can be replaced by ``weights.fillna(0)``.        """        return self._weighted_cls(self, weights)     def rolling(         self,         dim: Mapping[Hashable, int] = None,diff --git a/xarray/core/dataarray.py b/xarray/core/dataarray.py-- a/xarray/core/dataarray.py++ b/xarray/core/dataarray.py

add average function It would be nice to be able to do to compute weighted averages (e.g. for geo data). Of course this would require the axes to be in a predictable order. Or to give a weight per dimension...


Question: resample,     rolling,     utils,<ins>    weighted,</ins> ) from .accessor_dt import CombinedDatetimelikeAccessor from .accessor_str import StringAccessor


Question: _rolling_cls = rolling.DataArrayRolling     _coarsen_cls = rolling.DataArrayCoarsen     _resample_cls = resample.DataArrayResample    _weighted_cls = weighted.DataArrayWeighted      dt = property(CombinedDatetimelikeAccessor) diff --git a/xarray/core/dataset.py b/xarray/core/dataset.py-- a/xarray/core/dataset.py++ b/xarray/core/dataset.py


Question: resample,     rolling,     utils,<ins>    weighted,</ins> ) from .alignment import _broadcast_helper, _get_broadcast_dims_map_common_coords, align from .common import (


Question: _rolling_cls = rolling.DatasetRolling     _coarsen_cls = rolling.DatasetCoarsen     _resample_cls = resample.DatasetResample    _weighted_cls = weighted.DatasetWeighted      def __init__(         self,diff --git a/xarray/core/weighted.py b/xarray/core/weighted.pynew file mode 100644-- /dev/null++ b/xarray/core/weighted.py


Question: <ins>from typing import TYPE_CHECKING, Hashable, Iterable, Optional, Union, overloadfrom .computation import dotfrom .options import _get_keep_attrsif TYPE_CHECKING:    from .dataarray import DataArray, Dataset_WEIGHTED_REDUCE_DOCSTRING_TEMPLATE = """    Reduce this {cls}'s data by a weighted ``{fcn}`` along some dimension(s).    Parameters    ----------    dim : str or sequence of str, optional        Dimension(s) over which to apply the weighted ``{fcn}``.    skipna : bool, optional        If True, skip missing values (as marked by NaN). By default, only        skips missing values for float dtypes; other dtypes either do not        have a sentinel missing value (int) or skipna=True has not been        implemented (object, datetime64 or timedelta64).    keep_attrs : bool, optional        If True, the attributes (``attrs``) will be copied from the original        object to the new one.  If False (default), the new object will be        returned without attributes.    Returns    -------    reduced : {cls}        New {cls} object with weighted ``{fcn}`` applied to its data and        the indicated dimension(s) removed.    Notes    -----        Returns {on_zero} if the ``weights`` sum to 0.0 along the reduced        dimension(s).    """_SUM_OF_WEIGHTS_DOCSTRING = """    Calculate the sum of weights, accounting for missing values in the data    Parameters    ----------    dim : str or sequence of str, optional        Dimension(s) over which to sum the weights.    keep_attrs : bool, optional        If True, the attributes (``attrs``) will be copied from the original        object to the new one.  If False (default), the new object will be        returned without attributes.    Returns    -------    reduced : {cls}        New {cls} object with the sum of the weights over the given dimension.    """class Weighted:    """An object that implements weighted operations.    You should create a Weighted object by using the ``DataArray.weighted`` or    ``Dataset.weighted`` methods.    See Also    --------    Dataset.weighted    DataArray.weighted    """    __slots__ = ("obj", "weights")    @overload    def __init__(self, obj: "DataArray", weights: "DataArray") -> None:        ...    @overload  # noqa: F811    def __init__(self, obj: "Dataset", weights: "DataArray") -> None:  # noqa: F811        ...    def __init__(self, obj, weights):  # noqa: F811        """        Create a Weighted object        Parameters        ----------        obj : DataArray or Dataset            Object over which the weighted reduction operation is applied.        weights : DataArray            An array of weights associated with the values in the obj.            Each value in the obj contributes to the reduction operation            according to its associated weight.        Notes        -----        ``weights`` must be a ``DataArray`` and cannot contain missing values.        Missing values can be replaced by ``weights.fillna(0)``.        """        from .dataarray import DataArray        if not isinstance(weights, DataArray):            raise ValueError("`weights` must be a DataArray")        if weights.isnull().any():            raise ValueError(                "`weights` cannot contain missing values. "                "Missing values can be replaced by `weights.fillna(0)`."            )        self.obj = obj        self.weights = weights    @staticmethod    def _reduce(        da: "DataArray",        weights: "DataArray",        dim: Optional[Union[Hashable, Iterable[Hashable]]] = None,        skipna: Optional[bool] = None,    ) -> "DataArray":        """reduce using dot; equivalent to (da * weights).sum(dim, skipna)            for internal use only        """        # need to infer dims as we use `dot`        if dim is None:            dim = ...        # need to mask invalid values in da, as `dot` does not implement skipna        if skipna or (skipna is None and da.dtype.kind in "cfO"):            da = da.fillna(0.0)        # `dot` does not broadcast arrays, so this avoids creating a large        # DataArray (if `weights` has additional dimensions)        # maybe add fasttrack (`(da * weights).sum(dims=dim, skipna=skipna)`)        return dot(da, weights, dims=dim)    def _sum_of_weights(        self, da: "DataArray", dim: Optional[Union[Hashable, Iterable[Hashable]]] = None    ) -> "DataArray":        """ Calculate the sum of weights, accounting for missing values """        # we need to mask data values that are nan; else the weights are wrong        mask = da.notnull()        sum_of_weights = self._reduce(mask, self.weights, dim=dim, skipna=False)        # 0-weights are not valid        valid_weights = sum_of_weights != 0.0        return sum_of_weights.where(valid_weights)    def _weighted_sum(        self,        da: "DataArray",        dim: Optional[Union[Hashable, Iterable[Hashable]]] = None,        skipna: Optional[bool] = None,    ) -> "DataArray":        """Reduce a DataArray by a by a weighted ``sum`` along some dimension(s)."""        return self._reduce(da, self.weights, dim=dim, skipna=skipna)    def _weighted_mean(        self,        da: "DataArray",        dim: Optional[Union[Hashable, Iterable[Hashable]]] = None,        skipna: Optional[bool] = None,    ) -> "DataArray":        """Reduce a DataArray by a weighted ``mean`` along some dimension(s)."""        weighted_sum = self._weighted_sum(da, dim=dim, skipna=skipna)        sum_of_weights = self._sum_of_weights(da, dim=dim)        return weighted_sum / sum_of_weights    def _implementation(self, func, dim, **kwargs):        raise NotImplementedError("Use `Dataset.weighted` or `DataArray.weighted`")    def sum_of_weights(        self,        dim: Optional[Union[Hashable, Iterable[Hashable]]] = None,        keep_attrs: Optional[bool] = None,    ) -> Union["DataArray", "Dataset"]:        return self._implementation(            self._sum_of_weights, dim=dim, keep_attrs=keep_attrs        )    def sum(        self,        dim: Optional[Union[Hashable, Iterable[Hashable]]] = None,        skipna: Optional[bool] = None,        keep_attrs: Optional[bool] = None,    ) -> Union["DataArray", "Dataset"]:        return self._implementation(            self._weighted_sum, dim=dim, skipna=skipna, keep_attrs=keep_attrs        )    def mean(        self,        dim: Optional[Union[Hashable, Iterable[Hashable]]] = None,        skipna: Optional[bool] = None,        keep_attrs: Optional[bool] = None,    ) -> Union["DataArray", "Dataset"]:        return self._implementation(            self._weighted_mean, dim=dim, skipna=skipna, keep_attrs=keep_attrs        )    def __repr__(self):        """provide a nice str repr of our Weighted object"""        klass = self.__class__.__name__        weight_dims = ", ".join(self.weights.dims)        return f"{klass} with weights along dimensions: {weight_dims}"class DataArrayWeighted(Weighted):    def _implementation(self, func, dim, **kwargs):        keep_attrs = kwargs.pop("keep_attrs")        if keep_attrs is None:            keep_attrs = _get_keep_attrs(default=False)        weighted = func(self.obj, dim=dim, **kwargs)        if keep_attrs:            weighted.attrs = self.obj.attrs        return weightedclass DatasetWeighted(Weighted):    def _implementation(self, func, dim, **kwargs) -> "Dataset":        return self.obj.map(func, dim=dim, **kwargs)def _inject_docstring(cls, cls_name):    cls.sum_of_weights.__doc__ = _SUM_OF_WEIGHTS_DOCSTRING.format(cls=cls_name)    cls.sum.__doc__ = _WEIGHTED_REDUCE_DOCSTRING_TEMPLATE.format(        cls=cls_name, fcn="sum", on_zero="0"    )    cls.mean.__doc__ = _WEIGHTED_REDUCE_DOCSTRING_TEMPLATE.format(        cls=cls_name, fcn="mean", on_zero="NaN"    )_inject_docstring(DataArrayWeighted, "DataArray")_inject_docstring(DatasetWeighted, "Dataset")</ins>


Question: from collections import defaultdict from contextlib import suppress from datetime import timedeltafrom typing import Sequencefrom typing import Any, Tuple, Sequence, Union  import numpy as np import pandas as pd  from . import duck_array_ops, nputils, utilsfrom .npcompat import DTypeLike from .pycompat import dask_array_type, integer_types from .utils import is_dict_like

REGRESSION: copy(deep=True) casts unicode indices to object (deep=True) and (deep=True/False) accidentally cast IndexVariable's with dtype='<U*' to object. Same applies to () and copy.deepcopy(). This is a regression in xarray = 0.12.2. xarray 0.12.1 and earlier are unaffected.


Question: class PandasIndexAdapter(ExplicitlyIndexedNDArrayMixin):    """Wrap a pandas.Index to preserve dtypes and handle explicit indexing."""    """Wrap a pandas.Index to preserve dtypes and handle explicit indexing.    """     def __init__(self, array, dtype=None):    def __init__(self, array: Any, dtype: DTypeLike = None):         self.array = utils.safe_cast_to_index(array)         if dtype is None:             if isinstance(array, pd.PeriodIndex):


Question: dtype = np.dtype('O')             else:                 dtype = array.dtype        else:            dtype = np.dtype(dtype)         self._dtype = dtype      @property    def dtype(self):    def dtype(self) -> np.dtype:         return self._dtype     def __array__(self, dtype=None):    def __array__(self, dtype: DTypeLike = None) -> np.ndarray:         if dtype is None:             dtype = self.dtype         array = self.array


Question: return np.asarray(array.values, dtype=dtype)      @property    def shape(self):    def shape(self) -> Tuple[int]:         # .shape is broken on pandas prior to v0.15.2         return (len(self.array),)     def __getitem__(self, indexer):    def __getitem__(            self, indexer    ) -> Union[        NumpyIndexingAdapter,        np.ndarray,        np.datetime64,        np.timedelta64,    ]:         key = indexer.tuple         if isinstance(key, tuple) and len(key) == 1:             # unpack key so it can index a pandas.Index object (pandas.Index


Question: return result     def transpose(self, order):    def transpose(self, order) -> pd.Index:         return self.array  # self.array should be always one-dimensional     def __repr__(self):    def __repr__(self) -> str:         return ('%s(array=%r, dtype=%r)'                 % (type(self).__name__, self.array, self.dtype))    def copy(self, deep: bool = True) -> 'PandasIndexAdapter':        # Not the same as just writing `self.array.copy(deep=deep)`, as        # shallow copies of the underlying numpy.ndarrays become deep ones        # upon pickling        # >>> len(pickle.dumps((self.array, self.array)))        # 4000281        # >>> len(pickle.dumps((self.array, self.array.copy(deep=False))))        # 8000341        array = self.array.copy(deep=True) if deep else self.array        return PandasIndexAdapter(array, self._dtype)diff --git a/xarray/core/variable.py b/xarray/core/variable.py-- a/xarray/core/variable.py++ b/xarray/core/variable.py


Question: data copied from original.         """         if data is None:            if deep:                # self._data should be a `PandasIndexAdapter` instance at this                # point, which doesn't have a copy method, so make a deep copy                # of the underlying `pandas.MultiIndex` and create a new                # `PandasIndexAdapter` instance with it.                data = PandasIndexAdapter(self._data.array.copy(deep=True))            else:                data = self._data            data = self._data.copy(deep=deep)         else:             data = as_compatible_data(data)             if self.shape != data.shape:


Question: Mapping[Hashable, Any]] = None,                     axis=None, **dim_kwargs: Any) -> 'DataArray':         """Return a new object with an additional axis (or axes) inserted at        the corresponding position in the array shape.        the corresponding position in the array shape. The new object is a        view into the underlying array, not a copy.          If dim is already a scalar coordinate, it will be promoted to a 1D         coordinate consisting of a single value.diff --git a/xarray/core/dataset.py b/xarray/core/dataset.py-- a/xarray/core/dataset.py++ b/xarray/core/dataset.py

expanddims() modifies to write only, upon manually reverting this flag back, attempting to set a single inner value using .loc will instead set all of the inner array values I am using the newly updated expanddims API that was recently updated with this PR . However the flag setting behaviour can also be observed using the old API syntax. When applying the operation 'expanddims({'z': 3}, -1)' on a DataArray the underlying Numpy array flags are changed. 'CCONTIGUOUS' is set to False, and 'WRITEABLE' is set to False, and 'OWNDATA' is set to False. Upon changing 'WRITEABLE' back to True, when I try to set a single value in the DataArray using the '.loc' operator it will instead set all the values in that selected inner array. I am new to Xarray so I can't be entirely sure if this expected behaviour. Regardless I would expect that adding a new dimension to the array would not make that array 'read-only'. I would also not expect the '.loc' method to work differently to how it would otherwise. It's also not congruent with the Numpy 'expanddims' operation. Because when I call the operation 'np.expanddims(nparr, axis=-1)' the 'CCONTIGUOUS ' and 'WRITEABLE ' flags will not be modified.


Question: Mapping[Hashable, Any]] = None,                     axis=None, **dim_kwargs: Any) -> 'DataArray':         """Return a new object with an additional axis (or axes) inserted at        the corresponding position in the array shape.        the corresponding position in the array shape. The new object is a        view into the underlying array, not a copy.          If dim is already a scalar coordinate, it will be promoted to a 1D         coordinate consisting of a single value.diff --git a/xarray/core/dataset.py b/xarray/core/dataset.py-- a/xarray/core/dataset.py++ b/xarray/core/dataset.py

Here is a similar flow of operations that demonstrates the behaviour I would expect from the DataArray after applying 'expanddims': <detailsINSTALLED VERSIONS commit: None python: 3.6.7 (default, Dec 29 2018, 12:05:36) [GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.11.45.5)] python-bits: 64 OS: Darwin OS-release: 18.2.0 machine: x8664 processor: i386 byteorder: little LCALL: None LANG: enAU.UTF-8 LOCALE: enAU.UTF-8 libhdf5: 1.10.2 libnetcdf: 4.4.1.1 xarray: 0.12.1 pandas: 0.24.2 numpy: 1.16.2 scipy: 1.2.1 netCDF4: 1.5.0 pydap: None h5netcdf: None h5py: None Nio: None zarr: None cftime: 1.0.3.4 nctime_axis: None PseudonetCDF: None rasterio: None cfgrib: 0.9.6.1.post1 iris: None bottleneck: None dask: None distributed: None matplotlib: 3.0.3 cartopy: 0.17.0 seaborn: None setuptools: 39.0.1 pip: 10.0.1 conda: None pytest: None IPython: None sphinx: None </details


Question: def expand_dims(self, dim=None, axis=None, **dim_kwargs):         """Return a new object with an additional axis (or axes) inserted at        the corresponding position in the array shape.        the corresponding position in the array shape.  The new object is a        view into the underlying array, not a copy.          If dim is already a scalar coordinate, it will be promoted to a 1D         coordinate consisting of a single value.diff --git a/xarray/core/indexing.py b/xarray/core/indexing.py-- a/xarray/core/indexing.py++ b/xarray/core/indexing.py


Question: def expand_dims(self, dim=None, axis=None, **dim_kwargs):         """Return a new object with an additional axis (or axes) inserted at        the corresponding position in the array shape.        the corresponding position in the array shape.  The new object is a        view into the underlying array, not a copy.          If dim is already a scalar coordinate, it will be promoted to a 1D         coordinate consisting of a single value.diff --git a/xarray/core/indexing.py b/xarray/core/indexing.py-- a/xarray/core/indexing.py++ b/xarray/core/indexing.py


Question: def __setitem__(self, key, value):         array, key = self._indexing_array_and_key(key)        array[key] = value        try:            array[key] = value        except ValueError:            # More informative exception if read-only view            if not array.flags.writeable and not array.flags.owndata:                raise ValueError("Assignment destination is a view.  "                                 "Do you want to .copy() array first?")            else:                raise   class DaskIndexingAdapter(ExplicitlyIndexedNDArrayMixin):


Question: def __setitem__(self, key, value):         array, key = self._indexing_array_and_key(key)        array[key] = value        try:            array[key] = value        except ValueError:            # More informative exception if read-only view            if not array.flags.writeable and not array.flags.owndata:                raise ValueError("Assignment destination is a view.  "                                 "Do you want to .copy() array first?")            else:                raise   class DaskIndexingAdapter(ExplicitlyIndexedNDArrayMixin):


Question: fill_value=fill_value)          # Check the overall coordinates are monotonically increasing        for dim in concatenated.dims:            if dim in concatenated:                indexes = concatenated.indexes.get(dim)                if not (indexes.is_monotonic_increasing                        or indexes.is_monotonic_decreasing):                    raise ValueError("Resulting object does not have monotonic"                                     " global indexes along dimension {}"                                     .format(dim))        for dim in concat_dims:            indexes = concatenated.indexes.get(dim)            if not (indexes.is_monotonic_increasing                    or indexes.is_monotonic_decreasing):                raise ValueError("Resulting object does not have monotonic"                                 " global indexes along dimension {}"                                 .format(dim))         concatenated_grouped_by_data_vars.append(concatenated)      return merge(concatenated_grouped_by_data_vars, compat=compat,

xr.combinebycoords raises ValueError if identical coordinates are non-monotonic <!-- In order for the maintainers to efficiently understand and prioritize issues, we ask you post a "Minimal, Complete and Verifiable Example" (MCVE): --should return without error. Running the example with raises an error: The documentation for says that "Non-coordinate dimensions will be ignored, as will any coordinate dimensions which do not vary between each dataset". This is not the case with the current implementation, since identical coordinate dimensions are still required to be monotonic. <detailsINSTALLED VERSIONS commit: None python: 3.7.1 (v3.7.1:, Oct 20 2018, 14:57:15) [MSC v.1915 64 bit (AMD64)] python-bits: 64 OS: Windows OS-release: 10 machine: AMD64 processor: Intel64 Family 6 Model 94 Stepping 3, GenuineIntel byteorder: little LCALL: None LANG: None LOCALE: None.None libhdf5: None libnetcdf: None xarray: 0.12.3 pandas: 0.24.2 numpy: 1.16.4 scipy: 1.3.0 netCDF4: None pydap: None h5netcdf: None h5py: None Nio: None zarr: None cftime: None nctime_axis: None PseudoNetCDF: None rasterio: None cfgrib: None iris: None bottleneck: None dask: None distributed: None matplotlib: 3.1.1 cartopy: None seaborn: 0.9.0 numbagg: None setuptools: 39.0.1 pip: 10.0.1 conda: None pytest: None IPython: 7.1.1 sphinx: None </details


Question: raise TypeError('`group` must be an xarray.DataArray or the '                                 'name of an xarray variable or dimension')             group = obj[group]<ins>            if len(group) == 0:                raise ValueError("{} must not be empty".format(group.name))</ins>             if group.name not in obj.coords and group.name in obj.dims:                 # DummyGroups should not appear on groupby results                 group = _DummyGroup(obj, group.name, group.coords)

groupby on empty DataArray raises StopIteration This seems similar to and so apologies if it is a duplicate, but I have a minimal example where it happens on an empty DataArray: Using groupby on an empty DataArray or Dataset raises . It should raise a more meaningful error. In particular, I had this issue in a function I was calling inside of a generator, so the StopIteration just broke out of the generator and it took some digging to figure out what was going wrong in my code. <detailsINSTALLED VERSIONS commit: None python: python-bits: 64 OS: Linux OS-release: 3.10.0-862.14.4.el7.x8664 machine: x8664 processor: x8664 byteorder: little LCALL: None LANG: enUS.utf8 LOCALE: enUS.UTF-8 xarray: 0.10.7 pandas: 0.23.0 numpy: 1.14.3 scipy: 1.1.0 netCDF4: 1.4.0 h5netcdf: 0.6.1 h5py: 2.7.1 Nio: None zarr: None bottleneck: 1.2.1 cyordereddict: None dask: 0.17.5 distributed: 1.21.8 matplotlib: 2.2.2 cartopy: None seaborn: 0.8.1 setuptools: 39.1.0 pip: 10.0.1 conda: None pytest: 3.5.1 IPython: 6.4.0 sphinx: 1.7.4 </details


Question: return new_coords, dims  <ins>def _check_data_shape(data, coords, dims):    if data is dtypes.NA:        data = np.nan    if coords is not None and utils.is_scalar(data, include_0d=False):        if utils.is_dict_like(coords):            if dims is None:                return data            else:                data_shape = tuple(                    as_variable(coords[k], k).size if k in coords.keys() else 1                    for k in dims                )        else:            data_shape = tuple(as_variable(coord, "foo").size for coord in coords)        data = np.full(data_shape, data)    return data</ins> class _LocIndexer:     def __init__(self, data_array: "DataArray"):         self.data_array = data_array

Allow passing a default value (instead of ndarray) for data argument for DataArray Hi, First of all, thanks a lot for the amazing module. It seems when I create a DataArray, I have to pass a numpy.ndarray with a correct size for the argument. It works well when I already have some data, but sometimes I want to create an "empty" DataArray with known coordinates and fill up the data later. For these cases, it would be great if xarray allows passing just a value for the argument, and fill up all the elements of the array with the value. For example, with pandas, I can do: and the resulting would be: Thanks a lot!


Question: def __init__(         self,        data: Any,        data: Any = dtypes.NA,         coords: Union[Sequence[Tuple], Mapping[Hashable, Any], None] = None,         dims: Union[Hashable, Sequence[Hashable], None] = None,         name: Hashable = None,


Question: if encoding is None:                 encoding = getattr(data, "encoding", None)             data = _check_data_shape(data, coords, dims)             data = as_compatible_data(data)             coords, dims = _infer_coords_and_dims(data.shape, coords, dims)             variable = Variable(dims, data, attrs, encoding, fastpath=True)diff --git a/xarray/core/utils.py b/xarray/core/utils.py-- a/xarray/core/utils.py++ b/xarray/core/utils.py


Question: import numpy as np import pandas as pd <del>from .pycompat import dask_array_type</del> K = TypeVar("K") V = TypeVar("V") T = TypeVar("T")


Question: return cast(Mapping[Hashable, T], kw_kwargs)  def is_scalar(value: Any) -> bool:def is_scalar(value: Any, include_0d: bool = True) -> bool:     """Whether to treat a value as a scalar.      Any non-iterable, string, or 0-D array     """    from .variable import NON_NUMPY_SUPPORTED_ARRAY_TYPES    if include_0d:        include_0d = getattr(value, "ndim", None) == 0     return (        getattr(value, "ndim", None) == 0        include_0d         or isinstance(value, (str, bytes))         or not (            isinstance(value, (Iterable,) + dask_array_type)            isinstance(value, (Iterable,) + NON_NUMPY_SUPPORTED_ARRAY_TYPES)             or hasattr(value, "__array_function__")         )     )


Question: # the former is often more efficient                             reduce_dims = None                         variables[name] = var.quantile(                            q, dim=reduce_dims, interpolation=interpolation                            q,                            dim=reduce_dims,                            interpolation=interpolation,                            keep_attrs=keep_attrs,                         )              else:diff --git a/xarray/core/variable.py b/xarray/core/variable.py-- a/xarray/core/variable.py++ b/xarray/core/variable.py

DataArray.quantile does not honor <!-- In order for the maintainers to efficiently understand and prioritize issues, we ask you post a "Minimal, Complete and Verifiable Example" (MCVE): --returns <detailsINSTALLED VERSIONS commit: python: 3.6.8 (default, Dec 30 2018, 01:22:34) [GCC 7.3.0] python-bits: 64 OS: Linux OS-release: 4.15.0-60-generic machine: x8664 processor: x8664 byteorder: little LCALL: None LANG: enCA.UTF-8 LOCALE: enCA.UTF-8 libhdf5: 1.10.2 libnetcdf: 4.6.1 xarray: pandas: 0.23.4 numpy: 1.16.1 scipy: 1.1.0 netCDF4: 1.3.1 pydap: installed h5netcdf: None h5py: None Nio: None zarr: None cftime: 1.0.3.4 nctimeaxis: None PseudoNetCDF: None rasterio: None cfgrib: None iris: None bottleneck: 1.2.1 dask: 0.19.0 distributed: 1.23.0 matplotlib: 3.0.2 cartopy: 0.17.0 seaborn: None numbagg: None setuptools: 41.0.0 pip: 9.0.1 conda: None pytest: 4.4.0 IPython: 7.0.1 sphinx: 1.7.1 </details


Question: """         return self.broadcast_equals(other, equiv=duck_array_ops.array_notnull_equiv)     def quantile(self, q, dim=None, interpolation="linear"):    def quantile(self, q, dim=None, interpolation="linear", keep_attrs=None):         """Compute the qth quantile of the data along the specified dimension.          Returns the qth quantiles(s) of the array elements.


Question: * higher: ``j``.                 * nearest: ``i`` or ``j``, whichever is nearest.                 * midpoint: ``(i + j) / 2``.<ins>        keep_attrs : bool, optional            If True, the variable's attributes (`attrs`) will be copied from            the original object to the new one.  If False (default), the new            object will be returned without attributes.</ins>          Returns         -------


Question: is a scalar. If multiple percentiles are given, first axis of             the result corresponds to the quantile and a quantile dimension             is added to the return array. The other dimensions are the             dimensions that remain after the reduction of the array.            dimensions that remain after the reduction of the array.          See Also         --------


Question: axis = None             new_dims = []         # only add the quantile dimension if q is array like        # Only add the quantile dimension if q is array-like         if q.ndim != 0:             new_dims = ["quantile"] + new_dims          qs = np.nanpercentile(             self.data, q * 100.0, axis=axis, interpolation=interpolation         )        return Variable(new_dims, qs)        if keep_attrs is None:            keep_attrs = _get_keep_attrs(default=False)        attrs = self._attrs if keep_attrs else None        return Variable(new_dims, qs, attrs)      def rank(self, dim, pct=False):         """Ranks the data.


Question: to_merge = {var: [] for var in variables_to_merge}          for ds in datasets:            absent_merge_vars = variables_to_merge - set(ds.variables)            if absent_merge_vars:                raise ValueError(                    "variables %r are present in some datasets but not others. "                    % absent_merge_vars                )             for var in variables_to_merge:                to_merge[var].append(ds.variables[var])                if var in ds:                    to_merge[var].append(ds.variables[var])          for var in variables_to_merge:             result_vars[var] = unique_variable(

Ignore missing variables when concatenating datasets? Several users ( now myself) have wondered about how to concatenate xray Datasets with different variables. With the current , you need to awkwardly create dummy variables filled with in datasets that don't have them (or drop mismatched variables entirely). Neither of these are great options -- should have an option (the default?) to take care of this for the user. This would also be more consistent with , which takes a more relaxed approach to matching dataframes with different variables (it does an outer join).


Question: group_indices = [slice(i, i + 1) for i in group_indices]             unique_coord = group         else:<ins>            if group.isnull().any():                # drop any NaN valued groups.                # also drop obj values where group was NaN                # Use where instead of reindex to account for duplicate coordinate labels.                obj = obj.where(group.notnull(), drop=True)                group = group.dropna(group_dim)</ins>             # look through group to find the unique values             unique_values, group_indices = unique_value_groups(                 safe_cast_to_index(group), sort=(bins is None)

groupby().apply() on variable with NaNs raises IndexError This results in an IndexError. This is mildly confusing, it took me a little while to figure out the NaN's were to blame. I'm guessing the NaN doesn't get filtered out everywhere. The traceback: My assumption was that it would throw out the values that fall within the NaN group, like: Out: <details</details


Question: assert_coordinate_consistent,     remap_label_indexers, )from .dataset import Dataset, merge_indexes, split_indexesfrom .dataset import Dataset, split_indexes from .formatting import format_item from .indexes import Indexes, default_indexes from .merge import PANDAS_TYPES

selection from MultiIndex does not work properly Should select the array <detailsINSTALLED VERSIONS commit: None python: 3.6.8 (default, Dec 30 2018, 01:22:34) [GCC 7.3.0] python-bits: 64 OS: Linux OS-release: 3.10.0-957.10.1.el7.x8664 machine: x8664 processor: x8664 byteorder: little LCALL: None LANG: enUS.UTF-8 LOCALE: enUS.UTF-8 libhdf5: 1.10.4 libnetcdf: 4.6.1 xarray: 0.14.0 pandas: 0.24.2 numpy: 1.15.4 scipy: 1.2.1 netCDF4: 1.4.2 pydap: None h5netcdf: None h5py: 2.9.0 Nio: None zarr: None cftime: 1.0.3.4 nctimeaxis: None PseudoNetCDF: None rasterio: None cfgrib: None iris: None bottleneck: 1.2.1 dask: None distributed: None matplotlib: 3.0.2 cartopy: None seaborn: 0.9.0 numbagg: None setuptools: 40.8.0 pip: 19.0.3 conda: None pytest: 5.0.0 IPython: 7.3.0 sphinx: None </detailsSorry for being quiet for a long time. I hope I could send a fix for this in a few days...


Question: --------         DataArray.reset_index         """        _check_inplace(inplace)        indexes = either_dict_or_kwargs(indexes, indexes_kwargs, "set_index")        coords, _ = merge_indexes(indexes, self._coords, set(), append=append)        return self._replace(coords=coords)        ds = self._to_temp_dataset().set_index(            indexes, append=append, inplace=inplace, **indexes_kwargs        )        return self._from_temp_dataset(ds)      def reset_index(         self,diff --git a/xarray/core/dataset.py b/xarray/core/dataset.py-- a/xarray/core/dataset.py++ b/xarray/core/dataset.py


Question: """     vars_to_replace: Dict[Hashable, Variable] = {}     vars_to_remove: List[Hashable] = []<ins>    dims_to_replace: Dict[Hashable, Hashable] = {}</ins>     error_msg = "{} is not the name of an existing variable."      for dim, var_names in indexes.items():


Question: if not len(names) and len(var_names) == 1:             idx = pd.Index(variables[var_names[0]].values)         else:        else:  # MultiIndex             for n in var_names:                 try:                     var = variables[n]


Question: levels.append(cat.categories)              idx = pd.MultiIndex(levels, codes, names=names)            for n in names:                dims_to_replace[n] = dim          vars_to_replace[dim] = IndexVariable(dim, idx)         vars_to_remove.extend(var_names)      new_variables = {k: v for k, v in variables.items() if k not in vars_to_remove}     new_variables.update(vars_to_replace)    # update dimensions if necessary  GH: 3512    for k, v in new_variables.items():        if any(d in dims_to_replace for d in v.dims):            new_dims = [dims_to_replace.get(d, d) for d in v.dims]            new_variables[k] = v._replace(dims=new_dims)     new_coord_names = coord_names | set(vars_to_replace)     new_coord_names -= set(vars_to_remove)     return new_variables, new_coord_names


Question: out = ops.fillna(self, value)         return out <ins>    def quantile(self, q, dim=None, interpolation="linear", keep_attrs=None):        """Compute the qth quantile over each array in the groups and        concatenate them together into a new array.        Parameters        ----------        q : float in range of [0,1] (or sequence of floats)            Quantile to compute, which must be between 0 and 1            inclusive.        dim : `...`, str or sequence of str, optional            Dimension(s) over which to apply quantile.            Defaults to the grouped dimension.        interpolation : {'linear', 'lower', 'higher', 'midpoint', 'nearest'}            This optional parameter specifies the interpolation method to            use when the desired quantile lies between two data points            ``i < j``:                * linear: ``i + (j - i) * fraction``, where ``fraction`` is                  the fractional part of the index surrounded by ``i`` and                  ``j``.                * lower: ``i``.                * higher: ``j``.                * nearest: ``i`` or ``j``, whichever is nearest.                * midpoint: ``(i + j) / 2``.        Returns        -------        quantiles : Variable            If `q` is a single quantile, then the result is a            scalar. If multiple percentiles are given, first axis of            the result corresponds to the quantile. In either case a            quantile dimension is added to the return array. The other            dimensions are the dimensions that remain after the            reduction of the array.        See Also        --------        numpy.nanpercentile, pandas.Series.quantile, Dataset.quantile,        DataArray.quantile        """        if dim is None:            dim = self._group_dim        out = self.map(            self._obj.__class__.quantile,            shortcut=False,            q=q,            dim=dim,            interpolation=interpolation,            keep_attrs=keep_attrs,        )        return out</ins>     def where(self, cond, other=dtypes.NA):         """Return elements from `self` or `other` depending on `cond`.

DatasetGroupBy does not implement quantile The docs claim works on grouped datasets, but that does not seem to be the case: this was found while trying to silence the nit-picky sphinx warnings in


Question: combined = self._maybe_unstack(combined)         return combined <del>    def quantile(self, q, dim=None, interpolation="linear", keep_attrs=None):        """Compute the qth quantile over each array in the groups and        concatenate them together into a new array.        Parameters        ----------        q : float in range of [0,1] (or sequence of floats)            Quantile to compute, which must be between 0 and 1            inclusive.        dim : `...`, str or sequence of str, optional            Dimension(s) over which to apply quantile.            Defaults to the grouped dimension.        interpolation : {'linear', 'lower', 'higher', 'midpoint', 'nearest'}            This optional parameter specifies the interpolation method to            use when the desired quantile lies between two data points            ``i < j``:                * linear: ``i + (j - i) * fraction``, where ``fraction`` is                  the fractional part of the index surrounded by ``i`` and                  ``j``.                * lower: ``i``.                * higher: ``j``.                * nearest: ``i`` or ``j``, whichever is nearest.                * midpoint: ``(i + j) / 2``.        Returns        -------        quantiles : Variable            If `q` is a single quantile, then the result            is a scalar. If multiple percentiles are given, first axis of            the result corresponds to the quantile and a quantile dimension            is added to the return array. The other dimensions are the            dimensions that remain after the reduction of the array.        See Also        --------        numpy.nanpercentile, pandas.Series.quantile, Dataset.quantile,        DataArray.quantile        """        if dim is None:            dim = self._group_dim        out = self.map(            self._obj.__class__.quantile,            shortcut=False,            q=q,            dim=dim,            interpolation=interpolation,            keep_attrs=keep_attrs,        )        if np.asarray(q, dtype=np.float64).ndim == 0:            out = out.drop_vars("quantile")        return out</del>     def reduce(         self, func, dim=None, axis=None, keep_attrs=None, shortcut=True, **kwargs     ):


Question: scalar = utils.is_scalar(q)         q = np.atleast_1d(np.asarray(q, dtype=np.float64)) <ins>        # TODO: remove once numpy >= 1.15.0 is the minimum requirement        if np.count_nonzero(q < 0.0) or np.count_nonzero(q > 1.0):            raise ValueError("Quantiles must be in the range [0, 1]")</ins>         if dim is None:             dim = self.dims

"ValueError: Percentiles must be in the range [0, 100]" <!-- In order for the maintainers to efficiently understand and prioritize issues, we ask you post a "Minimal, Complete and Verifiable Example" (MCVE): --By wrapping (xref: ) we also get the numpy error. However, the error message is wrong as xarray needs it to be in 0..1. BTW: thanks for , makes my life easier! Edit: uses internally. <detailsINSTALLED VERSIONS commit: None python: 3.7.3 (default, Jul 1 2019, 21:52:21) [GCC 7.3.0] python-bits: 64 OS: Linux OS-release: 4.12.14-lp151.28.36-default machine: x8664 processor: x8664 byteorder: little LCALL: None LANG: enGB.UTF-8 LOCALE: enUS.UTF-8 libhdf5: 1.10.5 libnetcdf: 4.7.1 xarray: 0.14.1+28.gf2b2f9f6 (current master) pandas: 0.25.2 numpy: 1.17.3 scipy: 1.3.1 netCDF4: 1.5.3 pydap: None h5netcdf: 0.7.4 h5py: 2.10.0 Nio: None zarr: None cftime: 1.0.4.2 nctime_axis: 1.2.0 PseudoNetCDF: None rasterio: 1.1.1 cfgrib: None iris: None bottleneck: 1.2.1 dask: 2.6.0 distributed: 2.6.0 matplotlib: 3.1.2 cartopy: 0.17.0 seaborn: 0.9.0 numbagg: None setuptools: 41.4.0 pip: 19.3.1 conda: None pytest: 5.2.2 IPython: 7.9.0 sphinx: 2.2.1 </details


Question: def _wrapper(npa, **kwargs):             # move quantile axis to end. required for apply_ufunc<ins>            # TODO: use np.nanquantile once numpy >= 1.15.0 is the minimum requirement</ins>             return np.moveaxis(np.nanpercentile(npa, **kwargs), 0, -1)          axis = np.arange(-1, -1 * len(dim) - 1, -1)


Question: those of the first object with that dimension. Indexes for the same           dimension must have the same size in all objects.     indexers, mode, concat_over : deprecated     Returns     -------     concatenated : type of objs     Notes    -----    Each concatenated Variable preserves corresponding ``attrs`` from the first element of ``objs``.     See also     --------     mergediff --git a/xarray/core/variable.py b/xarray/core/variable.py-- a/xarray/core/variable.py++ b/xarray/core/variable.py

Confusing error message when attribute not equal during concat If two or more s are concatentated and they have list attributes that are not equal an exception is raised about arrays not being truth values. I guess the expected result would be that the list attribute is not included in the resulting DataArray's attributes. <details</detailsconcat fails with attrs that are dictionaries with ndarrays Fails with the following error: This is a problem because the following code actually executes properly Equivalence for an array within a nested dictionary as an attribute is evaluated differently than an array attribute, which is non-intuitive. This bug is related to but is additionally pointing out a difference in evaluation for more complex attributes. The output of the code sample should concatenate successfully with the nested dictionary attribute, or a more easily interpretable error should be thrown telling me I'm dumb for using dictionaries in attributes. (See ) <detailsINSTALLED VERSIONS commit: None python: python-bits: 64 OS: Linux OS-release: 4.15.0-23-generic machine: x8664 processor: x8664 byteorder: little LCALL: None LANG: enUS.UTF-8 LOCALE: enUS.UTF-8 xarray: 0.10.9 pandas: 0.23.4 numpy: 1.15.3 scipy: 1.1.0 netCDF4: None h5netcdf: None h5py: None Nio: None zarr: None cftime: None PseudonetCDF: None rasterio: None iris: None bottleneck: None cyordereddict: None dask: None distributed: None matplotlib: 3.0.0 cartopy: None seaborn: None setuptools: 40.4.3 pip: 9.0.1 conda: None pytest: None IPython: 7.0.1 sphinx: None </detailsconcat fails with attrs that are dictionaries with ndarrays Fails with the following error: This is a problem because the following code actually executes properly Equivalence for an array within a nested dictionary as an attribute is evaluated differently than an array attribute, which is non-intuitive. This bug is related to but is additionally pointing out a difference in evaluation for more complex attributes. The output of the code sample should concatenate successfully with the nested dictionary attribute, or a more easily interpretable error should be thrown telling me I'm dumb for using dictionaries in attributes.


Question: those of the first object with that dimension. Indexes for the same           dimension must have the same size in all objects.     indexers, mode, concat_over : deprecated     Returns     -------     concatenated : type of objs     Notes    -----    Each concatenated Variable preserves corresponding ``attrs`` from the first element of ``objs``.     See also     --------     mergediff --git a/xarray/core/variable.py b/xarray/core/variable.py-- a/xarray/core/variable.py++ b/xarray/core/variable.py

(See ) <detailsINSTALLED VERSIONS commit: None python: python-bits: 64 OS: Linux OS-release: 4.15.0-23-generic machine: x8664 processor: x8664 byteorder: little LCALL: None LANG: enUS.UTF-8 LOCALE: enUS.UTF-8 xarray: 0.10.9 pandas: 0.23.4 numpy: 1.15.3 scipy: 1.1.0 netCDF4: None h5netcdf: None h5py: None Nio: None zarr: None cftime: None PseudonetCDF: None rasterio: None iris: None bottleneck: None cyordereddict: None dask: None distributed: None matplotlib: 3.0.0 cartopy: None seaborn: None setuptools: 40.4.3 pip: 9.0.1 conda: None pytest: None IPython: 7.0.1 sphinx: None </details


Question: if not shortcut:             for var in variables:                 if var.dims != first_var.dims:                    raise ValueError("inconsistent dimensions")                utils.remove_incompatible_items(attrs, var.attrs)                    raise ValueError(                        f"Variable has dimensions {list(var.dims)} but first Variable has dimensions {list(first_var.dims)}"                    )          return cls(dims, data, attrs, encoding)


Question: if not shortcut:             for var in variables:                 if var.dims != first_var.dims:                    raise ValueError("inconsistent dimensions")                utils.remove_incompatible_items(attrs, var.attrs)                    raise ValueError(                        f"Variable has dimensions {list(var.dims)} but first Variable has dimensions {list(first_var.dims)}"                    )          return cls(dims, data, attrs, encoding)


Question: return combined_ids, concat_dims  def _check_shape_tile_ids(combined_tile_ids):def _check_dimension_depth_tile_ids(combined_tile_ids):    """    Check all tuples are the same length, i.e. check that all lists are    nested to the same depth.    """     tile_ids = combined_tile_ids.keys()    # Check all tuples are the same length    # i.e. check that all lists are nested to the same depth     nesting_depths = [len(tile_id) for tile_id in tile_ids]     if not nesting_depths:         nesting_depths = [0]

combinebycoords should allow for missing panels in hypercube Currently, it throws the following error: This is b/c calls , which mandates that the passed datasets form a complete hypercube. This check functiono also serves the purpose of mandating that the dimension depths are the same. Could we pull that part out as a separate function and, for , only call this first part but NOT mandate that the hypercube is complete? The expected behavior, in my mind, should be to simply replace the missing tiles of the hypercube with . I'll file a PR to this effect and welcome comments. <detailsINSTALLED VERSIONS commit: None python: 3.7.3 (default, Dec 6 2019, 08:54:18) [GCC 7.3.0] python-bits: 64 OS: Linux OS-release: 4.14.150+ machine: x8664 processor: x8664 byteorder: little LCALL: enUS.UTF-8 LANG: enUS.UTF-8 LOCALE: enUS.UTF-8 libhdf5: 1.10.5 libnetcdf: 4.7.1 xarray: 0.14.1 pandas: 0.25.3 numpy: 1.17.3 scipy: 1.3.2 netCDF4: 1.5.3 pydap: None h5netcdf: 0.7.4 h5py: 2.10.0 Nio: None zarr: 2.3.2 cftime: 1.0.4.2 nctime_axis: 1.2.0 PseudoNetCDF: None rasterio: 1.1.1 cfgrib: None iris: 2.2.0 bottleneck: None dask: 2.8.1 distributed: 2.8.1 matplotlib: 3.1.2 cartopy: 0.17.0 seaborn: 0.9.0 numbagg: None setuptools: 42.0.2.post20191201 pip: 19.3.1 conda: 4.8.0 pytest: 5.3.1 IPython: 7.10.1 sphinx: 2.2.2 </details


Question: "The supplied objects do not form a hypercube because"             " sub-lists do not have consistent depths"         )    # return these just to be reused in _check_shape_tile_ids    return tile_ids, nesting_depths     # Check all lists along one dimension are same lengthdef _check_shape_tile_ids(combined_tile_ids):    """Check all lists along one dimension are same length."""    tile_ids, nesting_depths = _check_dimension_depth_tile_ids(combined_tile_ids)     for dim in range(nesting_depths[0]):         indices_along_dim = [tile_id[dim] for tile_id in tile_ids]         occurrences = Counter(indices_along_dim)


Question: coords : {'minimal', 'different', 'all' or list of str}, optional         As per the 'data_vars' kwarg, but for coordinate variables.     fill_value : scalar, optional        Value to use for newly missing values        Value to use for newly missing values. If None, raises a ValueError if        the passed Datasets do not create a complete hypercube.     join : {'outer', 'inner', 'left', 'right', 'exact'}, optional         String indicating how to combine differing indexes         (excluding concat_dim) in objects


Question: temperature    (y, x) float64 1.654 10.63 7.015 2.543 ... 12.46 2.22 15.96     precipitation  (y, x) float64 0.2136 0.9974 0.7603 ... 0.6125 0.4654 0.5953 <ins>    >>> xr.combine_by_coords([x1, x2, x3])    <xarray.Dataset>    Dimensions:        (x: 6, y: 4)    Coordinates:    * x              (x) int64 10 20 30 40 50 60    * y              (y) int64 0 1 2 3    Data variables:    temperature    (y, x) float64 1.654 10.63 7.015 nan ... 12.46 2.22 15.96    precipitation  (y, x) float64 0.2136 0.9974 0.7603 ... 0.6125 0.4654 0.5953</ins>     """      # Group by data vars


Question: list(datasets_with_same_vars)         )         _check_shape_tile_ids(combined_ids)        if fill_value is None:            # check that datasets form complete hypercube            _check_shape_tile_ids(combined_ids)        else:            # check only that all datasets have same dimension depth for these            # vars            _check_dimension_depth_tile_ids(combined_ids)          # Concatenate along all of concat_dims one by one to create single ds         concatenated = _combine_nd(


Question: If any variables conflict (see ``compat``).         """         _check_inplace(inplace)<ins>        other = other.to_dataset() if isinstance(other, xr.DataArray) else other</ins>         merge_result = dataset_merge_method(             self,             other,

Merging dataArray into dataset using dataset method fails While it's possible to merge a dataset and a dataarray object using the top-level function, if you try the same thing with the method it fails. Output:


Question: from .core.alignment import align, broadcast from .core.combine import auto_combine, combine_by_coords, combine_nested from .core.common import ALL_DIMS, full_like, ones_like, zeros_likefrom .core.computation import apply_ufunc, dot, wherefrom .core.computation import apply_ufunc, dot, polyval, where from .core.concat import concat from .core.dataarray import DataArray from .core.dataset import Dataset

Implement polyfit? Fitting a line (or curve) to data along a specified axis is a long-standing need of xarray users. There are many blog posts and SO questions about how to do it: The main use case in my domain is finding the temporal trend on a 3D variable (e.g. temperature in time, lon, lat). Yes, you can do it with applyufunc, but applyufunc is inaccessibly complex for many users. Much of our existing API could be removed and replaced with applyufunc calls, but that doesn't mean we should do it. I am proposing we add a Dataarray method called . It would work like this: Numpy's function is already vectorized in the sense that it accepts 1D x and 2D y, performing the fit independently over each column of y. To extend this to ND, we would just need to reshape the data going in and out of the function. We do this already in . For dask, we could simply require that the dimension over which the fit is calculated be contiguous, and then call mapblocks. Thoughts?


Question: "open_mfdataset",     "open_rasterio",     "open_zarr",    "polyval",     "register_dataarray_accessor",     "register_dataset_accessor",     "save_mfdataset",diff --git a/xarray/core/computation.py b/xarray/core/computation.py-- a/xarray/core/computation.py++ b/xarray/core/computation.py


Question: dataset_join="exact",         dask="allowed",     )def polyval(coord, coeffs, degree_dim="degree"):    """Evaluate a polynomial at specific values    Parameters    ----------    coord : DataArray        The 1D coordinate along which to evaluate the polynomial.    coeffs : DataArray        Coefficients of the polynomials.    degree_dim : str, default "degree"        Name of the polynomial degree dimension in `coeffs`.    See also    --------    xarray.DataArray.polyfit    numpy.polyval    """    from .dataarray import DataArray    from .missing import get_clean_interp_index    x = get_clean_interp_index(coord, coord.name)    deg_coord = coeffs[degree_dim]    lhs = DataArray(        np.vander(x, int(deg_coord.max()) + 1),        dims=(coord.name, degree_dim),        coords={coord.name: coord, degree_dim: np.arange(deg_coord.max() + 1)[::-1]},    )    return (lhs * coeffs).sum(degree_dim)diff --git a/xarray/core/dask_array_ops.py b/xarray/core/dask_array_ops.py-- a/xarray/core/dask_array_ops.py++ b/xarray/core/dask_array_ops.py


Question: # crop boundary.     index = (slice(None),) * axis + (slice(drop_size, drop_size + orig_shape[axis]),)     return out[index]def least_squares(lhs, rhs, rcond=None, skipna=False):    import dask.array as da    lhs_da = da.from_array(lhs, chunks=(rhs.chunks[0], lhs.shape[1]))    if skipna:        added_dim = rhs.ndim == 1        if added_dim:            rhs = rhs.reshape(rhs.shape[0], 1)        results = da.apply_along_axis(            nputils._nanpolyfit_1d,            0,            rhs,            lhs_da,            dtype=float,            shape=(lhs.shape[1] + 1,),            rcond=rcond,        )        coeffs = results[:-1, ...]        residuals = results[-1, ...]        if added_dim:            coeffs = coeffs.reshape(coeffs.shape[0])            residuals = residuals.reshape(residuals.shape[0])    else:        coeffs, residuals, _, _ = da.linalg.lstsq(lhs_da, rhs)    return coeffs, residualsdiff --git a/xarray/core/dataarray.py b/xarray/core/dataarray.py-- a/xarray/core/dataarray.py++ b/xarray/core/dataarray.py


Question: return map_blocks(func, self, args, kwargs)     def polyfit(        self,        dim: Hashable,        deg: int,        skipna: bool = None,        rcond: float = None,        w: Union[Hashable, Any] = None,        full: bool = False,        cov: bool = False,    ):        """        Least squares polynomial fit.        This replicates the behaviour of `numpy.polyfit` but differs by skipping        invalid values when `skipna = True`.        Parameters        ----------        dim : hashable            Coordinate along which to fit the polynomials.        deg : int            Degree of the fitting polynomial.        skipna : bool, optional            If True, removes all invalid values before fitting each 1D slices of the array.            Default is True if data is stored in a dask.array or if there is any            invalid values, False otherwise.        rcond : float, optional            Relative condition number to the fit.        w : Union[Hashable, Any], optional            Weights to apply to the y-coordinate of the sample points.            Can be an array-like object or the name of a coordinate in the dataset.        full : bool, optional            Whether to return the residuals, matrix rank and singular values in addition            to the coefficients.        cov : Union[bool, str], optional            Whether to return to the covariance matrix in addition to the coefficients.            The matrix is not scaled if `cov='unscaled'`.        Returns        -------        polyfit_results : Dataset            A single dataset which contains:            polyfit_coefficients                The coefficients of the best fit.            polyfit_residuals                The residuals of the least-square computation (only included if `full=True`)            [dim]_matrix_rank                The effective rank of the scaled Vandermonde coefficient matrix (only included if `full=True`)            [dim]_singular_value                The singular values of the scaled Vandermonde coefficient matrix (only included if `full=True`)            polyfit_covariance                The covariance matrix of the polynomial coefficient estimates (only included if `full=False` and `cov=True`)        See also        --------        numpy.polyfit        """        return self._to_temp_dataset().polyfit(            dim, deg, skipna=skipna, rcond=rcond, w=w, full=full, cov=cov        )     def pad(         self,         pad_width: Mapping[Hashable, Union[int, Tuple[int, int]]] = None,diff --git a/xarray/core/dataset.py b/xarray/core/dataset.py-- a/xarray/core/dataset.py++ b/xarray/core/dataset.py


Question: merge_coordinates_without_align,     merge_data_and_coords, )<ins>from .missing import get_clean_interp_index</ins> from .options import OPTIONS, _get_keep_attrs from .pycompat import dask_array_type from .utils import (


Question: return map_blocks(func, self, args, kwargs)     def polyfit(        self,        dim: Hashable,        deg: int,        skipna: bool = None,        rcond: float = None,        w: Union[Hashable, Any] = None,        full: bool = False,        cov: Union[bool, str] = False,    ):        """        Least squares polynomial fit.        This replicates the behaviour of `numpy.polyfit` but differs by skipping        invalid values when `skipna = True`.        Parameters        ----------        dim : hashable            Coordinate along which to fit the polynomials.        deg : int            Degree of the fitting polynomial.        skipna : bool, optional            If True, removes all invalid values before fitting each 1D slices of the array.            Default is True if data is stored in a dask.array or if there is any            invalid values, False otherwise.        rcond : float, optional            Relative condition number to the fit.        w : Union[Hashable, Any], optional            Weights to apply to the y-coordinate of the sample points.            Can be an array-like object or the name of a coordinate in the dataset.        full : bool, optional            Whether to return the residuals, matrix rank and singular values in addition            to the coefficients.        cov : Union[bool, str], optional            Whether to return to the covariance matrix in addition to the coefficients.            The matrix is not scaled if `cov='unscaled'`.        Returns        -------        polyfit_results : Dataset            A single dataset which contains (for each "var" in the input dataset):            [var]_polyfit_coefficients                The coefficients of the best fit for each variable in this dataset.            [var]_polyfit_residuals                The residuals of the least-square computation for each variable (only included if `full=True`)            [dim]_matrix_rank                The effective rank of the scaled Vandermonde coefficient matrix (only included if `full=True`)            [dim]_singular_values                The singular values of the scaled Vandermonde coefficient matrix (only included if `full=True`)            [var]_polyfit_covariance                The covariance matrix of the polynomial coefficient estimates (only included if `full=False` and `cov=True`)        See also        --------        numpy.polyfit        """        variables = {}        skipna_da = skipna        x = get_clean_interp_index(self, dim)        xname = "{}_".format(self[dim].name)        order = int(deg) + 1        lhs = np.vander(x, order)        if rcond is None:            rcond = x.shape[0] * np.core.finfo(x.dtype).eps        # Weights:        if w is not None:            if isinstance(w, Hashable):                w = self.coords[w]            w = np.asarray(w)            if w.ndim != 1:                raise TypeError("Expected a 1-d array for weights.")            if w.shape[0] != lhs.shape[0]:                raise TypeError("Expected w and {} to have the same length".format(dim))            lhs *= w[:, np.newaxis]        # Scaling        scale = np.sqrt((lhs * lhs).sum(axis=0))        lhs /= scale        degree_dim = utils.get_temp_dimname(self.dims, "degree")        rank = np.linalg.matrix_rank(lhs)        if rank != order and not full:            warnings.warn(                "Polyfit may be poorly conditioned", np.RankWarning, stacklevel=4            )        if full:            rank = xr.DataArray(rank, name=xname + "matrix_rank")            variables[rank.name] = rank            sing = np.linalg.svd(lhs, compute_uv=False)            sing = xr.DataArray(                sing,                dims=(degree_dim,),                coords={degree_dim: np.arange(order)[::-1]},                name=xname + "singular_values",            )            variables[sing.name] = sing        for name, da in self.data_vars.items():            if dim not in da.dims:                continue            if skipna is None:                if isinstance(da.data, dask_array_type):                    skipna_da = True                else:                    skipna_da = np.any(da.isnull())            dims_to_stack = [dimname for dimname in da.dims if dimname != dim]            stacked_coords = {}            if dims_to_stack:                stacked_dim = utils.get_temp_dimname(dims_to_stack, "stacked")                rhs = da.transpose(dim, *dims_to_stack).stack(                    {stacked_dim: dims_to_stack}                )                stacked_coords = {stacked_dim: rhs[stacked_dim]}                scale_da = scale[:, np.newaxis]            else:                rhs = da                scale_da = scale            if w is not None:                rhs *= w[:, np.newaxis]            coeffs, residuals = duck_array_ops.least_squares(                lhs, rhs.data, rcond=rcond, skipna=skipna_da            )            if isinstance(name, str):                name = "{}_".format(name)            else:                # Thus a ReprObject => polyfit was called on a DataArray                name = ""            coeffs = xr.DataArray(                coeffs / scale_da,                dims=[degree_dim] + list(stacked_coords.keys()),                coords={degree_dim: np.arange(order)[::-1], **stacked_coords},                name=name + "polyfit_coefficients",            )            if dims_to_stack:                coeffs = coeffs.unstack(stacked_dim)            variables[coeffs.name] = coeffs            if full or (cov is True):                residuals = xr.DataArray(                    residuals if dims_to_stack else residuals.squeeze(),                    dims=list(stacked_coords.keys()),                    coords=stacked_coords,                    name=name + "polyfit_residuals",                )                if dims_to_stack:                    residuals = residuals.unstack(stacked_dim)                variables[residuals.name] = residuals            if cov:                Vbase = np.linalg.inv(np.dot(lhs.T, lhs))                Vbase /= np.outer(scale, scale)                if cov == "unscaled":                    fac = 1                else:                    if x.shape[0] <= order:                        raise ValueError(                            "The number of data points must exceed order to scale the covariance matrix."                        )                    fac = residuals / (x.shape[0] - order)                covariance = xr.DataArray(Vbase, dims=("cov_i", "cov_j"),) * fac                variables[name + "polyfit_covariance"] = covariance        return Dataset(data_vars=variables, attrs=self.attrs.copy())     def pad(         self,         pad_width: Mapping[Hashable, Union[int, Tuple[int, int]]] = None,diff --git a/xarray/core/duck_array_ops.py b/xarray/core/duck_array_ops.py-- a/xarray/core/duck_array_ops.py++ b/xarray/core/duck_array_ops.py


Question: return dask_array_ops.rolling_window(array, axis, window, center, fill_value)     else:  # np.ndarray         return nputils.rolling_window(array, axis, window, center, fill_value)def least_squares(lhs, rhs, rcond=None, skipna=False):    """Return the coefficients and residuals of a least-squares fit.    """    if isinstance(rhs, dask_array_type):        return dask_array_ops.least_squares(lhs, rhs, rcond=rcond, skipna=skipna)    else:        return nputils.least_squares(lhs, rhs, rcond=rcond, skipna=skipna)diff --git a/xarray/core/nputils.py b/xarray/core/nputils.py-- a/xarray/core/nputils.py++ b/xarray/core/nputils.py


Question: return f  <ins>def _nanpolyfit_1d(arr, x, rcond=None):    out = np.full((x.shape[1] + 1,), np.nan)    mask = np.isnan(arr)    if not np.all(mask):        out[:-1], out[-1], _, _ = np.linalg.lstsq(x[~mask, :], arr[~mask], rcond=rcond)    return outdef least_squares(lhs, rhs, rcond=None, skipna=False):    if skipna:        added_dim = rhs.ndim == 1        if added_dim:            rhs = rhs.reshape(rhs.shape[0], 1)        nan_cols = np.any(np.isnan(rhs), axis=0)        out = np.empty((lhs.shape[1] + 1, rhs.shape[1]))        if np.any(nan_cols):            out[:, nan_cols] = np.apply_along_axis(                _nanpolyfit_1d, 0, rhs[:, nan_cols], lhs            )        if np.any(~nan_cols):            out[:-1, ~nan_cols], out[-1, ~nan_cols], _, _ = np.linalg.lstsq(                lhs, rhs[:, ~nan_cols], rcond=rcond            )        coeffs = out[:-1, :]        residuals = out[-1, :]        if added_dim:            coeffs = coeffs.reshape(coeffs.shape[0])            residuals = residuals.reshape(residuals.shape[0])    else:        coeffs, residuals, _, _ = np.linalg.lstsq(lhs, rhs, rcond=rcond)    return coeffs, residuals</ins> nanmin = _create_bottleneck_method("nanmin") nanmax = _create_bottleneck_method("nanmax") nanmean = _create_bottleneck_method("nanmean")


Question: CMAP_SEQUENTIAL: "viridis",     CMAP_DIVERGENT: "RdBu_r",     KEEP_ATTRS: "default",    DISPLAY_STYLE: "text",    DISPLAY_STYLE: "html", }  _JOIN_OPTIONS = frozenset(["inner", "outer", "left", "right", "exact"])

Turn on reprhtml by default? I just wanted to open this to discuss turning the reprhtml on by default. This PR it as a style option, but I suspect that more people will use if it is on by default. Does that seem like a reasonable change?


Question: import contextlib import functools from datetime import datetime, timedeltafrom itertools import zip_longestfrom itertools import chain, zip_longest from typing import Hashable  import numpy as np

Truncate array repr based on line count <!-- In order for the maintainers to efficiently understand and prioritize issues, we ask you post a "Minimal, Complete and Verifiable Example" (MCVE): --I thought we might have had an issue (and maybe solved it?) but couldn't find it anywhere. Forgive me if I'm duplicating. With larger arrays, it's much more reasonable: <!-- this should explain why the current behavior is a problem and why the expected output is a better solution --Something like 40 lines is probably a reasonable place to truncate? <detailsINSTALLED VERSIONS commit: None python: 3.7.3 (default, Jul 1 2019, 21:52:21) [GCC 7.3.0] python-bits: 64 OS: Linux OS-release: [...] machine: x8664 processor: x8664 byteorder: little LCALL: None LANG: enUS.utf8 LOCALE: enUS.UTF-8 libhdf5: 1.10.5 libnetcdf: 4.7.1 xarray: 0.15.0 pandas: 0.25.3 numpy: 1.17.3 scipy: 1.3.2 netCDF4: 1.5.3 pydap: None h5netcdf: 0.7.4 h5py: 2.10.0 Nio: None zarr: None cftime: 1.0.4.2 nctime_axis: None PseudoNetCDF: None rasterio: None cfgrib: None iris: None bottleneck: 1.2.1 dask: 2.7.0 distributed: 2.7.0 matplotlib: 3.1.2 cartopy: None seaborn: 0.9.0 numbagg: None setuptools: 41.6.0.post20191101 pip: 19.3.1 conda: None pytest: 5.2.2 IPython: 7.9.0 sphinx: 2.2.1 </details


Question: np.set_printoptions(**original)  <ins>def limit_lines(string: str, *, limit: int):    """    If the string is more lines than the limit,    this returns the middle lines replaced by an ellipsis    """    lines = string.splitlines()    if len(lines) > limit:        string = "\n".join(chain(lines[: limit // 2], ["..."], lines[-limit // 2 :]))    return string</ins> def short_numpy_repr(array):     array = np.asarray(array)


Question: elif hasattr(internal_data, "__array_function__") or isinstance(         internal_data, dask_array_type     ):        return repr(array.data)        return limit_lines(repr(array.data), limit=40)     elif array._in_memory or array.size < 1e5:         return short_numpy_repr(array)     else:


Question: from .formatting import format_item from .indexes import Indexes, default_indexes, propagate_indexes from .indexing import is_fancy_indexerfrom .merge import PANDAS_TYPES, _extract_indexes_from_coordsfrom .merge import PANDAS_TYPES, MergeError, _extract_indexes_from_coords from .options import OPTIONS from .utils import Default, ReprObject, _check_inplace, _default, either_dict_or_kwargs from .variable import (

In-place addition of arrays with the same coords but in a different order I have two DataArrays with the same dimension, but the index is in a different order. Adding them with works fine, but the in-place addition fails. is working fine. I would expect to do the same. The in-place addition fails: <details<summaryOutput of </summaryINSTALLED VERSIONS commit: None python: 3.8.2 (default, Mar 26 2020, 15:53:00) [GCC 7.3.0] python-bits: 64 OS: Linux OS-release: 4.19.112-1-MANJARO machine: x8664 processor: byteorder: little LCALL: None LANG: frFR.UTF-8 LOCALE: frFR.UTF-8 libhdf5: None libnetcdf: None xarray: 0.15.0 pandas: 1.0.3 numpy: 1.18.1 scipy: None netCDF4: None pydap: None h5netcdf: None h5py: None Nio: None zarr: None cftime: None nctime_axis: None PseudoNetCDF: None rasterio: None cfgrib: None iris: None bottleneck: None dask: None distributed: None matplotlib: None cartopy: None seaborn: None numbagg: None setuptools: 46.1.1.post20200323 pip: 20.0.2 conda: None pytest: None IPython: None sphinx: None </details


Question: # don't support automatic alignment with in-place arithmetic.             other_coords = getattr(other, "coords", None)             other_variable = getattr(other, "variable", other)            with self.coords._merge_inplace(other_coords):                f(self.variable, other_variable)            try:                with self.coords._merge_inplace(other_coords):                    f(self.variable, other_variable)            except MergeError as exc:                raise MergeError(                    "Automatic alignment is not supported for in-place operations.\n"                    "Consider aligning the indices manually or using a not-in-place operation.\n"                    "See https://github.com/pydata/xarray/issues/3910 for more explanations."                ) from exc             return self          return func


Question: from .options import OPTIONS, _get_keep_attrs from .pycompat import dask_array_type from .rolling_exp import RollingExpfrom .utils import Frozen, either_dict_or_kwargsfrom .utils import Frozen, either_dict_or_kwargs, is_scalar  # Used as a sentinel value to indicate a all dimensions ALL_DIMS = ...

xr.fulllike (often) fails when other is chunked and fillvalue is non-scalar I've been running into some issues when using , when my is a chunked dask array, and the is a numpy array. Now, I just checked, mentions only scalar in the signature. However, this is a very convenient way to get all the coordinates and dimensions attached to an array like this, so it feels like desirable functionality. And as I mention below, both numpy and dask function similary, taking much more than just scalars. This results in an error: Expected is a DataArray with the dimensions and coords of , and the numpy array of as its data. The issue lies here: Calling with the given number of chunks results in it trying to to apply the for every individual chunk. As one would expect, if I set to the size of a single chunk it doesn't error: It does fail on a similarly chunked dask array (since it's applying it for every chunk): The most obvious solution would be to force it down the route, since all the values already exist in memory anyway. So maybe another type check does the trick. However, accepts quite a variety of arguments for the fill value (scalars, numpy arrays, lists, tuples, ranges). The dask docs mention only a scalar in the signature for: As does However, in all cases, they still broadcast automatically... So kind of undefined behavior of a blocked ?