Question: Transport adapters don't work as advertised Having: We cannot currently reliably do:: We should, this is documented: Instead, because of the following might or might not work as expected: This is because there is already a default adapter for in the form of . Depending on the (seemingly random) order of keys in the dictionary, for some combinations of keys it will work, for others it won't. There are a couple. As usual, a compromise between backwards compatibility, clarity and performance. Change so that it sorts the keys every time it fetches an adapter. The advantage is that is still a vanilla dictionary, if you consider it a public API. Something like Django's which would sort the keys on insertion. The advantage is that is still a mapping and now is faster. The disadvantage is the maintenance cost of a custom data structure. Also, mild backwards incompatibility if someone somewhere used the fact that is exactly of the type (unlikely, isn't it?). Having and as the public API for adapters, we might just as well implement the sorting while mounting and just maintain as a list of tuples. The advantage is that it's the fastest solution with no additional maintenance burden. The disadvantage is backwards incompatibility. This is another design issue. It appears that sorting first from the longest key to shortest, and then alphabetically (for keys of the same length), is the way. What should we do about it? By the way, this test adapter in the example above .

v.close()      def mount(self, prefix, adapter):        """Registers a connection adapter to a prefix."""        """Registers a connection adapter to a prefix.        Adapters are sorted in descending order by key length."""         self.adapters[prefix] = adapter        keys_to_move = [k for k in self.adapters if len(k) < len(prefix)]        for key in keys_to_move:            self.adapters[key] = self.adapters.pop(key)      def __getstate__(self):         return dict((attr, getattr(self, attr, None)) for attr in self.__attrs__)


Question: CaseInsensitiveDict setitem faulty for differing-case keys I think I figured out the cause of the issue described here: It appears that if one isn't careful to use a common case convention when assigning a header after it has been assigned before, the header will fail to work. The request will use the old header value... I think I know why, too. Here's an example: The output: Notice how in both passes, only the first assignment actually "works". It stores the data for all three assignments, but will only return the value set by the initial key. I believe the only change necessary would be to the setitem method on CaseInsensitiveDict: Broken: Fixed: When I use the corrected module, the CaseInsensitiveDict behaves as I believe it should: Note that the value for that key is the last one assigned, instead of the first. If needed, I can put together a pull request for this, but it might take some time since I've never done it before :)

"""  import os<ins>import collections</ins> from itertools import islice


Question: CaseInsensitiveDict setitem faulty for differing-case keys I think I figured out the cause of the issue described here: It appears that if one isn't careful to use a common case convention when assigning a header after it has been assigned before, the header will fail to work. The request will use the old header value... I think I know why, too. Here's an example: The output: Notice how in both passes, only the first assignment actually "works". It stores the data for all three assignments, but will only return the value set by the initial key. I believe the only change necessary would be to the setitem method on CaseInsensitiveDict: Broken: Fixed: When I use the corrected module, the CaseInsensitiveDict behaves as I believe it should: Note that the value for that key is the last one assigned, instead of the first. If needed, I can put together a pull request for this, but it might take some time since I've never done it before :)

return "".join(islice(self.i, None, n))  class CaseInsensitiveDict(dict):    """Case-insensitive Dictionaryclass CaseInsensitiveDict(collections.MutableMapping):    """    A case-insensitive ``dict``-like object.    Implements all methods and operations of    ``collections.MutableMapping`` as well as dict's ``copy``. Also    provides ``lower_items``.    All keys are expected to be strings. The structure remembers the    case of the last key to be set, and ``iter(instance)``,    ``keys()``, ``items()``, ``iterkeys()``, and ``iteritems()``    will contain case-sensitive keys. However, querying and contains    testing is case insensitive:        cid = CaseInsensitiveDict()        cid['Accept'] = 'application/json'        cid['aCCEPT'] == 'application/json'  # True        list(cid) == ['Accept']  # True      For example, ``headers['content-encoding']`` will return the    value of a ``'Content-Encoding'`` response header."""    value of a ``'Content-Encoding'`` response header, regardless    of how the header name was originally stored.     @property    def lower_keys(self):        if not hasattr(self, '_lower_keys') or not self._lower_keys:            self._lower_keys = dict((k.lower(), k) for k in list(self.keys()))        return self._lower_keys    If the constructor, ``.update``, or equality comparison    operations are given keys that have equal ``.lower()``s, the    behavior is undefined.     def _clear_lower_keys(self):        if hasattr(self, '_lower_keys'):            self._lower_keys.clear()    """    def __init__(self, data=None, **kwargs):        self._store = dict()        if data is None:            data = {}        self.update(data, **kwargs)      def __setitem__(self, key, value):        dict.__setitem__(self, key, value)        self._clear_lower_keys()        # Use the lowercased key for lookups, but store the actual        # key alongside the value.        self._store[key.lower()] = (key, value)     def __delitem__(self, key):        dict.__delitem__(self, self.lower_keys.get(key.lower(), key))        self._lower_keys.clear()    def __getitem__(self, key):        return self._store[key.lower()][1]     def __contains__(self, key):        return key.lower() in self.lower_keys    def __delitem__(self, key):        del self._store[key.lower()]     def __getitem__(self, key):        # We allow fall-through here, so values default to None        if key in self:            return dict.__getitem__(self, self.lower_keys[key.lower()])    def __iter__(self):        return (casedkey for casedkey, mappedvalue in self._store.values())     def get(self, key, default=None):        if key in self:            return self[key]    def __len__(self):        return len(self._store)    def lower_items(self):        """Like iteritems(), but with all lowercase keys."""        return (            (lowerkey, keyval[1])            for (lowerkey, keyval)            in self._store.items()        )    def __eq__(self, other):        if isinstance(other, collections.Mapping):            other = CaseInsensitiveDict(other)         else:            return default            return NotImplemented        # Compare insensitively        return dict(self.lower_items()) == dict(other.lower_items())    # Copy is required    def copy(self):         return CaseInsensitiveDict(self._store.values())    def __repr__(self):        return '%s(%r)' % (self.__class__.__name__, dict(self.items()))   class LookupDict(dict):diff --git a/requests/utils.py b/requests/utils.py-- a/requests/utils.py++ b/requests/utils.py


Question: CaseInsensitiveDict setitem faulty for differing-case keys I think I figured out the cause of the issue described here: It appears that if one isn't careful to use a common case convention when assigning a header after it has been assigned before, the header will fail to work. The request will use the old header value... I think I know why, too. Here's an example: The output: Notice how in both passes, only the first assignment actually "works". It stores the data for all three assignments, but will only return the value set by the initial key. I believe the only change necessary would be to the setitem method on CaseInsensitiveDict: Broken: Fixed: When I use the corrected module, the CaseInsensitiveDict behaves as I believe it should: Note that the value for that key is the last one assigned, instead of the first. If needed, I can put together a pull request for this, but it might take some time since I've never done it before :)

from .compat import parse_http_list as _parse_list_header from .compat import quote, urlparse, bytes, str, OrderedDict, urlunparse from .cookies import RequestsCookieJar, cookiejar_from_dict<ins>from .structures import CaseInsensitiveDict</ins>  _hush_pyflakes = (RequestsCookieJar,)


Question: CaseInsensitiveDict setitem faulty for differing-case keys I think I figured out the cause of the issue described here: It appears that if one isn't careful to use a common case convention when assigning a header after it has been assigned before, the header will fail to work. The request will use the old header value... I think I know why, too. Here's an example: The output: Notice how in both passes, only the first assignment actually "works". It stores the data for all three assignments, but will only return the value set by the initial key. I believe the only change necessary would be to the setitem method on CaseInsensitiveDict: Broken: Fixed: When I use the corrected module, the CaseInsensitiveDict behaves as I believe it should: Note that the value for that key is the last one assigned, instead of the first. If needed, I can put together a pull request for this, but it might take some time since I've never done it before :)

def default_headers():    return {    return CaseInsensitiveDict({         'User-Agent': default_user_agent(),         'Accept-Encoding': ', '.join(('gzip', 'deflate', 'compress')),         'Accept': '*/*'    }    })   def parse_header_links(value):


Question: multipart/form-data and datetime data I raise an bug that you already fix in the past on this issue : or I tried the same methodology with that code : With the 1.2.0 version, no error is raised. With 1.2.3 version, I have that traceback : My original problem was with a python datetime in the data dict Thanks,

val = [val]             for v in val:                 if v is not None:<ins>                    # Don't call str() on bytestrings: in Py3 it all goes wrong.                    if not isinstance(v, bytes):                        v = str(v)</ins>                     new_fields.append(                         (field.decode('utf-8') if isinstance(field, bytes) else field,                          v.encode('utf-8') if isinstance(v, str) else v))


Question: Cookies not persisted when set via functional API. Cookies set as part of a call to (or any of the top level methods that call it) are not persisted, including on redirects. Expected behaviour: Actual behaviour: And, a super extra bonus bug: even though we definitely sent the cookie on the first request.

return c  def cookiejar_from_dict(cookie_dict, cookiejar=None):def cookiejar_from_dict(cookie_dict, cookiejar=None, overwrite=True):     """Returns a CookieJar from a key/value dictionary.      :param cookie_dict: Dict of key/values to insert into CookieJar.    :param cookiejar: (optional) A cookiejar to add the cookies to.    :param overwrite: (optional) If False, will not replace cookies        already in the jar with new ones.     """     if cookiejar is None:         cookiejar = RequestsCookieJar()      if cookie_dict is not None:        names_from_jar = [cookie.name for cookie in cookiejar]         for name in cookie_dict:            cookiejar.set_cookie(create_cookie(name, cookie_dict[name]))            if overwrite or (name not in names_from_jar):                cookiejar.set_cookie(create_cookie(name, cookie_dict[name]))     return cookiejardiff --git a/requests/models.py b/requests/models.py-- a/requests/models.py++ b/requests/models.py


Question: Cookies not persisted when set via functional API. Cookies set as part of a call to (or any of the top level methods that call it) are not persisted, including on redirects. Expected behaviour: Actual behaviour: And, a super extra bonus bug: even though we definitely sent the cookie on the first request.

p = PreparedRequest()         p.method = self.method         p.url = self.url        p.headers = self.headers        p.headers = self.headers.copy()         p.body = self.body         p.hooks = self.hooks         return pdiff --git a/requests/sessions.py b/requests/sessions.py-- a/requests/sessions.py++ b/requests/sessions.py


Question: Cookies not persisted when set via functional API. Cookies set as part of a call to (or any of the top level methods that call it) are not persisted, including on redirects. Expected behaviour: Actual behaviour: And, a super extra bonus bug: even though we definitely sent the cookie on the first request.

)         prep = self.prepare_request(req) <ins>        # Add param cookies to session cookies        self.cookies = cookiejar_from_dict(cookies, cookiejar=self.cookies, overwrite=False)</ins>         proxies = proxies or {}          # Gather clues from the surrounding environment.


Question: Session hooks broken Request hooks are being with session hooks; since both hook dicts have a list as the value, one simply overwrites the other.

return merged_setting  <ins>def merge_hooks(request_hooks, session_hooks, dict_class=OrderedDict):    """    Properly merges both requests and session hooks.    This is necessary because when request_hooks == {'response': []}, the    merge breaks Session hooks entirely.    """    if session_hooks is None or session_hooks.get('response') == []:        return request_hooks    if request_hooks is None or request_hooks.get('response') == []:        return session_hooks    return merge_setting(request_hooks, session_hooks, dict_class)</ins> class SessionRedirectMixin(object):     def resolve_redirects(self, resp, req, stream=False, timeout=None,                           verify=True, cert=None, proxies=None):


Question: Session hooks broken Request hooks are being with session hooks; since both hook dicts have a list as the value, one simply overwrites the other.

params=merge_setting(request.params, self.params),             auth=merge_setting(auth, self.auth),             cookies=merged_cookies,            hooks=merge_setting(request.hooks, self.hooks),            hooks=merge_hooks(request.hooks, self.hooks),         )         return p


Question: Problem POST'ing png file because of UnicodeError Here is the code I'm using: This raises the error: This problem is caused by the fact that the content-length header is actually a unicode object. When the actual body of the request is being constructed, python attempts to coerce the entire request into unicode resulting in the decode error. After tracing it, the cause is the following lines: where is declared in

raise NotImplementedError('Streamed bodies and files are mutually exclusive.')              if length is not None:                self.headers['Content-Length'] = str(length)                self.headers['Content-Length'] = builtin_str(length)             else:                 self.headers['Transfer-Encoding'] = 'chunked'         else:


Question: Problem POST'ing png file because of UnicodeError Here is the code I'm using: This raises the error: This problem is caused by the fact that the content-length header is actually a unicode object. When the actual body of the request is being constructed, python attempts to coerce the entire request into unicode resulting in the decode error. After tracing it, the cause is the following lines: where is declared in

def prepare_content_length(self, body):         if hasattr(body, 'seek') and hasattr(body, 'tell'):             body.seek(0, 2)            self.headers['Content-Length'] = str(body.tell())            self.headers['Content-Length'] = builtin_str(body.tell())             body.seek(0, 0)         elif body is not None:             l = super_len(body)             if l:                self.headers['Content-Length'] = str(l)                self.headers['Content-Length'] = builtin_str(l)         elif self.method not in ('GET', 'HEAD'):             self.headers['Content-Length'] = '0'


Question: Regression 2.0.1: Using MozillaCookieJar does not work Could not find an issue raised for this, not sure if this was an expected change either. This is reproducible on master. Existing code fails on update to . The cause seems to be triggered by the change at The parameter expects either or . Treating as a dict triggers the error in this instance. The following code highlights the issue: The following is thrown:

cookiejar.set_cookie(create_cookie(name, cookie_dict[name]))      return cookiejardef merge_cookies(cookiejar, cookies):    """Add cookies to cookiejar and returns a merged CookieJar.    :param cookiejar: CookieJar object to add the cookies to.    :param cookies: Dictionary or CookieJar object to be added.    """    if not isinstance(cookiejar, cookielib.CookieJar):        raise ValueError('You can only merge into CookieJar')        if isinstance(cookies, dict):        cookiejar = cookiejar_from_dict(            cookies, cookiejar=cookiejar, overwrite=False)    elif isinstance(cookies, cookielib.CookieJar):        try:            cookiejar.update(cookies)        except AttributeError:            for cookie_in_jar in cookies:                cookiejar.set_cookie(cookie_in_jar)    return cookiejardiff --git a/requests/sessions.py b/requests/sessions.py-- a/requests/sessions.py++ b/requests/sessions.py


Question: Regression 2.0.1: Using MozillaCookieJar does not work Could not find an issue raised for this, not sure if this was an expected change either. This is reproducible on master. Existing code fails on update to . The cause seems to be triggered by the change at The parameter expects either or . Treating as a dict triggers the error in this instance. The following code highlights the issue: The following is thrown:

from datetime import datetime  from .compat import cookielib, OrderedDict, urljoin, urlparse, builtin_strfrom .cookies import cookiejar_from_dict, extract_cookies_to_jar, RequestsCookieJarfrom .cookies import (    cookiejar_from_dict, extract_cookies_to_jar, RequestsCookieJar, merge_cookies) from .models import Request, PreparedRequest from .hooks import default_hooks, dispatch_hook from .utils import to_key_val_list, default_headers


Question: Regression 2.0.1: Using MozillaCookieJar does not work Could not find an issue raised for this, not sure if this was an expected change either. This is reproducible on master. Existing code fails on update to . The cause seems to be triggered by the change at The parameter expects either or . Treating as a dict triggers the error in this instance. The following code highlights the issue: The following is thrown:

cookies = cookiejar_from_dict(cookies)          # Merge with session cookies        merged_cookies = RequestsCookieJar()        merged_cookies.update(self.cookies)        merged_cookies.update(cookies)        merged_cookies = merge_cookies(            merge_cookies(RequestsCookieJar(), self.cookies), cookies)           # Set environment's basic authentication if not explicitly set.


Question: Regression 2.0.1: Using MozillaCookieJar does not work Could not find an issue raised for this, not sure if this was an expected change either. This is reproducible on master. Existing code fails on update to . The cause seems to be triggered by the change at The parameter expects either or . Treating as a dict triggers the error in this instance. The following code highlights the issue: The following is thrown:

prep = self.prepare_request(req)          # Add param cookies to session cookies        self.cookies = cookiejar_from_dict(cookies, cookiejar=self.cookies, overwrite=False)        self.cookies = merge_cookies(self.cookies, cookies)          proxies = proxies or {}


Question: Unicode method names cause UnicodeDecodeError for some requests in Python 2.7.2 The following example works fine: But the following example (using instead of ) produces a UnicodeDecodeError: My guess is that is infecting the header with unicode when it should be a string. This is because is simply: My requests version is 1.2.3, but I see the same being used in the current source. Unicode method names cause UnicodeDecodeError for some requests in Python 2.7.2 The following example works fine: But the following example (using instead of ) produces a UnicodeDecodeError: My guess is that is infecting the header with unicode when it should be a string. This is because is simply: My requests version is 1.2.3, but I see the same being used in the current source. Unicode method names cause UnicodeDecodeError for some requests in Python 2.7.2 The following example works fine: But the following example (using instead of ) produces a UnicodeDecodeError: My guess is that is infecting the header with unicode when it should be a string. This is because is simply: My requests version is 1.2.3, but I see the same being used in the current source.

from collections import Mapping from datetime import datetime from .compat import cookielib, OrderedDict, urljoin, urlparse, urlunparsefrom .compat import cookielib, OrderedDict, urljoin, urlparse, urlunparse, builtin_str from .cookies import cookiejar_from_dict, extract_cookies_to_jar, RequestsCookieJar from .models import Request, PreparedRequest from .hooks import default_hooks, dispatch_hook


Question: Unicode method names cause UnicodeDecodeError for some requests in Python 2.7.2 The following example works fine: But the following example (using instead of ) produces a UnicodeDecodeError: My guess is that is infecting the header with unicode when it should be a string. This is because is simply: My requests version is 1.2.3, but I see the same being used in the current source. Unicode method names cause UnicodeDecodeError for some requests in Python 2.7.2 The following example works fine: But the following example (using instead of ) produces a UnicodeDecodeError: My guess is that is infecting the header with unicode when it should be a string. This is because is simply: My requests version is 1.2.3, but I see the same being used in the current source. Unicode method names cause UnicodeDecodeError for some requests in Python 2.7.2 The following example works fine: But the following example (using instead of ) produces a UnicodeDecodeError: My guess is that is infecting the header with unicode when it should be a string. This is because is simply: My requests version is 1.2.3, but I see the same being used in the current source.

:param cert: (optional) if String, path to ssl client cert file (.pem).             If Tuple, ('cert', 'key') pair.         """<ins>        method = builtin_str(method)</ins>         # Create the Request.         req = Request(             method = method.upper(),


Question: quote qop options in Digest Auth Based on RFC2617 (), the value of 'qop-options' directive should be quoted with double quotes: curl comamnd-line tool also appends these quotes. You can see this by . Unfortunately, some minor server-side implementations seem to be sensitive on this difference.

A1 = '%s:%s:%s' % (self.username, realm, self.password)         A2 = '%s:%s' % (method, path)<del>        </del><ins></ins>         HA1 = hash_utf8(A1)         HA2 = hash_utf8(A2)


Question: quote qop options in Digest Auth Based on RFC2617 (), the value of 'qop-options' directive should be quoted with double quotes: curl comamnd-line tool also appends these quotes. You can see this by . Unfortunately, some minor server-side implementations seem to be sensitive on this difference.

if entdig:             base += ', digest="%s"' % entdig         if qop:            base += ', qop=auth, nc=%s, cnonce="%s"' % (ncvalue, cnonce)            base += ', qop="auth", nc=%s, cnonce="%s"' % (ncvalue, cnonce)          return 'Digest %s' % (base)


Question: HTTP Basic Auth credentials extracted from URL are %-encoded I was relying on credential auto-extraction from the hostname [1] to perform HTTP Basic Auth but I was getting "401 Unauthorized": the spaces in the password were substituted by . Manually extracting them with and passing them to the parameter as a tuple works. [1] (second paragraph)

from . import __version__ from . import certs from .compat import parse_http_list as _parse_list_headerfrom .compat import (quote, urlparse, bytes, str, OrderedDict, urlunparse,                     is_py2, is_py3, builtin_str, getproxies, proxy_bypass)from .compat import (quote, urlparse, bytes, str, OrderedDict, unquote, is_py2,                     builtin_str, getproxies, proxy_bypass) from .cookies import RequestsCookieJar, cookiejar_from_dict from .structures import CaseInsensitiveDict from .exceptions import MissingSchema, InvalidURL


Question: HTTP Basic Auth credentials extracted from URL are %-encoded I was relying on credential auto-extraction from the hostname [1] to perform HTTP Basic Auth but I was getting "401 Unauthorized": the spaces in the password were substituted by . Manually extracting them with and passing them to the parameter as a tuple works. [1] (second paragraph)

"""Given a url with authentication components, extract them into a tuple of     username,password."""     if url:<ins>        url = unquote(url)</ins>         parsed = urlparse(url)         return (parsed.username, parsed.password)     else:


Question: Request cookies should not be persisted to session After the fix for , cookies sent with a request are now incorrectly persisted to the session. Specifically, problem lies here: Removing that breaks the test case for though, still investigating a solution.

from base64 import b64encode  from .compat import urlparse, str<ins>from .cookies import extract_cookies_to_jar</ins> from .utils import parse_dict_header  log = logging.getLogger(__name__)


Question: Request cookies should not be persisted to session After the fix for , cookies sent with a request are now incorrectly persisted to the session. Specifically, problem lies here: Removing that breaks the test case for though, still investigating a solution.

r.content             r.raw.release_conn()             prep = r.request.copy()            prep.prepare_cookies(r.cookies)            extract_cookies_to_jar(prep._cookies, r.request, r.raw)            prep.prepare_cookies(prep._cookies)              prep.headers['Authorization'] = self.build_digest_header(                 prep.method, prep.url)diff --git a/requests/models.py b/requests/models.py-- a/requests/models.py++ b/requests/models.py


Question: Request cookies should not be persisted to session After the fix for , cookies sent with a request are now incorrectly persisted to the session. Specifically, problem lies here: Removing that breaks the test case for though, still investigating a solution.

self.url = None         #: dictionary of HTTP headers.         self.headers = None<ins>        # The `CookieJar` used to create the Cookie header will be stored here        # after prepare_cookies is called        self._cookies = None</ins>         #: request body to send to the server.         self.body = None         #: dictionary of callback hooks, for internal usage.


Question: Request cookies should not be persisted to session After the fix for , cookies sent with a request are now incorrectly persisted to the session. Specifically, problem lies here: Removing that breaks the test case for though, still investigating a solution.

p.method = self.method         p.url = self.url         p.headers = self.headers.copy()<ins>        p._cookies = self._cookies.copy()</ins>         p.body = self.body         p.hooks = self.hooks         return p


Question: Request cookies should not be persisted to session After the fix for , cookies sent with a request are now incorrectly persisted to the session. Specifically, problem lies here: Removing that breaks the test case for though, still investigating a solution.

"""Prepares the given HTTP cookie data."""          if isinstance(cookies, cookielib.CookieJar):            cookies = cookies            self._cookies = cookies         else:            cookies = cookiejar_from_dict(cookies)            self._cookies = cookiejar_from_dict(cookies)         if 'cookie' not in self.headers:            cookie_header = get_cookie_header(cookies, self)            if cookie_header is not None:                self.headers['Cookie'] = cookie_header        cookie_header = get_cookie_header(self._cookies, self)        if cookie_header is not None:            self.headers['Cookie'] = cookie_header      def prepare_hooks(self, hooks):         """Prepares the given hooks."""diff --git a/requests/sessions.py b/requests/sessions.py-- a/requests/sessions.py++ b/requests/sessions.py


Question: Request cookies should not be persisted to session After the fix for , cookies sent with a request are now incorrectly persisted to the session. Specifically, problem lies here: Removing that breaks the test case for though, still investigating a solution.

except KeyError:                 pass             prepared_request.prepare_cookies(self.cookies)            extract_cookies_to_jar(prepared_request._cookies,                                   prepared_request, resp.raw)            prepared_request.prepare_cookies(prepared_request._cookies)              resp = self.send(                 prepared_request,


Question: Request cookies should not be persisted to session After the fix for , cookies sent with a request are now incorrectly persisted to the session. Specifically, problem lies here: Removing that breaks the test case for though, still investigating a solution.

)         prep = self.prepare_request(req) <del>        # Add param cookies to session cookies        self.cookies = merge_cookies(self.cookies, cookies)</del>         proxies = proxies or {}          # Gather clues from the surrounding environment.


Question: 301 redirect broken with latest pyopenssl/SNI With the latest pyopenssl on Windows 64bit: I get an exception raised when ing a response to a HTTPS request. I see that after the redirect is received the returned URL is . Then requests passes the response to which uses the url to make a new request. This leads to a Unicode string being passed to urllib3 and eventually pyopenssl. And because in pyopenssl they now check that the data is of type bytes, an exception is thrown. I Wrote this test: and this is the result of

cookiejar_from_dict, extract_cookies_to_jar, RequestsCookieJar, merge_cookies) from .models import Request, PreparedRequest from .hooks import default_hooks, dispatch_hookfrom .utils import to_key_val_list, default_headersfrom .utils import to_key_val_list, default_headers, to_native_string from .exceptions import TooManyRedirects, InvalidSchema from .structures import CaseInsensitiveDict


Question: 301 redirect broken with latest pyopenssl/SNI With the latest pyopenssl on Windows 64bit: I get an exception raised when ing a response to a HTTPS request. I see that after the redirect is received the returned URL is . Then requests passes the response to which uses the url to make a new request. This leads to a Unicode string being passed to urllib3 and eventually pyopenssl. And because in pyopenssl they now check that the data is of type bytes, an exception is thrown. I Wrote this test: and this is the result of

else:                 url = requote_uri(url)             prepared_request.url = url            prepared_request.url = to_native_string(url)              # http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.3.4             if (resp.status_code == codes.see_other and


Question: Removing a default header of a session say that you can prevent sending a session header by setting the headers value to None in the method's arguments. You would expect (as ) that this would work for session's default headers, too: What happens is that "None" gets sent as the value of header. For the reference, here is a way that works:

if v is None:             del merged_setting[k] <ins>    merged_setting = dict((k, v) for (k, v) in merged_setting.items() if v is not None)</ins>     return merged_setting


Question: Why decode the response body of a redirect? Requests fails on the URL , which is a 301 redirect to . The issue seems to be that the server's initial 301 response has a header falsely claiming that the response body (a simple HTML page) is gzipped, when it's actually uncompressed. When resolving redirects, Requests does (in ): which attempts to decode One could legitimately say this is the server's problem. However, conceptually, why decode the response body of a redirect, which won't get returned? Other programs (Chromium, Firefox, ) don't do this. For example, gives an error, as expected, when not following redirects: whereas it works if you add the flag (follow redirects).

else:                 raise         r = self.build_response(request, resp)        if not stream:            r.content        return r        return self.build_response(request, resp)diff --git a/requests/sessions.py b/requests/sessions.py-- a/requests/sessions.py++ b/requests/sessions.py


Question: Why decode the response body of a redirect? Requests fails on the URL , which is a 301 redirect to . The issue seems to be that the server's initial 301 response has a header falsely claiming that the response body (a simple HTML page) is gzipped, when it's actually uncompressed. When resolving redirects, Requests does (in ): which attempts to decode One could legitimately say this is the server's problem. However, conceptually, why decode the response body of a redirect, which won't get returned? Other programs (Chromium, Firefox, ) don't do this. For example, gives an error, as expected, when not following redirects: whereas it works if you add the flag (follow redirects).

from .models import Request, PreparedRequest, DEFAULT_REDIRECT_LIMIT from .hooks import default_hooks, dispatch_hook from .utils import to_key_val_list, default_headers, to_native_stringfrom .exceptions import TooManyRedirects, InvalidSchemafrom .exceptions import (    TooManyRedirects, InvalidSchema, ChunkedEncodingError, ContentDecodingError) from .structures import CaseInsensitiveDict  from .adapters import HTTPAdapter


Question: Why decode the response body of a redirect? Requests fails on the URL , which is a 301 redirect to . The issue seems to be that the server's initial 301 response has a header falsely claiming that the response body (a simple HTML page) is gzipped, when it's actually uncompressed. When resolving redirects, Requests does (in ): which attempts to decode One could legitimately say this is the server's problem. However, conceptually, why decode the response body of a redirect, which won't get returned? Other programs (Chromium, Firefox, ) don't do this. For example, gives an error, as expected, when not following redirects: whereas it works if you add the flag (follow redirects).

while resp.is_redirect:             prepared_request = req.copy()             resp.content  # Consume socket so it can be released            try:                resp.content  # Consume socket so it can be released            except (ChunkedEncodingError, ContentDecodingError, RuntimeError):                resp.raw.read(decode_content=False)              if i >= self.max_redirects:                 raise TooManyRedirects('Exceeded %s redirects.' % self.max_redirects)


Question: Why decode the response body of a redirect? Requests fails on the URL , which is a 301 redirect to . The issue seems to be that the server's initial 301 response has a header falsely claiming that the response body (a simple HTML page) is gzipped, when it's actually uncompressed. When resolving redirects, Requests does (in ): which attempts to decode One could legitimately say this is the server's problem. However, conceptually, why decode the response body of a redirect, which won't get returned? Other programs (Chromium, Firefox, ) don't do this. For example, gives an error, as expected, when not following redirects: whereas it works if you add the flag (follow redirects).

r = history.pop()             r.history = history <ins>        if not stream:            r.content</ins>         return r      def get_adapter(self, url):


Question: request.history can be either a list or a tuple IMHO r.history should always be a list for least surprise. In some cases, it is returned as a tuple: Thanks!

history.insert(0, r)             # Get the last request made             r = history.pop()            r.history = tuple(history)            r.history = history          return r


Question: copies the original request for all subsequent requests, can cause incorrect method selection Consider the following redirection chain: The intermediate 303 See Other has caused the POST to be converted to a GET. The subsequent 307 should preserve the GET. However, because starts each iteration by copying the original request object, Requests will issue a POST!

if new_auth is not None:                 prepared_request.prepare_auth(new_auth)             # Override the original request.            req = prepared_request             resp = self.send(                prepared_request,                req,                 stream=stream,                 timeout=timeout,                 verify=verify,


Question: exception not caught/wrapped in a requests exception (ConnectionError perhaps?) I just noticed a case where I had a socket reset on me, and was raised to me as a raw socket error as opposed to something like a requests.exceptions.ConnectionError: Not sure if this is by accident or design... in general, I guess I'd expect a requests exception when using requests, but I can start looking for socket errors and the like as well.

import collections import datetime<ins>import socket</ins>  from io import BytesIO, UnsupportedOperation from .hooks import default_hooks


Question: exception not caught/wrapped in a requests exception (ConnectionError perhaps?) I just noticed a case where I had a socket reset on me, and was raised to me as a raw socket error as opposed to something like a requests.exceptions.ConnectionError: Not sure if this is by accident or design... in general, I guess I'd expect a requests exception when using requests, but I can start looking for socket errors and the like as well.

from .packages.urllib3.exceptions import DecodeError from .exceptions import (     HTTPError, RequestException, MissingSchema, InvalidURL,    ChunkedEncodingError, ContentDecodingError)    ChunkedEncodingError, ContentDecodingError, ConnectionError) from .utils import (     guess_filename, get_auth_from_url, requote_uri,     stream_decode_response_unicode, to_key_val_list, parse_header_links,


Question: exception not caught/wrapped in a requests exception (ConnectionError perhaps?) I just noticed a case where I had a socket reset on me, and was raised to me as a raw socket error as opposed to something like a requests.exceptions.ConnectionError: Not sure if this is by accident or design... in general, I guess I'd expect a requests exception when using requests, but I can start looking for socket errors and the like as well.

raise ChunkedEncodingError(e)                 except DecodeError as e:                     raise ContentDecodingError(e)<ins>                except socket.error as e:                    raise ConnectionError(e)</ins>             except AttributeError:                 # Standard file-like object.                 while True:


Question: Uncaught socket.timeout during post Hallo requests devs (and thanks for an awesome lib) During a specific I most of the time get a (on timeouts) but I also sometimes get a exception. Since there is a requests exception for it, I assume that the socket exception was supposed to be caught and the requests one raise instead. The full stack trace is: The development is for a plugin for the music service Wimp for the project, which means that I could post code to reproduce, but you will not be able to run it without a Sonos speaker and a Wimp subscription. I understand if this difficulty to reproduce may mean that you cannot work with this issue. Thanks in advance Kenneth

from Cookie import Morsel     from StringIO import StringIO     from .packages.urllib3.packages.ordered_dict import OrderedDict<del>    from httplib import IncompleteRead</del>      builtin_str = str     bytes = str


Question: Uncaught socket.timeout during post Hallo requests devs (and thanks for an awesome lib) During a specific I most of the time get a (on timeouts) but I also sometimes get a exception. Since there is a requests exception for it, I assume that the socket exception was supposed to be caught and the requests one raise instead. The full stack trace is: The development is for a plugin for the music service Wimp for the project, which means that I could post code to reproduce, but you will not be able to run it without a Sonos speaker and a Wimp subscription. I understand if this difficulty to reproduce may mean that you cannot work with this issue. Thanks in advance Kenneth

from http.cookies import Morsel     from io import StringIO     from collections import OrderedDict    from http.client import IncompleteRead      builtin_str = str     str = strdiff --git a/requests/models.py b/requests/models.py-- a/requests/models.py++ b/requests/models.py


Question: Uncaught socket.timeout during post Hallo requests devs (and thanks for an awesome lib) During a specific I most of the time get a (on timeouts) but I also sometimes get a exception. Since there is a requests exception for it, I assume that the socket exception was supposed to be caught and the requests one raise instead. The full stack trace is: The development is for a plugin for the music service Wimp for the project, which means that I could post code to reproduce, but you will not be able to run it without a Sonos speaker and a Wimp subscription. I understand if this difficulty to reproduce may mean that you cannot work with this issue. Thanks in advance Kenneth

import collections import datetime<del>import socket</del>  from io import BytesIO, UnsupportedOperation from .hooks import default_hooks


Question: Uncaught socket.timeout during post Hallo requests devs (and thanks for an awesome lib) During a specific I most of the time get a (on timeouts) but I also sometimes get a exception. Since there is a requests exception for it, I assume that the socket exception was supposed to be caught and the requests one raise instead. The full stack trace is: The development is for a plugin for the music service Wimp for the project, which means that I could post code to reproduce, but you will not be able to run it without a Sonos speaker and a Wimp subscription. I understand if this difficulty to reproduce may mean that you cannot work with this issue. Thanks in advance Kenneth

from .packages.urllib3.fields import RequestField from .packages.urllib3.filepost import encode_multipart_formdata from .packages.urllib3.util import parse_urlfrom .packages.urllib3.exceptions import DecodeErrorfrom .packages.urllib3.exceptions import (    DecodeError, ReadTimeoutError, ProtocolError) from .exceptions import (     HTTPError, RequestException, MissingSchema, InvalidURL,     ChunkedEncodingError, ContentDecodingError, ConnectionError)


Question: Uncaught socket.timeout during post Hallo requests devs (and thanks for an awesome lib) During a specific I most of the time get a (on timeouts) but I also sometimes get a exception. Since there is a requests exception for it, I assume that the socket exception was supposed to be caught and the requests one raise instead. The full stack trace is: The development is for a plugin for the music service Wimp for the project, which means that I could post code to reproduce, but you will not be able to run it without a Sonos speaker and a Wimp subscription. I understand if this difficulty to reproduce may mean that you cannot work with this issue. Thanks in advance Kenneth

iter_slices, guess_json_utf, super_len, to_native_string) from .compat import (     cookielib, urlunparse, urlsplit, urlencode, str, bytes, StringIO,    is_py2, chardet, json, builtin_str, basestring, IncompleteRead)    is_py2, chardet, json, builtin_str, basestring) from .status_codes import codes  #: The set of HTTP status codes that indicate an automatically


Question: Uncaught socket.timeout during post Hallo requests devs (and thanks for an awesome lib) During a specific I most of the time get a (on timeouts) but I also sometimes get a exception. Since there is a requests exception for it, I assume that the socket exception was supposed to be caught and the requests one raise instead. The full stack trace is: The development is for a plugin for the music service Wimp for the project, which means that I could post code to reproduce, but you will not be able to run it without a Sonos speaker and a Wimp subscription. I understand if this difficulty to reproduce may mean that you cannot work with this issue. Thanks in advance Kenneth

try:                     for chunk in self.raw.stream(chunk_size, decode_content=True):                         yield chunk                except IncompleteRead as e:                except ProtocolError as e:                     raise ChunkedEncodingError(e)                 except DecodeError as e:                     raise ContentDecodingError(e)                except socket.error as e:                except ReadTimeoutError as e:                     raise ConnectionError(e)             except AttributeError:                 # Standard file-like object.


Question: method = builtinstr(method) problem In is a command: method = builtinstr(method) Converts method from b’GET’ to "b'GET’" Which is the literal string, no longer a binary string. When requests tries to use the method "b'GET’”, it gets a 404 Not Found response. I am using python3.4 and python-neutronclient (2.3.9) with requests (2.4.3). neutronclient is broken because it uses this "args = utils.safeencodelist(args)" command which converts all the values to binary string, including method. I'm not sure if this is a bug with neutronclient or a bug with requests, but I'm starting here. Seems if requests handled the method value being a binary string, we wouldn't have any problem. Also, I tried in python2.6 and this bug doesn't exist there. Some difference between 2.6 and 3.4 makes this not work right.

from datetime import datetime  from .auth import _basic_auth_strfrom .compat import cookielib, OrderedDict, urljoin, urlparse, builtin_strfrom .compat import cookielib, OrderedDict, urljoin, urlparse from .cookies import (     cookiejar_from_dict, extract_cookies_to_jar, RequestsCookieJar, merge_cookies) from .models import Request, PreparedRequest, DEFAULT_REDIRECT_LIMIT


Question: method = builtinstr(method) problem In is a command: method = builtinstr(method) Converts method from b’GET’ to "b'GET’" Which is the literal string, no longer a binary string. When requests tries to use the method "b'GET’”, it gets a 404 Not Found response. I am using python3.4 and python-neutronclient (2.3.9) with requests (2.4.3). neutronclient is broken because it uses this "args = utils.safeencodelist(args)" command which converts all the values to binary string, including method. I'm not sure if this is a bug with neutronclient or a bug with requests, but I'm starting here. Seems if requests handled the method value being a binary string, we wouldn't have any problem. Also, I tried in python2.6 and this bug doesn't exist there. Some difference between 2.6 and 3.4 makes this not work right.

If Tuple, ('cert', 'key') pair.         """         method = builtin_str(method)        method = to_native_string(method)          # Create the Request.         req = Request(


Question: Requests unable to follow/retrieve links with percent in url A simple (url) doesn't work for the following: !;&placement=%pplacement=!;

This function passes the given URI through an unquote/quote cycle to     ensure that it is fully and consistently quoted.     """    # Unquote only the unreserved characters    # Then quote only illegal characters (do not quote reserved, unreserved,    # or '%')    return quote(unquote_unreserved(uri), safe="!#$%&'()*+,/:;=?@[]~")    safe_with_percent = "!#$%&'()*+,/:;=?@[]~"    safe_without_percent = "!#$&'()*+,/:;=?@[]~"    try:        # Unquote only the unreserved characters        # Then quote only illegal characters (do not quote reserved,        # unreserved, or '%')        return quote(unquote_unreserved(uri), safe=safe_with_percent)    except InvalidURL:        # We couldn't unquote the given URI, so let's try quoting it, but        # there may be unquoted '%'s in the URI. We need to make sure they're        # properly quoted so they do not cause issues elsewhere.        return quote(uri, safe=safe_without_percent)   def address_in_network(ip, net):


Question: Requests 2.5.1 doesn't recognize unicode filenames for uploads After merge of , to allow filenames to be types, unicode filenames are no longer recognized under Python 2. This checks that the filename is a , which has different behaviour on Python 2 and Python 3: In , is defines as , which is non-unicode in Python 2 and unicode in Python 3. Perhaps the check should be against basestring, or is this change in behaviour intended?

basestring = basestring     numeric_types = (int, long, float)  elif is_py3:     from urllib.parse import urlparse, urlunparse, urljoin, urlsplit, urlencode, quote, unquote, quote_plus, unquote_plus, urldefrag     from urllib.request import parse_http_list, getproxies, proxy_bypassdiff --git a/requests/utils.py b/requests/utils.py-- a/requests/utils.py++ b/requests/utils.py


Question: Requests 2.5.1 doesn't recognize unicode filenames for uploads After merge of , to allow filenames to be types, unicode filenames are no longer recognized under Python 2. This checks that the filename is a , which has different behaviour on Python 2 and Python 3: In , is defines as , which is non-unicode in Python 2 and unicode in Python 3. Perhaps the check should be against basestring, or is this change in behaviour intended?

from . import certs from .compat import parse_http_list as _parse_list_header from .compat import (quote, urlparse, bytes, str, OrderedDict, unquote, is_py2,                     builtin_str, getproxies, proxy_bypass, urlunparse)                     builtin_str, getproxies, proxy_bypass, urlunparse,                     basestring) from .cookies import RequestsCookieJar, cookiejar_from_dict from .structures import CaseInsensitiveDict from .exceptions import InvalidURL


Question: Requests 2.5.1 doesn't recognize unicode filenames for uploads After merge of , to allow filenames to be types, unicode filenames are no longer recognized under Python 2. This checks that the filename is a , which has different behaviour on Python 2 and Python 3: In , is defines as , which is non-unicode in Python 2 and unicode in Python 3. Perhaps the check should be against basestring, or is this change in behaviour intended?

def guess_filename(obj):     """Tries to guess the filename of the given object."""     name = getattr(obj, 'name', None)    if name and isinstance(name, builtin_str) and name[0] != '<' and name[-1] != '>':    if (name and isinstance(name, basestring) and name[0] != '<' and            name[-1] != '>'):         return os.path.basename(name)


Question: Issue with pyinstaller Can't seem to wrap my head around this. having a problem compiling a windows exe in python 2.7.9 that uses the requests library and can't find anything on google about the specific error. My script runs fine from the interpreter but when i use pyinstaller, i get :ImportError: No module named '' I can also compile windows executables that don't use requests just fine.

class VendorAlias(object):     def __init__(self):    def __init__(self, package_names):        self._package_names = package_names         self._vendor_name = __name__         self._vendor_pkg = self._vendor_name + "."        self._vendor_pkgs = [            self._vendor_pkg + name for name in self._package_names        ]      def find_module(self, fullname, path=None):         if fullname.startswith(self._vendor_pkg):


Question: Issue with pyinstaller Can't seem to wrap my head around this. having a problem compiling a windows exe in python 2.7.9 that uses the requests library and can't find anything on google about the specific error. My script runs fine from the interpreter but when i use pyinstaller, i get :ImportError: No module named '' I can also compile windows executables that don't use requests just fine.

)             ) <ins>        if not (name == self._vendor_name or                any(name.startswith(pkg) for pkg in self._vendor_pkgs)):            raise ImportError(                "Cannot import %s, must be one of %s." % (                    name, self._vendor_pkgs                )            )</ins>         # Check to see if we already have this item in sys.modules, if we do         # then simply return that.         if name in sys.modules:


Question: Issue with pyinstaller Can't seem to wrap my head around this. having a problem compiling a windows exe in python 2.7.9 that uses the requests library and can't find anything on google about the specific error. My script runs fine from the interpreter but when i use pyinstaller, i get :ImportError: No module named '' I can also compile windows executables that don't use requests just fine.

return module  sys.meta_path.append(VendorAlias())sys.meta_path.append(VendorAlias(["urllib3", "chardet"]))


Question: Prepared requests containing binary files will not send when unicodeliterals is imported The above program works perfectly in python3, and in python2 when is not imported. If the request isn't prepared it works without a problem unfortunately, I require both prepared requests and in my project. The exception raised is: Traceback (most recent call last): File "", line 15, in (main()) File "", line 12, in main requests.Session().send(prepared) File "", line 573, in send r = (request, kwargs) File "", line 370, in send timeout=timeout File "", line 544, in urlopen body=body, headers=headers) File "", line 349, in makerequest conn.request(method, url, httplibrequestkw) File "", line 1001, in request self.sendrequest(method, url, body, headers) File "", line 1035, in sendrequest self.endheaders(body) File "", line 997, in endheaders self.sendoutput(messagebody) File "", line 848, in sendoutput msg += message_body UnicodeDecodeError: 'ascii' codec can't decode byte 0xff in position 109: ordinal not in range(128)

def prepare_method(self, method):         """Prepares the given HTTP method."""         self.method = method        if self.method is not None:            self.method = self.method.upper()        if self.method is None:            raise ValueError('Request method cannot be "None"')        self.method = to_native_string(self.method).upper()      def prepare_url(self, url, params):         """Prepares the given HTTP URL."""diff --git a/requests/sessions.py b/requests/sessions.py-- a/requests/sessions.py++ b/requests/sessions.py


Question: Prepared requests containing binary files will not send when unicodeliterals is imported The above program works perfectly in python3, and in python2 when is not imported. If the request isn't prepared it works without a problem unfortunately, I require both prepared requests and in my project. The exception raised is: Traceback (most recent call last): File "", line 15, in (main()) File "", line 12, in main requests.Session().send(prepared) File "", line 573, in send r = (request, kwargs) File "", line 370, in send timeout=timeout File "", line 544, in urlopen body=body, headers=headers) File "", line 349, in makerequest conn.request(method, url, httplibrequestkw) File "", line 1001, in request self.sendrequest(method, url, body, headers) File "", line 1035, in sendrequest self.endheaders(body) File "", line 997, in endheaders self.sendoutput(messagebody) File "", line 848, in sendoutput msg += message_body UnicodeDecodeError: 'ascii' codec can't decode byte 0xff in position 109: ordinal not in range(128)

:param cert: (optional) if String, path to ssl client cert file (.pem).             If Tuple, ('cert', 'key') pair.         """<del>        method = to_native_string(method)</del>         # Create the Request.         req = Request(             method = method.upper(),


Question: urllib3 exceptions passing through requests API I don't know if it's a design goal of requests to hide urllib3's exceptions and wrap them around requests.exceptions types. (If it's not IMHO it should be, but that's another discussion) If it is, I have at least two of them passing through that I have to catch in addition to requests' exceptions. They are requests.packages.urllib3.exceptions.DecodeError and requests.packages.urllib3.exceptions.TimeoutError (this one I get when a proxy timeouts) Thanks!

from .utils import (DEFAULT_CA_BUNDLE_PATH, get_encoding_from_headers,                     prepend_scheme_if_needed, get_auth_from_url, urldefragauth) from .structures import CaseInsensitiveDict<ins>from .packages.urllib3.exceptions import ClosedPoolError</ins> from .packages.urllib3.exceptions import ConnectTimeoutError from .packages.urllib3.exceptions import HTTPError as _HTTPError from .packages.urllib3.exceptions import MaxRetryError


Question: urllib3 exceptions passing through requests API I don't know if it's a design goal of requests to hide urllib3's exceptions and wrap them around requests.exceptions types. (If it's not IMHO it should be, but that's another discussion) If it is, I have at least two of them passing through that I have to catch in addition to requests' exceptions. They are requests.packages.urllib3.exceptions.DecodeError and requests.packages.urllib3.exceptions.TimeoutError (this one I get when a proxy timeouts) Thanks!

raise ConnectionError(e, request=request) <ins>        except ClosedPoolError as e:            raise ConnectionError(e, request=request)</ins>         except _ProxyError as e:             raise ProxyError(e)


Question: .htaccesss redirect to non ASCII folder does not work Hello, I have the following setup on a shared hoster: Apache 2.2.15 A Japanese language . (.minna; xn--q9jyb4c) IDN domain. A blog which is in the subfolder (blog) A redirect in the .htaccess file like this: So I usually open the domain http://test. and the server redirects to http://test./. This works fine in Firefox etc. With requests, I get the following error (Python 3.4 with Requests 2.7.0 on a Japanese Ubuntu 15.04): So I guess the request lib gets a redirect from a server with Japanese characters, but then fails to convert the characters correctly. If I do directly it works, only the redirect does not.

from datetime import datetime  from .auth import _basic_auth_strfrom .compat import cookielib, OrderedDict, urljoin, urlparsefrom .compat import cookielib, OrderedDict, urljoin, urlparse, is_py3, str from .cookies import (     cookiejar_from_dict, extract_cookies_to_jar, RequestsCookieJar, merge_cookies) from .models import Request, PreparedRequest, DEFAULT_REDIRECT_LIMIT


Question: .htaccesss redirect to non ASCII folder does not work Hello, I have the following setup on a shared hoster: Apache 2.2.15 A Japanese language . (.minna; xn--q9jyb4c) IDN domain. A blog which is in the subfolder (blog) A redirect in the .htaccess file like this: So I usually open the domain http://test. and the server redirects to http://test./. This works fine in Firefox etc. With requests, I get the following error (Python 3.4 with Requests 2.7.0 on a Japanese Ubuntu 15.04): So I guess the request lib gets a redirect from a server with Japanese characters, but then fails to convert the characters correctly. If I do directly it works, only the redirect does not.

parsed = urlparse(location_url)             location_url = parsed.geturl()             # On Python 3, the location header was decoded using Latin 1, but            # urlparse in requote_uri will encode it with UTF-8 before quoting.            # Because of this insanity, we need to fix it up ourselves by            # sending the URL back to bytes ourselves.            if is_py3 and isinstance(location_url, str):                location_url = location_url.encode('latin1')             # Facilitate relative 'location' headers, as allowed by RFC 7231.             # (e.g. '/path/to/resource' instead of 'http://domain.tld/path/to/resource')             # Compliant with RFC3986, we percent encode the url.diff --git a/requests/utils.py b/requests/utils.py-- a/requests/utils.py++ b/requests/utils.py


Question: .htaccesss redirect to non ASCII folder does not work Hello, I have the following setup on a shared hoster: Apache 2.2.15 A Japanese language . (.minna; xn--q9jyb4c) IDN domain. A blog which is in the subfolder (blog) A redirect in the .htaccess file like this: So I usually open the domain http://test. and the server redirects to http://test./. This works fine in Firefox etc. With requests, I get the following error (Python 3.4 with Requests 2.7.0 on a Japanese Ubuntu 15.04): So I guess the request lib gets a redirect from a server with Japanese characters, but then fails to convert the characters correctly. If I do directly it works, only the redirect does not.

from .compat import parse_http_list as _parse_list_header from .compat import (quote, urlparse, bytes, str, OrderedDict, unquote, is_py2,                      builtin_str, getproxies, proxy_bypass, urlunparse,                     basestring)                     basestring, is_py3) from .cookies import RequestsCookieJar, cookiejar_from_dict from .structures import CaseInsensitiveDict from .exceptions import InvalidURL, FileModeWarning


Question: .htaccesss redirect to non ASCII folder does not work Hello, I have the following setup on a shared hoster: Apache 2.2.15 A Japanese language . (.minna; xn--q9jyb4c) IDN domain. A blog which is in the subfolder (blog) A redirect in the .htaccess file like this: So I usually open the domain http://test. and the server redirects to http://test./. This works fine in Firefox etc. With requests, I get the following error (Python 3.4 with Requests 2.7.0 on a Japanese Ubuntu 15.04): So I guess the request lib gets a redirect from a server with Japanese characters, but then fails to convert the characters correctly. If I do directly it works, only the redirect does not.

"""Un-escape any percent-escape sequences in a URI that are unreserved     characters. This leaves all reserved, illegal and non-ASCII bytes encoded.     """    parts = uri.split('%')    # This convert function is used to optionally convert the output of `chr`.    # In Python 3, `chr` returns a unicode string, while in Python 2 it returns    # a bytestring. Here we deal with that by optionally converting.    def convert(is_bytes, c):        if is_py2 and not is_bytes:            return c.decode('ascii')        elif is_py3 and is_bytes:            return c.encode('ascii')        else:            return c    # Handle both bytestrings and unicode strings.    is_bytes = isinstance(uri, bytes)    splitchar = u'%'    base = u''    if is_bytes:        splitchar = splitchar.encode('ascii')        base = base.encode('ascii')    parts = uri.split(splitchar)     for i in range(1, len(parts)):         h = parts[i][0:2]         if len(h) == 2 and h.isalnum():


Question: .htaccesss redirect to non ASCII folder does not work Hello, I have the following setup on a shared hoster: Apache 2.2.15 A Japanese language . (.minna; xn--q9jyb4c) IDN domain. A blog which is in the subfolder (blog) A redirect in the .htaccess file like this: So I usually open the domain http://test. and the server redirects to http://test./. This works fine in Firefox etc. With requests, I get the following error (Python 3.4 with Requests 2.7.0 on a Japanese Ubuntu 15.04): So I guess the request lib gets a redirect from a server with Japanese characters, but then fails to convert the characters correctly. If I do directly it works, only the redirect does not.

raise InvalidURL("Invalid percent-escape sequence: '%s'" % h)              if c in UNRESERVED_SET:                parts[i] = c + parts[i][2:]                parts[i] = convert(is_bytes, c) + parts[i][2:]             else:                parts[i] = '%' + parts[i]                parts[i] = splitchar + parts[i]         else:            parts[i] = '%' + parts[i]    return ''.join(parts)            parts[i] = splitchar + parts[i]    return base.join(parts)   def requote_uri(uri):


Question: TypeError after 2.7.0 -2.8.0 upgrade (cannot make memory view...) I'm running into this traceback after upgrading to 2.8.0 from 2.7.0: The problem goes away after downgrading to 2.7.0. A full traceback can be found . Here are the versions of relevant packages: And a full list of packages: Any ideas?

"""Prepares the given HTTP method."""         self.method = method         if self.method is not None:            self.method = self.method.upper()            self.method = to_native_string(self.method.upper())      def prepare_url(self, url, params):         """Prepares the given HTTP URL."""diff --git a/requests/sessions.py b/requests/sessions.py-- a/requests/sessions.py++ b/requests/sessions.py


Question: TypeError after 2.7.0 -2.8.0 upgrade (cannot make memory view...) I'm running into this traceback after upgrading to 2.8.0 from 2.7.0: The problem goes away after downgrading to 2.7.0. A full traceback can be found . Here are the versions of relevant packages: And a full list of packages: Any ideas?


Question: Post request hangs in certain cases when body is a StringIO This is related to a report for the . The following hangs: After a cursory look, it looks like the request isn't fully formed so the server doesn't attempt to send a response which leaves the client hanging. If we call , this works. A bit more counterintuitively, this also works:

def super_len(o):    total_length = 0    current_position = 0     if hasattr(o, '__len__'):        return len(o)        total_length = len(o)    elif hasattr(o, 'len'):        total_length = o.len     if hasattr(o, 'len'):        return o.len    elif hasattr(o, 'getvalue'):        # e.g. BytesIO, cStringIO.StringIO        total_length = len(o.getvalue())     if hasattr(o, 'fileno'):    elif hasattr(o, 'fileno'):         try:             fileno = o.fileno()         except io.UnsupportedOperation:             pass         else:            filesize = os.fstat(fileno).st_size            total_length = os.fstat(fileno).st_size              # Having used fstat to determine the file length, we need to             # confirm that this file was opened up in binary mode.


Question: Post request hangs in certain cases when body is a StringIO This is related to a report for the . The following hangs: After a cursory look, it looks like the request isn't fully formed so the server doesn't attempt to send a response which leaves the client hanging. If we call , this works. A bit more counterintuitively, this also works:

FileModeWarning                 )             return filesize    if hasattr(o, 'tell'):        current_position = o.tell()     if hasattr(o, 'getvalue'):        # e.g. BytesIO, cStringIO.StringIO        return len(o.getvalue())    return max(0, total_length - current_position)   def get_netrc_auth(url, raise_errors=False):


Question: Request with binary payload fails due to calling tonativestring Introduced with This works with 2.8.1, but not with 2.9.

"""          if isinstance(data, (str, bytes)):            return to_native_string(data)            return data         elif hasattr(data, 'read'):             return data         elif hasattr(data, '__iter__'):


Question: Request with binary payload fails due to calling tonativestring Introduced with This works with 2.8.1, but not with 2.9.

if isinstance(fragment, str):                 fragment = fragment.encode('utf-8') <ins>        if isinstance(params, (str, bytes)):            params = to_native_string(params)</ins>         enc_params = self._encode_params(params)         if enc_params:             if query:


Question: Uncertain about content/text vs itercontent(decodeunicode=True/False) When requesting an application/json document, I'm seeing returning bytes, whereas returns unicode. My understanding was that both should return a unicode object. In essence, I thought "itercontent" was equivalent to "itertext" when decode_unicode was True. Have I misunderstood something? I can provide an example if needed. For reference, I'm using python 3.5.1 and requests 2.10.0. Thanks!

def stream_decode_response_unicode(iterator, r):     """Stream decodes a iterator."""    encoding = r.encoding     if r.encoding is None:        for item in iterator:            yield item        return    if encoding is None:        encoding = r.apparent_encoding    try:        decoder = codecs.getincrementaldecoder(encoding)(errors='replace')    except (LookupError, TypeError):        # A LookupError is raised if the encoding was not found which could        # indicate a misspelling or similar mistake.        #        # A TypeError can be raised if encoding is None        raise UnicodeError("Unable to decode contents with encoding %s." % encoding)     decoder = codecs.getincrementaldecoder(r.encoding)(errors='replace')     for chunk in iterator:         rv = decoder.decode(chunk)         if rv:


Question: AttributeError: 'NoneType' object has no attribute 'read' Hello :) After a recent upgrade for our project to 2.12.1 we encounter an exception in our test suites which seems to be caused by . Build: Relevant part: happens on Windows and Linux. Thanks in advance :)

raise RuntimeError(                     'The content for this response was already consumed')             if self.status_code == 0:            if self.status_code == 0 or self.raw is None:                 self._content = None             else:                 self._content = bytes().join(self.iter_content(CONTENT_CHUNK_SIZE)) or bytes()


Question: Requests v2.12.2 does not add parameters to URLs with schemes it does not understand. Follows from . When we patched to ignore all unrecognised schemes instead of just those that didn't begin , we stopped handling parameters for those URLs. This may break some more benign uses such as , which wanted to add parameters to their URLs. While this is inline with our intended policies (we do not understand URLs that have schemes we don't know anything about), this, along with the IDNA behaviour in v2.12.{0,1} will probably have broken a whole bunch of people using these non-standard URL schemes. We should consider whether Session objects should have a registry of places to look for URL preparation based on scheme. This will allow people to opt-in to the HTTP-like processing of URLs, as well as to register their own.

url = url.lstrip()          # Don't do any URL preparation for non-HTTP schemes like `mailto`,        # `data`, `http+unix` etc to work around exceptions from `url_parse`,        # which handles RFC 3986 only.        if ':' in url and not url.lower().startswith(('http://', 'https://')):        # `data` etc to work around exceptions from `url_parse`, which        # handles RFC 3986 only.        if ':' in url and not url.lower().startswith('http'):             self.url = url             return


Question: AttributeError: module 'requests.packages' has no attribute 'urllib3' This seems to have broken requests.packages. requests.packages.urllib3 to be the urllib3 package AttributeError: module 'requests.packages' has no attribute 'urllib3' $ python -m

# I don't like it either. Just look the other way. :)  for package in ('urllib3', 'idna', 'chardet'):    __import__(package)    locals()[package] = __import__(package)     # This traversal is apparently necessary such that the identities are     # preserved (requests.packages.urllib3.* is urllib3.*)     for mod in list(sys.modules):


Question: Misleading exception with invalid protocol in proxy variable When the value of or variable(s) accidentally miss one '/' in the protocol, a traceback is thrown to the user which doesn't pin point that the issue is with the proxy configuration. A better exception An exception which doesn't pin point exactly what went wrong. I am not sure what is the correct place to fix this. Should the fix/check be in requests, urllib3, or urlparse?

from urllib3.poolmanager import PoolManager, proxy_from_url from urllib3.response import HTTPResponse<ins>from urllib3.util import parse_url</ins> from urllib3.util import Timeout as TimeoutSauce from urllib3.util.retry import Retry from urllib3.exceptions import ClosedPoolError


Question: Misleading exception with invalid protocol in proxy variable When the value of or variable(s) accidentally miss one '/' in the protocol, a traceback is thrown to the user which doesn't pin point that the issue is with the proxy configuration. A better exception An exception which doesn't pin point exactly what went wrong. I am not sure what is the correct place to fix this. Should the fix/check be in requests, urllib3, or urlparse?

from .structures import CaseInsensitiveDict from .cookies import extract_cookies_to_jar from .exceptions import (ConnectionError, ConnectTimeout, ReadTimeout, SSLError,                         ProxyError, RetryError, InvalidSchema)                         ProxyError, RetryError, InvalidSchema, InvalidProxyURL) from .auth import _basic_auth_str  try:


Question: Misleading exception with invalid protocol in proxy variable When the value of or variable(s) accidentally miss one '/' in the protocol, a traceback is thrown to the user which doesn't pin point that the issue is with the proxy configuration. A better exception An exception which doesn't pin point exactly what went wrong. I am not sure what is the correct place to fix this. Should the fix/check be in requests, urllib3, or urlparse?

if proxy:             proxy = prepend_scheme_if_needed(proxy, 'http')            proxy_url = parse_url(proxy)            if not proxy_url.host:                raise InvalidProxyURL("Please check proxy URL. It is malformed"                                      " and could be missing the host.")             proxy_manager = self.proxy_manager_for(proxy)             conn = proxy_manager.connection_from_url(url)         else:diff --git a/requests/exceptions.py b/requests/exceptions.py-- a/requests/exceptions.py++ b/requests/exceptions.py


Question: Misleading exception with invalid protocol in proxy variable When the value of or variable(s) accidentally miss one '/' in the protocol, a traceback is thrown to the user which doesn't pin point that the issue is with the proxy configuration. A better exception An exception which doesn't pin point exactly what went wrong. I am not sure what is the correct place to fix this. Should the fix/check be in requests, urllib3, or urlparse?

"""The header value provided was somehow invalid."""  <ins>class InvalidProxyURL(InvalidURL):    """The proxy URL provided is invalid."""</ins> class ChunkedEncodingError(RequestException):     """The server declared chunked encoding but sent an invalid chunk."""


Question: Should Authorization header be cleared in https -http redirect? This may be considered intentional behaviour (in which case feel free to close this), but if a request is made to an https endpoint with authorization and it redirects to http on the same host, the Authorization header is not stripped and will be exposed on the wire. rebuild_auth would strip the Authorization header if the scheme is changed from https to http. The credentials that were intended to be sent over TLS were transmitted in plaintext with the redirected request. Run an HTTPS server on localhost:4443 that replies with a 302 redirect to , and a plain HTTP server (or netcat) on localhost:8000. Then run The basic auth credentials are sent in plaintext to (the is just because I had a self-signed cert). Here's the code I used for the SSL server:

return to_native_string(location, 'utf8')         return None <ins>    def should_strip_auth(self, old_url, new_url):        """Decide whether Authorization header should be removed when redirecting"""        old_parsed = urlparse(old_url)        new_parsed = urlparse(new_url)        if old_parsed.hostname != new_parsed.hostname:            return True        # Special case: allow http -> https redirect when using the standard        # ports. This isn't specified by RFC 7235, but is kept to avoid        # breaking backwards compatibility with older versions of requests        # that allowed any redirects on the same host.        if (old_parsed.scheme == 'http' and old_parsed.port in (80, None)                and new_parsed.scheme == 'https' and new_parsed.port in (443, None)):            return False        # Standard case: root URI must match        return old_parsed.port != new_parsed.port or old_parsed.scheme != new_parsed.scheme</ins>     def resolve_redirects(self, resp, req, stream=False, timeout=None,                           verify=True, cert=None, proxies=None, yield_requests=False, **adapter_kwargs):         """Receives a Response. Returns a generator of Responses or Requests."""


Question: Should Authorization header be cleared in https -http redirect? This may be considered intentional behaviour (in which case feel free to close this), but if a request is made to an https endpoint with authorization and it redirects to http on the same host, the Authorization header is not stripped and will be exposed on the wire. rebuild_auth would strip the Authorization header if the scheme is changed from https to http. The credentials that were intended to be sent over TLS were transmitted in plaintext with the redirected request. Run an HTTPS server on localhost:4443 that replies with a 302 redirect to , and a plain HTTP server (or netcat) on localhost:8000. Then run The basic auth credentials are sent in plaintext to (the is just because I had a self-signed cert). Here's the code I used for the SSL server:

headers = prepared_request.headers         url = prepared_request.url         if 'Authorization' in headers:        if 'Authorization' in headers and self.should_strip_auth(response.request.url, url):             # If we get redirected to a new host, we should strip out any             # authentication headers.            original_parsed = urlparse(response.request.url)            redirect_parsed = urlparse(url)            if (original_parsed.hostname != redirect_parsed.hostname):                del headers['Authorization']            del headers['Authorization']          # .netrc might have more auth for us on our new host.         new_auth = get_netrc_auth(url) if self.trust_env else None


Question: Accessing response.content twice removes forgets read error I had a hard debugging time today because an error in the response stream is only reported when accessing for the first time. This is especially irritating when running code in a debugger. If accessing the first time raises an exception I would expect that accessing again would also raise an exception (ideally the same). Instead after raising on the first get, getting again returns an empty string. Here is a patch with a new test case for this: . Basically, it boils down to this: Output of my test case: $ python -m Edit: Oops, I used which actually called into system python 2.7. Here comes the real data: Thanks for looking into this!

#: is a response.         self.request = None <ins>        #: If there was an error in the processing of content,        #: then save the error that would return the same error when you re-appeal.        self._error = None</ins>     def __enter__(self):         return self


Question: Accessing response.content twice removes forgets read error I had a hard debugging time today because an error in the response stream is only reported when accessing for the first time. This is especially irritating when running code in a debugger. If accessing the first time raises an exception I would expect that accessing again would also raise an exception (ideally the same). Instead after raising on the first get, getting again returns an empty string. Here is a patch with a new test case for this: . Basically, it boils down to this: Output of my test case: $ python -m Edit: Oops, I used which actually called into system python 2.7. Here comes the real data: Thanks for looking into this!

try:                     for chunk in self.raw.stream(chunk_size, decode_content=True):                         yield chunk                 except ProtocolError as e:                    raise ChunkedEncodingError(e)                    self._error = ChunkedEncodingError(e)                 except DecodeError as e:                    raise ContentDecodingError(e)                    self._error = ContentDecodingError(e)                 except ReadTimeoutError as e:                    raise ConnectionError(e)                    self._error = ConnectionError(e)                finally:                    # if we had an error - throw the saved error                    if self._error:                        raise self._error             else:                 # Standard file-like object.                 while True:


Question: Accessing response.content twice removes forgets read error I had a hard debugging time today because an error in the response stream is only reported when accessing for the first time. This is especially irritating when running code in a debugger. If accessing the first time raises an exception I would expect that accessing again would also raise an exception (ideally the same). Instead after raising on the first get, getting again returns an empty string. Here is a patch with a new test case for this: . Basically, it boils down to this: Output of my test case: $ python -m Edit: Oops, I used which actually called into system python 2.7. Here comes the real data: Thanks for looking into this!

else:                 self._content = b''.join(self.iter_content(CONTENT_CHUNK_SIZE)) or b'' <ins>        # if we had an error - throw the saved error        if self._error is not None:            raise self._error</ins>         self._content_consumed = True         # don't need to release the connection; that's been handled by urllib3         # since we exhausted the data.


Question: Getting raises UnicodeError Attempting to get e.g. results in a . It seems like the intention so far has been to raise instead (see e.g. ). I see there was some hesitation in fixing a similar issue () and would like to add that even catching the error just to rethrow as a requests exception would be beneficial. Based on PR : $ python -m

host = self._get_idna_encoded_host(host)             except UnicodeError:                 raise InvalidURL('URL has an invalid label.')        elif host.startswith(u'*'):        elif host.startswith((u'*', u'.')):             raise InvalidURL('URL has an invalid label.')          # Carefully reconstruct the network location


Question: Proxy authentication bug <!-- Summary. --When using proxies in python 3.8.12, I get an error 407. Using any other version of python works fine. I am assuming it could be to do with this <!-- What you expected. --I should get a status of 200. <!-- What happened instead. --I get a status code of 407.

if not netloc:         netloc, path = path, netloc <ins>    if auth:        # parse_url doesn't provide the netloc with auth        # so we'll add it ourselves.        netloc = '@'.join([auth, netloc])</ins>     if scheme is None:         scheme = new_scheme     if path is None:


Question: Catch UnicodeError coming from Here's a Python 2.6 Requests 0.13.1 traceback. Looks like the call to needs a try/except.

if not scheme in SCHEMAS:             raise InvalidSchema("Invalid scheme %r" % scheme)         netloc = netloc.encode('idna').decode('utf-8')        try:            netloc = netloc.encode('idna').decode('utf-8')        except UnicodeError:            raise InvalidURL('URL has an invalid label.')          if not path:             path = '/'


Question: Allow lists in the dict values of the hooks argument Currently the Request class has a .registerhook() method but it parses the dictionary it expects from it's hooks argument weirdly: the argument can only specify one hook function per hook. If you pass in a list of hook functions per hook the code in () will wrap the list in a list which then fails when the hooks are consumed (since a list is not callable). This is especially annoying since you can not use multiple hooks from a session. The only way to get multiple hooks now is to create the request object without sending it, then call .registerhook() multiple times and then finally call .send(). This would all be much easier if () parsed the hooks parameter in a way that it accepts lists as it's values.

def register_hook(self, event, hook):         """Properly register a hook."""        self.hooks[event].append(hook)        if isinstance(hook, (list, tuple, set)):            self.hooks[event].extend(hook)        else:            self.hooks[event].append(hook)      def deregister_hook(self, event, hook):         """Deregister a previously registered hook.


Question: Variable.setitem coercing types on objects with a values property The current behavior prevents storing objects inside arrays of even when only performing non-broadcasted assignments if the RHS has a property. Many libraries produce objects with a property that gets coerced as a result. The use case I had in prior versions was to store instances from the curve fitting library , when fitting had be performed over an axis of a or . Ideally: Breaking changed introduced going from -as a result of , namely the change on line <detailsINSTALLED VERSIONS commit: None python: python-bits: 64 OS: Darwin OS-release: 16.7.0 machine: x8664 processor: i386 byteorder: little LCALL: None LANG: enUS.UTF-8 LOCALE: enUS.UTF-8 xarray: 0.10.1 pandas: 0.20.3 numpy: 1.13.1 scipy: 0.19.1 netCDF4: 1.3.0 h5netcdf: None h5py: 2.7.0 Nio: None zarr: None bottleneck: None cyordereddict: None dask: 0.15.2 distributed: None matplotlib: 2.0.2 cartopy: None seaborn: 0.8.1 setuptools: 38.4.0 pip: 9.0.1 conda: None pytest: 3.3.2 IPython: 6.1.0 sphinx: None </detailsThank you for your help! If I can be brought to better understand any constraints to adjacent issues, I can consider drafting a fix for this.

data = np.timedelta64(getattr(data, "value", data), "ns")      # we don't want nested self-described arrays    data = getattr(data, "values", data)    if isinstance(data, (pd.Series, pd.Index, pd.DataFrame)):        data = data.values      if isinstance(data, np.ma.MaskedArray):         mask = np.ma.getmaskarray(data)


Question: add average function It would be nice to be able to do to compute weighted averages (e.g. for geo data). Of course this would require the axes to be in a predictable order. Or to give a weight per dimension...

},         )     def weighted(self, weights):        """        Weighted operations.        Parameters        ----------        weights : DataArray            An array of weights associated with the values in this Dataset.            Each value in the data contributes to the reduction operation            according to its associated weight.        Notes        -----        ``weights`` must be a DataArray and cannot contain missing values.        Missing values can be replaced by ``weights.fillna(0)``.        """        return self._weighted_cls(self, weights)     def rolling(         self,         dim: Mapping[Hashable, int] = None,diff --git a/xarray/core/dataarray.py b/xarray/core/dataarray.py-- a/xarray/core/dataarray.py++ b/xarray/core/dataarray.py


Question: add average function It would be nice to be able to do to compute weighted averages (e.g. for geo data). Of course this would require the axes to be in a predictable order. Or to give a weight per dimension...

resample,     rolling,     utils,<ins>    weighted,</ins> ) from .accessor_dt import CombinedDatetimelikeAccessor from .accessor_str import StringAccessor


Question: add average function It would be nice to be able to do to compute weighted averages (e.g. for geo data). Of course this would require the axes to be in a predictable order. Or to give a weight per dimension...

_rolling_cls = rolling.DataArrayRolling     _coarsen_cls = rolling.DataArrayCoarsen     _resample_cls = resample.DataArrayResample    _weighted_cls = weighted.DataArrayWeighted      dt = property(CombinedDatetimelikeAccessor) diff --git a/xarray/core/dataset.py b/xarray/core/dataset.py-- a/xarray/core/dataset.py++ b/xarray/core/dataset.py


Question: add average function It would be nice to be able to do to compute weighted averages (e.g. for geo data). Of course this would require the axes to be in a predictable order. Or to give a weight per dimension...

resample,     rolling,     utils,<ins>    weighted,</ins> ) from .alignment import _broadcast_helper, _get_broadcast_dims_map_common_coords, align from .common import (


Question: add average function It would be nice to be able to do to compute weighted averages (e.g. for geo data). Of course this would require the axes to be in a predictable order. Or to give a weight per dimension...

_rolling_cls = rolling.DatasetRolling     _coarsen_cls = rolling.DatasetCoarsen     _resample_cls = resample.DatasetResample    _weighted_cls = weighted.DatasetWeighted      def __init__(         self,diff --git a/xarray/core/weighted.py b/xarray/core/weighted.pynew file mode 100644-- /dev/null++ b/xarray/core/weighted.py


Question: add average function It would be nice to be able to do to compute weighted averages (e.g. for geo data). Of course this would require the axes to be in a predictable order. Or to give a weight per dimension...

<ins>from typing import TYPE_CHECKING, Hashable, Iterable, Optional, Union, overloadfrom .computation import dotfrom .options import _get_keep_attrsif TYPE_CHECKING:    from .dataarray import DataArray, Dataset_WEIGHTED_REDUCE_DOCSTRING_TEMPLATE = """    Reduce this {cls}'s data by a weighted ``{fcn}`` along some dimension(s).    Parameters    ----------    dim : str or sequence of str, optional        Dimension(s) over which to apply the weighted ``{fcn}``.    skipna : bool, optional        If True, skip missing values (as marked by NaN). By default, only        skips missing values for float dtypes; other dtypes either do not        have a sentinel missing value (int) or skipna=True has not been        implemented (object, datetime64 or timedelta64).    keep_attrs : bool, optional        If True, the attributes (``attrs``) will be copied from the original        object to the new one.  If False (default), the new object will be        returned without attributes.    Returns    -------    reduced : {cls}        New {cls} object with weighted ``{fcn}`` applied to its data and        the indicated dimension(s) removed.    Notes    -----        Returns {on_zero} if the ``weights`` sum to 0.0 along the reduced        dimension(s).    """_SUM_OF_WEIGHTS_DOCSTRING = """    Calculate the sum of weights, accounting for missing values in the data    Parameters    ----------    dim : str or sequence of str, optional        Dimension(s) over which to sum the weights.    keep_attrs : bool, optional        If True, the attributes (``attrs``) will be copied from the original        object to the new one.  If False (default), the new object will be        returned without attributes.    Returns    -------    reduced : {cls}        New {cls} object with the sum of the weights over the given dimension.    """class Weighted:    """An object that implements weighted operations.    You should create a Weighted object by using the ``DataArray.weighted`` or    ``Dataset.weighted`` methods.    See Also    --------    Dataset.weighted    DataArray.weighted    """    __slots__ = ("obj", "weights")    @overload    def __init__(self, obj: "DataArray", weights: "DataArray") -> None:        ...    @overload  # noqa: F811    def __init__(self, obj: "Dataset", weights: "DataArray") -> None:  # noqa: F811        ...    def __init__(self, obj, weights):  # noqa: F811        """        Create a Weighted object        Parameters        ----------        obj : DataArray or Dataset            Object over which the weighted reduction operation is applied.        weights : DataArray            An array of weights associated with the values in the obj.            Each value in the obj contributes to the reduction operation            according to its associated weight.        Notes        -----        ``weights`` must be a ``DataArray`` and cannot contain missing values.        Missing values can be replaced by ``weights.fillna(0)``.        """        from .dataarray import DataArray        if not isinstance(weights, DataArray):            raise ValueError("`weights` must be a DataArray")        if weights.isnull().any():            raise ValueError(                "`weights` cannot contain missing values. "                "Missing values can be replaced by `weights.fillna(0)`."            )        self.obj = obj        self.weights = weights    @staticmethod    def _reduce(        da: "DataArray",        weights: "DataArray",        dim: Optional[Union[Hashable, Iterable[Hashable]]] = None,        skipna: Optional[bool] = None,    ) -> "DataArray":        """reduce using dot; equivalent to (da * weights).sum(dim, skipna)            for internal use only        """        # need to infer dims as we use `dot`        if dim is None:            dim = ...        # need to mask invalid values in da, as `dot` does not implement skipna        if skipna or (skipna is None and da.dtype.kind in "cfO"):            da = da.fillna(0.0)        # `dot` does not broadcast arrays, so this avoids creating a large        # DataArray (if `weights` has additional dimensions)        # maybe add fasttrack (`(da * weights).sum(dims=dim, skipna=skipna)`)        return dot(da, weights, dims=dim)    def _sum_of_weights(        self, da: "DataArray", dim: Optional[Union[Hashable, Iterable[Hashable]]] = None    ) -> "DataArray":        """ Calculate the sum of weights, accounting for missing values """        # we need to mask data values that are nan; else the weights are wrong        mask = da.notnull()        sum_of_weights = self._reduce(mask, self.weights, dim=dim, skipna=False)        # 0-weights are not valid        valid_weights = sum_of_weights != 0.0        return sum_of_weights.where(valid_weights)    def _weighted_sum(        self,        da: "DataArray",        dim: Optional[Union[Hashable, Iterable[Hashable]]] = None,        skipna: Optional[bool] = None,    ) -> "DataArray":        """Reduce a DataArray by a by a weighted ``sum`` along some dimension(s)."""        return self._reduce(da, self.weights, dim=dim, skipna=skipna)    def _weighted_mean(        self,        da: "DataArray",        dim: Optional[Union[Hashable, Iterable[Hashable]]] = None,        skipna: Optional[bool] = None,    ) -> "DataArray":        """Reduce a DataArray by a weighted ``mean`` along some dimension(s)."""        weighted_sum = self._weighted_sum(da, dim=dim, skipna=skipna)        sum_of_weights = self._sum_of_weights(da, dim=dim)        return weighted_sum / sum_of_weights    def _implementation(self, func, dim, **kwargs):        raise NotImplementedError("Use `Dataset.weighted` or `DataArray.weighted`")    def sum_of_weights(        self,        dim: Optional[Union[Hashable, Iterable[Hashable]]] = None,        keep_attrs: Optional[bool] = None,    ) -> Union["DataArray", "Dataset"]:        return self._implementation(            self._sum_of_weights, dim=dim, keep_attrs=keep_attrs        )    def sum(        self,        dim: Optional[Union[Hashable, Iterable[Hashable]]] = None,        skipna: Optional[bool] = None,        keep_attrs: Optional[bool] = None,    ) -> Union["DataArray", "Dataset"]:        return self._implementation(            self._weighted_sum, dim=dim, skipna=skipna, keep_attrs=keep_attrs        )    def mean(        self,        dim: Optional[Union[Hashable, Iterable[Hashable]]] = None,        skipna: Optional[bool] = None,        keep_attrs: Optional[bool] = None,    ) -> Union["DataArray", "Dataset"]:        return self._implementation(            self._weighted_mean, dim=dim, skipna=skipna, keep_attrs=keep_attrs        )    def __repr__(self):        """provide a nice str repr of our Weighted object"""        klass = self.__class__.__name__        weight_dims = ", ".join(self.weights.dims)        return f"{klass} with weights along dimensions: {weight_dims}"class DataArrayWeighted(Weighted):    def _implementation(self, func, dim, **kwargs):        keep_attrs = kwargs.pop("keep_attrs")        if keep_attrs is None:            keep_attrs = _get_keep_attrs(default=False)        weighted = func(self.obj, dim=dim, **kwargs)        if keep_attrs:            weighted.attrs = self.obj.attrs        return weightedclass DatasetWeighted(Weighted):    def _implementation(self, func, dim, **kwargs) -> "Dataset":        return self.obj.map(func, dim=dim, **kwargs)def _inject_docstring(cls, cls_name):    cls.sum_of_weights.__doc__ = _SUM_OF_WEIGHTS_DOCSTRING.format(cls=cls_name)    cls.sum.__doc__ = _WEIGHTED_REDUCE_DOCSTRING_TEMPLATE.format(        cls=cls_name, fcn="sum", on_zero="0"    )    cls.mean.__doc__ = _WEIGHTED_REDUCE_DOCSTRING_TEMPLATE.format(        cls=cls_name, fcn="mean", on_zero="NaN"    )_inject_docstring(DataArrayWeighted, "DataArray")_inject_docstring(DatasetWeighted, "Dataset")</ins>


Question: REGRESSION: copy(deep=True) casts unicode indices to object (deep=True) and (deep=True/False) accidentally cast IndexVariable's with dtype='<U*' to object. Same applies to () and copy.deepcopy(). This is a regression in xarray = 0.12.2. xarray 0.12.1 and earlier are unaffected.

from collections import defaultdict from contextlib import suppress from datetime import timedeltafrom typing import Sequencefrom typing import Any, Tuple, Sequence, Union  import numpy as np import pandas as pd  from . import duck_array_ops, nputils, utilsfrom .npcompat import DTypeLike from .pycompat import dask_array_type, integer_types from .utils import is_dict_like


Question: REGRESSION: copy(deep=True) casts unicode indices to object (deep=True) and (deep=True/False) accidentally cast IndexVariable's with dtype='<U*' to object. Same applies to () and copy.deepcopy(). This is a regression in xarray = 0.12.2. xarray 0.12.1 and earlier are unaffected.

class PandasIndexAdapter(ExplicitlyIndexedNDArrayMixin):    """Wrap a pandas.Index to preserve dtypes and handle explicit indexing."""    """Wrap a pandas.Index to preserve dtypes and handle explicit indexing.    """     def __init__(self, array, dtype=None):    def __init__(self, array: Any, dtype: DTypeLike = None):         self.array = utils.safe_cast_to_index(array)         if dtype is None:             if isinstance(array, pd.PeriodIndex):