Question: elif XX.shape != (X.shape[0], 1):             raise ValueError(                 "Incompatible dimensions for X and X_norm_squared")        if XX.dtype == np.float32:            XX = None    elif X.dtype == np.float32:        XX = None     else:         XX = row_norms(X, squared=True)[:, np.newaxis]     if X is Y:  # shortcut in the common case euclidean_distances(X, X)    if X is Y and XX is not None:        # shortcut in the common case euclidean_distances(X, X)         YY = XX.T     elif Y_norm_squared is not None:         YY = np.atleast_2d(Y_norm_squared)


Question: if YY.shape != (1, Y.shape[0]):             raise ValueError(                 "Incompatible dimensions for Y and Y_norm_squared")        if YY.dtype == np.float32:            YY = None    elif Y.dtype == np.float32:        YY = None     else:         YY = row_norms(Y, squared=True)[np.newaxis, :]     distances = safe_sparse_dot(X, Y.T, dense_output=True)    distances *= -2    distances += XX    distances += YY    if X.dtype == np.float32:        # To minimize precision issues with float32, we compute the distance        # matrix on chunks of X and Y upcast to float64        distances = _euclidean_distances_upcast(X, XX, Y, YY)    else:        # if dtype is already float64, no need to chunk and upcast        distances = - 2 * safe_sparse_dot(X, Y.T, dense_output=True)        distances += XX        distances += YY     np.maximum(distances, 0, out=distances)     # Ensure that distances between vectors and themselves are set to 0.0.    # This may not be the case due to floating point rounding errors.     if X is Y:        # Ensure that distances between vectors and themselves are set to 0.0.        # This may not be the case due to floating point rounding errors.        distances.flat[::distances.shape[0] + 1] = 0.0        np.fill_diagonal(distances, 0)      return distances if squared else np.sqrt(distances, out=distances)  def _euclidean_distances_upcast(X, XX=None, Y=None, YY=None):    """Euclidean distances between X and Y    Assumes X and Y have float32 dtype.    Assumes XX and YY have float64 dtype or are None.    X and Y are upcast to float64 by chunks, which size is chosen to limit    memory increase by approximately 10% (at least 10MiB).    """    n_samples_X = X.shape[0]    n_samples_Y = Y.shape[0]    n_features = X.shape[1]    distances = np.empty((n_samples_X, n_samples_Y), dtype=np.float32)    x_density = X.nnz / np.prod(X.shape) if issparse(X) else 1    y_density = Y.nnz / np.prod(Y.shape) if issparse(Y) else 1    # Allow 10% more memory than X, Y and the distance matrix take (at least    # 10MiB)    maxmem = max(        ((x_density * n_samples_X + y_density * n_samples_Y) * n_features         + (x_density * n_samples_X * y_density * n_samples_Y)) / 10,        10 * 2**17)    # The increase amount of memory in 8-byte blocks is:    # - x_density * batch_size * n_features (copy of chunk of X)    # - y_density * batch_size * n_features (copy of chunk of Y)    # - batch_size * batch_size (chunk of distance matrix)    # Hence x² + (xd+yd)kx = M, where x=batch_size, k=n_features, M=maxmem    #                                 xd=x_density and yd=y_density    tmp = (x_density + y_density) * n_features    batch_size = (-tmp + np.sqrt(tmp**2 + 4 * maxmem)) / 2    batch_size = max(int(batch_size), 1)    x_batches = gen_batches(X.shape[0], batch_size)    y_batches = gen_batches(Y.shape[0], batch_size)    for i, x_slice in enumerate(x_batches):        X_chunk = X[x_slice].astype(np.float64)        if XX is None:            XX_chunk = row_norms(X_chunk, squared=True)[:, np.newaxis]        else:            XX_chunk = XX[x_slice]        for j, y_slice in enumerate(y_batches):            if X is Y and j < i:                # when X is Y the distance matrix is symmetric so we only need                # to compute half of it.                d = distances[y_slice, x_slice].T            else:                Y_chunk = Y[y_slice].astype(np.float64)                if YY is None:                    YY_chunk = row_norms(Y_chunk, squared=True)[np.newaxis, :]                else:                    YY_chunk = YY[:, y_slice]                d = -2 * safe_sparse_dot(X_chunk, Y_chunk.T, dense_output=True)                d += XX_chunk                d += YY_chunk            distances[x_slice, y_slice] = d.astype(np.float32, copy=False)    return distances def _argmin_min_reduce(dist, start):     indices = dist.argmin(axis=1)     values = dist[np.arange(dist.shape[0]), indices]

Numerical precision of euclideandistances with float32 <!-- Instructions For Filing a Bug: --I noticed that sklearn.metrics.pairwise.pairwisedistances function agrees with when using np.float64 arrays, but disagrees when using np.float32 arrays. See the code snippet below. I expect that the results from sklearn.metrics.pairwise.pairwisedistances would agree with for both 64-bit and 32-bit. In other words, I expect the following output: The code snippet above produces the following output for me: [WIP] Stable and fast float32 implementation of euclideandistances Superseds PR These commits implement a block-wise casting to float64 and uses the older code to compute the euclidean distance matrix on the blocks. This is done useing only a fixed amount of additional (temporary) memory. This code implements several optimizations: since the distance matrix is symmetric when , copy the blocks of the upper triangle to the lower triangle; compute the optimal block size that would use most of the allowed additional memory; cast blocks of to float64; precompute blocks of if not given so it gets reused through the iterations over ; swap and when is given, but not . Note that all the optimizations listed here have proven useful in a benchmark. The hardcoded amount of additional memory of 10MB is also derived from a benchmark. As a side bonus, this implementation should also support float16 out of the box, should scikit-learn support it at some point. Add a test for numeric precision (see ) Surprisingly bad precision, isn't it? Note that the traditional computation sqrt(sum((x-y)*2)) gets the results exact. <!-- Thanks for contributing a pull request! Please ensure you have taken a look at the contribution guidelines: --<!-- Example: See also . Please use keywords (e.g., Fixes) to create link to the issues or pull requests you resolved, so that they will automatically be closed when your pull request is merged. See --<!-- Please be aware that we are a loose team of volunteers so patience is necessary; assistance handling other issues is very welcome. We value all user contributions, no matter how minor they are. If we are slow to review, either the pull request needs some benchmarking, tinkering, convincing, etc. or more likely the reviewers are simply busy.


Question: if YY.shape != (1, Y.shape[0]):             raise ValueError(                 "Incompatible dimensions for Y and Y_norm_squared")        if YY.dtype == np.float32:            YY = None    elif Y.dtype == np.float32:        YY = None     else:         YY = row_norms(Y, squared=True)[np.newaxis, :]     distances = safe_sparse_dot(X, Y.T, dense_output=True)    distances *= -2    distances += XX    distances += YY    if X.dtype == np.float32:        # To minimize precision issues with float32, we compute the distance        # matrix on chunks of X and Y upcast to float64        distances = _euclidean_distances_upcast(X, XX, Y, YY)    else:        # if dtype is already float64, no need to chunk and upcast        distances = - 2 * safe_sparse_dot(X, Y.T, dense_output=True)        distances += XX        distances += YY     np.maximum(distances, 0, out=distances)     # Ensure that distances between vectors and themselves are set to 0.0.    # This may not be the case due to floating point rounding errors.     if X is Y:        # Ensure that distances between vectors and themselves are set to 0.0.        # This may not be the case due to floating point rounding errors.        distances.flat[::distances.shape[0] + 1] = 0.0        np.fill_diagonal(distances, 0)      return distances if squared else np.sqrt(distances, out=distances)  def _euclidean_distances_upcast(X, XX=None, Y=None, YY=None):    """Euclidean distances between X and Y    Assumes X and Y have float32 dtype.    Assumes XX and YY have float64 dtype or are None.    X and Y are upcast to float64 by chunks, which size is chosen to limit    memory increase by approximately 10% (at least 10MiB).    """    n_samples_X = X.shape[0]    n_samples_Y = Y.shape[0]    n_features = X.shape[1]    distances = np.empty((n_samples_X, n_samples_Y), dtype=np.float32)    x_density = X.nnz / np.prod(X.shape) if issparse(X) else 1    y_density = Y.nnz / np.prod(Y.shape) if issparse(Y) else 1    # Allow 10% more memory than X, Y and the distance matrix take (at least    # 10MiB)    maxmem = max(        ((x_density * n_samples_X + y_density * n_samples_Y) * n_features         + (x_density * n_samples_X * y_density * n_samples_Y)) / 10,        10 * 2**17)    # The increase amount of memory in 8-byte blocks is:    # - x_density * batch_size * n_features (copy of chunk of X)    # - y_density * batch_size * n_features (copy of chunk of Y)    # - batch_size * batch_size (chunk of distance matrix)    # Hence x² + (xd+yd)kx = M, where x=batch_size, k=n_features, M=maxmem    #                                 xd=x_density and yd=y_density    tmp = (x_density + y_density) * n_features    batch_size = (-tmp + np.sqrt(tmp**2 + 4 * maxmem)) / 2    batch_size = max(int(batch_size), 1)    x_batches = gen_batches(X.shape[0], batch_size)    y_batches = gen_batches(Y.shape[0], batch_size)    for i, x_slice in enumerate(x_batches):        X_chunk = X[x_slice].astype(np.float64)        if XX is None:            XX_chunk = row_norms(X_chunk, squared=True)[:, np.newaxis]        else:            XX_chunk = XX[x_slice]        for j, y_slice in enumerate(y_batches):            if X is Y and j < i:                # when X is Y the distance matrix is symmetric so we only need                # to compute half of it.                d = distances[y_slice, x_slice].T            else:                Y_chunk = Y[y_slice].astype(np.float64)                if YY is None:                    YY_chunk = row_norms(Y_chunk, squared=True)[np.newaxis, :]                else:                    YY_chunk = YY[:, y_slice]                d = -2 * safe_sparse_dot(X_chunk, Y_chunk.T, dense_output=True)                d += XX_chunk                d += YY_chunk            distances[x_slice, y_slice] = d.astype(np.float32, copy=False)    return distances def _argmin_min_reduce(dist, start):     indices = dist.argmin(axis=1)     values = dist[np.arange(dist.shape[0]), indices]

In either case, we ask for your understanding during the review process. For more information, see our FAQ on this topic: http://scikit- Thanks for contributing! --


Question: """============================================Curve Fitting with Bayesian Ridge Regression============================================Computes a Bayesian Ridge Regression of Sinusoids.See :ref:`bayesian_ridge_regression` for more information on the regressor.In general, when fitting a curve with a polynomial by Bayesian ridgeregression, the selection of initial values ofthe regularization parameters (alpha, lambda) may be important.This is because the regularization parameters are determined by an iterativeprocedure that depends on initial values.In this example, the sinusoid is approximated by a polynomial using differentpairs of initial values.When starting from the default values (alpha_init = 1.90, lambda_init = 1.),the bias of the resulting curve is large, and the variance is small.So, lambda_init should be relatively small (1.e-3) so as to reduce the bias.Also, by evaluating log marginal likelihood (L) ofthese models, we can determine which one is better.It can be concluded that the model with larger L is more likely."""print(__doc__)# Author: Yoshihiro Uchida <nimbus1after2a1sun7shower@gmail.com>import numpy as npimport matplotlib.pyplot as pltfrom sklearn.linear_model import BayesianRidgedef func(x): return np.sin(2*np.pi*x)# ############################################################################## Generate sinusoidal data with noisesize = 25rng = np.random.RandomState(1234)x_train = rng.uniform(0., 1., size)y_train = func(x_train) + rng.normal(scale=0.1, size=size)x_test = np.linspace(0., 1., 100)# ############################################################################## Fit by cubic polynomialn_order = 3X_train = np.vander(x_train, n_order + 1, increasing=True)X_test = np.vander(x_test, n_order + 1, increasing=True)# ############################################################################## Plot the true and predicted curves with log marginal likelihood (L)reg = BayesianRidge(tol=1e-6, fit_intercept=False, compute_score=True)fig, axes = plt.subplots(1, 2, figsize=(8, 4))for i, ax in enumerate(axes):    # Bayesian ridge regression with different initial value pairs    if i == 0:        init = [1 / np.var(y_train), 1.]  # Default values    elif i == 1:        init = [1., 1e-3]        reg.set_params(alpha_init=init[0], lambda_init=init[1])    reg.fit(X_train, y_train)    ymean, ystd = reg.predict(X_test, return_std=True)    ax.plot(x_test, func(x_test), color="blue", label="sin($2\\pi x$)")    ax.scatter(x_train, y_train, s=50, alpha=0.5, label="observation")    ax.plot(x_test, ymean, color="red", label="predict mean")    ax.fill_between(x_test, ymean-ystd, ymean+ystd,                    color="pink", alpha=0.5, label="predict std")    ax.set_ylim(-1.3, 1.3)    ax.legend()    title = "$\\alpha$_init$={:.2f},\\ \\lambda$_init$={}$".format(            init[0], init[1])    if i == 0:        title += " (Default)"    ax.set_title(title, fontsize=12)    text = "$\\alpha={:.1f}$\n$\\lambda={:.3f}$\n$L={:.1f}$".format(           reg.alpha_, reg.lambda_, reg.scores_[-1])    ax.text(0.05, -1.0, text, fontsize=12)plt.tight_layout()plt.show()diff --git a/sklearn/linear_model/bayes.py b/sklearn/linear_model/bayes.py-- a/sklearn/linear_model/bayes.py++ b/sklearn/linear_model/bayes.py

Allow setting of initial hyperparameters of BayesianRidge <!-- If your issue is a usage question, submit it here instead: StackOverflow with the scikit-learn tag: Mailing List: For more information, see User Questions: http://scikit- --<!-- Instructions For Filing a Bug: --<!-- Example: Joblib Error thrown when calling fit on LatentDirichletAllocation with evaluateevery 0--I propose to be able to set initial values of the hyperparameters (alpha, lambda) in BayesianRidge().fit. I tried to fit a sinusoidal curve with polynomials by Bayesian ridge regression, but the default initial values did not work well (left figure). So, I corrected the fit method of the BayesianRidge class so that I could set the initial value, and the regression worked well (right figure). ! <!-- Example: If the code is too long, feel free to put it in a public gist and link it in the issue: --The code I wrote is Here. <!-- Example: No error is thrown. Please paste or describe the expected results.--The right figure <!-- Please paste or specifically describe the actual output or traceback. --The left figure <!-- Please run the following snippet and paste the output below.


Question: """============================================Curve Fitting with Bayesian Ridge Regression============================================Computes a Bayesian Ridge Regression of Sinusoids.See :ref:`bayesian_ridge_regression` for more information on the regressor.In general, when fitting a curve with a polynomial by Bayesian ridgeregression, the selection of initial values ofthe regularization parameters (alpha, lambda) may be important.This is because the regularization parameters are determined by an iterativeprocedure that depends on initial values.In this example, the sinusoid is approximated by a polynomial using differentpairs of initial values.When starting from the default values (alpha_init = 1.90, lambda_init = 1.),the bias of the resulting curve is large, and the variance is small.So, lambda_init should be relatively small (1.e-3) so as to reduce the bias.Also, by evaluating log marginal likelihood (L) ofthese models, we can determine which one is better.It can be concluded that the model with larger L is more likely."""print(__doc__)# Author: Yoshihiro Uchida <nimbus1after2a1sun7shower@gmail.com>import numpy as npimport matplotlib.pyplot as pltfrom sklearn.linear_model import BayesianRidgedef func(x): return np.sin(2*np.pi*x)# ############################################################################## Generate sinusoidal data with noisesize = 25rng = np.random.RandomState(1234)x_train = rng.uniform(0., 1., size)y_train = func(x_train) + rng.normal(scale=0.1, size=size)x_test = np.linspace(0., 1., 100)# ############################################################################## Fit by cubic polynomialn_order = 3X_train = np.vander(x_train, n_order + 1, increasing=True)X_test = np.vander(x_test, n_order + 1, increasing=True)# ############################################################################## Plot the true and predicted curves with log marginal likelihood (L)reg = BayesianRidge(tol=1e-6, fit_intercept=False, compute_score=True)fig, axes = plt.subplots(1, 2, figsize=(8, 4))for i, ax in enumerate(axes):    # Bayesian ridge regression with different initial value pairs    if i == 0:        init = [1 / np.var(y_train), 1.]  # Default values    elif i == 1:        init = [1., 1e-3]        reg.set_params(alpha_init=init[0], lambda_init=init[1])    reg.fit(X_train, y_train)    ymean, ystd = reg.predict(X_test, return_std=True)    ax.plot(x_test, func(x_test), color="blue", label="sin($2\\pi x$)")    ax.scatter(x_train, y_train, s=50, alpha=0.5, label="observation")    ax.plot(x_test, ymean, color="red", label="predict mean")    ax.fill_between(x_test, ymean-ystd, ymean+ystd,                    color="pink", alpha=0.5, label="predict std")    ax.set_ylim(-1.3, 1.3)    ax.legend()    title = "$\\alpha$_init$={:.2f},\\ \\lambda$_init$={}$".format(            init[0], init[1])    if i == 0:        title += " (Default)"    ax.set_title(title, fontsize=12)    text = "$\\alpha={:.1f}$\n$\\lambda={:.3f}$\n$L={:.1f}$".format(           reg.alpha_, reg.lambda_, reg.scores_[-1])    ax.text(0.05, -1.0, text, fontsize=12)plt.tight_layout()plt.show()diff --git a/sklearn/linear_model/bayes.py b/sklearn/linear_model/bayes.py-- a/sklearn/linear_model/bayes.py++ b/sklearn/linear_model/bayes.py

For scikit-learn = 0.20: import sklearn; sklearn.showversions() For scikit-learn < 0.20: import platform; print(platform.platform()) import sys; print("Python", sys.version) import numpy; print("NumPy", numpy.version) import scipy; print("SciPy", scipy.version) import sklearn; print("Scikit-Learn", sklearn.version) --System: python: 3.6.6 (default, Mar 8 2019, 18:24:30) [GCC 7.3.0] executable: /home/*/.pyenv/versions/3.6.6/bin/python machine: Linux-4.15.0-47-generic-x8664-with-debian-buster-sid BLAS: macros: NOATLASINFO=1, HAVECBLAS=None libdirs: /usr/lib/x8664-linux-gnu cblaslibs: cblas Python deps: pip: 19.0.3 setuptools: 41.0.0 sklearn: 0.20.3 numpy: 1.16.2 scipy: 1.2.1 Cython: 0.29.6 pandas: 0.24.2 <!-- Thanks for contributing! --


Question: Gamma distribution prior over the lambda parameter.         Default is 1.e-6 <ins>    alpha_init : float        Initial value for alpha (precision of the noise).        If not set, alpha_init is 1/Var(y).            .. versionadded:: 0.22    lambda_init : float        Initial value for lambda (precision of the weights).        If not set, lambda_init is 1.            .. versionadded:: 0.22</ins>     compute_score : boolean, optional         If True, compute the log marginal likelihood at each iteration of the         optimization. Default is False.


Question: Gamma distribution prior over the lambda parameter.         Default is 1.e-6 <ins>    alpha_init : float        Initial value for alpha (precision of the noise).        If not set, alpha_init is 1/Var(y).            .. versionadded:: 0.22    lambda_init : float        Initial value for lambda (precision of the weights).        If not set, lambda_init is 1.            .. versionadded:: 0.22</ins>     compute_score : boolean, optional         If True, compute the log marginal likelihood at each iteration of the         optimization. Default is False.


Question: >>> clf = linear_model.BayesianRidge()     >>> clf.fit([[0,0], [1, 1], [2, 2]], [0, 1, 2])     ... # doctest: +NORMALIZE_WHITESPACE    BayesianRidge(alpha_1=1e-06, alpha_2=1e-06, compute_score=False,            copy_X=True, fit_intercept=True, lambda_1=1e-06, lambda_2=1e-06,            n_iter=300, normalize=False, tol=0.001, verbose=False)    BayesianRidge(alpha_1=1e-06, alpha_2=1e-06, alpha_init=None,                  compute_score=False, copy_X=True, fit_intercept=True,                  lambda_1=1e-06, lambda_2=1e-06, lambda_init=None, n_iter=300,                  normalize=False, tol=0.001, verbose=False)     >>> clf.predict([[1, 1]])     array([1.])


Question: >>> clf = linear_model.BayesianRidge()     >>> clf.fit([[0,0], [1, 1], [2, 2]], [0, 1, 2])     ... # doctest: +NORMALIZE_WHITESPACE    BayesianRidge(alpha_1=1e-06, alpha_2=1e-06, compute_score=False,            copy_X=True, fit_intercept=True, lambda_1=1e-06, lambda_2=1e-06,            n_iter=300, normalize=False, tol=0.001, verbose=False)    BayesianRidge(alpha_1=1e-06, alpha_2=1e-06, alpha_init=None,                  compute_score=False, copy_X=True, fit_intercept=True,                  lambda_1=1e-06, lambda_2=1e-06, lambda_init=None, n_iter=300,                  normalize=False, tol=0.001, verbose=False)     >>> clf.predict([[1, 1]])     array([1.])


Question: """      def __init__(self, n_iter=300, tol=1.e-3, alpha_1=1.e-6, alpha_2=1.e-6,                 lambda_1=1.e-6, lambda_2=1.e-6, compute_score=False,                 fit_intercept=True, normalize=False, copy_X=True,                 verbose=False):                 lambda_1=1.e-6, lambda_2=1.e-6, alpha_init=None,                 lambda_init=None, compute_score=False, fit_intercept=True,                 normalize=False, copy_X=True, verbose=False):         self.n_iter = n_iter         self.tol = tol         self.alpha_1 = alpha_1         self.alpha_2 = alpha_2         self.lambda_1 = lambda_1         self.lambda_2 = lambda_2        self.alpha_init = alpha_init        self.lambda_init = lambda_init         self.compute_score = compute_score         self.fit_intercept = fit_intercept         self.normalize = normalize


Question: """      def __init__(self, n_iter=300, tol=1.e-3, alpha_1=1.e-6, alpha_2=1.e-6,                 lambda_1=1.e-6, lambda_2=1.e-6, compute_score=False,                 fit_intercept=True, normalize=False, copy_X=True,                 verbose=False):                 lambda_1=1.e-6, lambda_2=1.e-6, alpha_init=None,                 lambda_init=None, compute_score=False, fit_intercept=True,                 normalize=False, copy_X=True, verbose=False):         self.n_iter = n_iter         self.tol = tol         self.alpha_1 = alpha_1         self.alpha_2 = alpha_2         self.lambda_1 = lambda_1         self.lambda_2 = lambda_2        self.alpha_init = alpha_init        self.lambda_init = lambda_init         self.compute_score = compute_score         self.fit_intercept = fit_intercept         self.normalize = normalize


Question: eps = np.finfo(np.float64).eps         # Add `eps` in the denominator to omit division by zero if `np.var(y)`         # is zero        alpha_ = 1. / (np.var(y) + eps)        lambda_ = 1.        alpha_ = self.alpha_init        lambda_ = self.lambda_init        if alpha_ is None:            alpha_ = 1. / (np.var(y) + eps)        if lambda_ is None:            lambda_ = 1.          verbose = self.verbose         lambda_1 = self.lambda_1


Question: eps = np.finfo(np.float64).eps         # Add `eps` in the denominator to omit division by zero if `np.var(y)`         # is zero        alpha_ = 1. / (np.var(y) + eps)        lambda_ = 1.        alpha_ = self.alpha_init        lambda_ = self.lambda_init        if alpha_ is None:            alpha_ = 1. / (np.var(y) + eps)        if lambda_ is None:            lambda_ = 1.          verbose = self.verbose         lambda_1 = self.lambda_1


Question: Returns         -------         feature_importances_ : array, shape (n_features,)            The values of this array sum to 1, unless all trees are single node            trees consisting of only the root node, in which case it will be an            array of zeros.         """         self._check_initialized()         total_sum = np.zeros((self.n_features_, ), dtype=np.float64)        for stage in self.estimators_:            stage_sum = sum(tree.tree_.compute_feature_importances(                normalize=False) for tree in stage) / len(stage)            total_sum += stage_sum        importances = total_sum / total_sum.sum()        return importances        relevant_trees = [tree                          for stage in self.estimators_ for tree in stage                          if tree.tree_.node_count > 1]        if not relevant_trees:            # degenerate case where all trees have only one node            return np.zeros(shape=self.n_features_, dtype=np.float64)        relevant_feature_importances = [            tree.tree_.compute_feature_importances(normalize=False)            for tree in relevant_trees        ]        avg_feature_importances = np.mean(relevant_feature_importances,                                          axis=0, dtype=np.float64)        return avg_feature_importances / np.sum(avg_feature_importances)      def _validate_y(self, y, sample_weight):         # 'sample_weight' is not utilised but is used for

Bug in Gradient Boosting: Feature Importances do not sum to 1 I found conditions when Feature Importance values do not add up to 1 in ensemble tree methods, like Gradient Boosting Trees or AdaBoost Trees. This error occurs once the ensemble reaches a large number of estimators. The exact conditions depend variously. For example, the error shows up sooner with a smaller amount of training samples. Or, if the depth of the tree is large. When this error appears, the predicted value seems to have converged. But it’s unclear if the error is causing the predicted value not to change with more estimators. In fact, the feature importance sum goes lower and lower with more estimators thereafter. Consequently, it's questionable if the tree ensemble code is functioning as expected. Here's sample code to reproduce this: Output: In fact, if we examine the tree at each staged prediction, we'll see that the feature importance goes to 0 after we hit a certain number of estimators. (For the code above, it's 712.) Here's code to describe what I mean: Output: I wonder if we’re hitting some floating point calculation error. BTW, I've posted this issue on the mailing list . There aren't a lot of discussion, but others seem to think there's a bug here too. Hope we can get this fixed or clarified. Thank you! -Doug Windows-7;'Python', '2.7.9 ;'NumPy', '1.9.2';'SciPy', '0.15.1';'Scikit-Learn', '0.16.1'


Question: from .isotonic import IsotonicRegression from .svm import LinearSVC from .model_selection import check_cv<del>from .metrics.classification import _check_binary_probabilistic_predictions</del>   class CalibratedClassifierCV(BaseEstimator, ClassifierMixin):

brierscoreloss error Hello. I think I found a mistake in brierscoreloss. When you have a target = 1 and a prediction = 1 too, brierscoreloss should be 0 (best result), but it gives 1. Why is it happening? Because checkbinaryprobabilisticpredictions gets target with only one class and convert it to 0. And metric calculates for target = 0 and prediction = 1. The same problem for target = 1 and prediction = 0. brierscoreloss is 0 (the best result), but it should be 1. Examples: Approx = [0, 0, 0, 0] Target = [1, 1, 1, 1] Weight = [1, 1, 1, 1] brierscoreloss(Target, Approx, sampleweight=Weight) result is 0 Approx = [1, 1, 1, 1] Target = [1, 1, 1, 1] Weight = [1, 1, 1, 1] brierscoreloss(Target, Approx, sampleweight=Weight) result is 1 Maybe we should fix it? Thank you.


Question: """     y_true = column_or_1d(y_true)     y_prob = column_or_1d(y_prob)<ins>    check_consistent_length(y_true, y_prob)</ins>      if normalize:  # Normalize predicted values into interval [0, 1]         y_prob = (y_prob - y_prob.min()) / (y_prob.max() - y_prob.min())


Question: raise ValueError("y_prob has values outside [0, 1] and normalize is "                          "set to False.")     y_true = _check_binary_probabilistic_predictions(y_true, y_prob)    labels = np.unique(y_true)    if len(labels) > 2:        raise ValueError("Only binary classification is supported. "                         "Provided labels %s." % labels)    y_true = label_binarize(y_true, labels)[:, 0]      if strategy == 'quantile':  # Determine bin edges by distribution of data         quantiles = np.linspace(0, 1, n_bins + 1)diff --git a/sklearn/metrics/classification.py b/sklearn/metrics/classification.py-- a/sklearn/metrics/classification.py++ b/sklearn/metrics/classification.py


Question: from scipy.sparse import coo_matrix from scipy.sparse import csr_matrix from ..preprocessing import LabelBinarizer, label_binarizefrom ..preprocessing import LabelBinarizer from ..preprocessing import LabelEncoder from ..utils import assert_all_finite from ..utils import check_array


Question: return np.average(losses, weights=sample_weight)  <del>def _check_binary_probabilistic_predictions(y_true, y_prob):    """Check that y_true is binary and y_prob contains valid probabilities"""    check_consistent_length(y_true, y_prob)    labels = np.unique(y_true)    if len(labels) > 2:        raise ValueError("Only binary classification is supported. "                         "Provided labels %s." % labels)    if y_prob.max() > 1:        raise ValueError("y_prob contains values greater than 1.")    if y_prob.min() < 0:        raise ValueError("y_prob contains values less than 0.")    return label_binarize(y_true, labels)[:, 0]</del> def brier_score_loss(y_true, y_prob, sample_weight=None, pos_label=None):     """Compute the Brier score.     The smaller the Brier score, the better, hence the naming with "loss".


Question: Sample weights.      pos_label : int or str, default=None        Label of the positive class. If None, the maximum label is used as        positive class        Label of the positive class.        Defaults to the greater label unless y_true is all 0 or all -1        in which case pos_label defaults to 1.      Returns     -------


Question: assert_all_finite(y_prob)     check_consistent_length(y_true, y_prob, sample_weight)     labels = np.unique(y_true)    if len(labels) > 2:        raise ValueError("Only binary classification is supported. "                         "Labels in y_true: %s." % labels)    if y_prob.max() > 1:        raise ValueError("y_prob contains values greater than 1.")    if y_prob.min() < 0:        raise ValueError("y_prob contains values less than 0.")    # if pos_label=None, when y_true is in {-1, 1} or {0, 1},    # pos_labe is set to 1 (consistent with precision_recall_curve/roc_curve),    # otherwise pos_label is set to the greater label    # (different from precision_recall_curve/roc_curve,    # the purpose is to keep backward compatibility).     if pos_label is None:        pos_label = y_true.max()        if (np.array_equal(labels, [0]) or                np.array_equal(labels, [-1])):            pos_label = 1        else:            pos_label = y_true.max()     y_true = np.array(y_true == pos_label, int)    y_true = _check_binary_probabilistic_predictions(y_true, y_prob)     return np.average((y_true - y_prob) ** 2, weights=sample_weight)diff --git a/sklearn/metrics/ranking.py b/sklearn/metrics/ranking.py-- a/sklearn/metrics/ranking.py++ b/sklearn/metrics/ranking.py


Question: Parameters     ----------     y_true : array, shape = [n_samples]        True targets of binary classification in range {-1, 1} or {0, 1}.        True binary labels. If labels are not either {-1, 1} or {0, 1}, then        pos_label should be explicitly given.      probas_pred : array, shape = [n_samples]         Estimated probabilities or decision function.      pos_label : int or str, default=None        The label of the positive class        The label of the positive class.        When ``pos_label=None``, if y_true is in {-1, 1} or {0, 1},        ``pos_label`` is set to 1, otherwise an error will be raised.      sample_weight : array-like of shape = [n_samples], optional         Sample weights.


Question: (as returned by "decision_function" on some classifiers).      pos_label : int or str, default=None        Label considered as positive and others are considered negative.        The label of the positive class.        When ``pos_label=None``, if y_true is in {-1, 1} or {0, 1},        ``pos_label`` is set to 1, otherwise an error will be raised.      sample_weight : array-like of shape = [n_samples], optional         Sample weights.


Question: from ..utils.validation import check_is_fitted, check_array, FLOAT_DTYPES from ..utils import _IS_32BIT from ..utils.fixes import _astype_copy_false<ins>from ..exceptions import ChangedBehaviorWarning</ins>   __all__ = ['HashingVectorizer',

CountVectorizer with custom analyzer ignores input argument Example: Same for . Not sure if this should be fixed or just documented; I don't like changing the behavior of the vectorizers yet again...


Question: self._stop_words_id = id(self.stop_words)             return 'error'     def _validate_custom_analyzer(self):        # This is to check if the given custom analyzer expects file or a        # filename instead of data.        # Behavior changed in v0.21, function could be removed in v0.23        import tempfile        with tempfile.NamedTemporaryFile() as f:            fname = f.name        # now we're sure fname doesn't exist        msg = ("Since v0.21, vectorizers pass the data to the custom analyzer "               "and not the file names or the file objects. This warning "               "will be removed in v0.23.")        try:            self.analyzer(fname)        except FileNotFoundError:            warnings.warn(msg, ChangedBehaviorWarning)        except AttributeError as e:            if str(e) == "'str' object has no attribute 'read'":                warnings.warn(msg, ChangedBehaviorWarning)        except Exception:            pass     def build_analyzer(self):         """Return a callable that handles preprocessing and tokenization"""         if callable(self.analyzer):            return self.analyzer            if self.input in ['file', 'filename']:                self._validate_custom_analyzer()            return lambda doc: self.analyzer(self.decode(doc))          preprocess = self.build_preprocessor()


Question: If a callable is passed it is used to extract the sequence of features         out of the raw, unprocessed input. <ins>        .. versionchanged:: 0.21        Since v0.21, if ``input`` is ``filename`` or ``file``, the data is        first read from the file and then passed to the given callable        analyzer.</ins>     n_features : integer, default=(2 ** 20)         The number of features (columns) in the output matrices. Small numbers         of features are likely to cause hash collisions, but large numbers


Question: If a callable is passed it is used to extract the sequence of features         out of the raw, unprocessed input. <ins>        .. versionchanged:: 0.21        Since v0.21, if ``input`` is ``filename`` or ``file``, the data is        first read from the file and then passed to the given callable        analyzer.</ins>     max_df : float in range [0.0, 1.0] or int, default=1.0         When building the vocabulary ignore terms that have a document         frequency strictly higher than the given threshold (corpus-specific


Question: If a callable is passed it is used to extract the sequence of features         out of the raw, unprocessed input. <ins>        .. versionchanged:: 0.21        Since v0.21, if ``input`` is ``filename`` or ``file``, the data is        first read from the file and then passed to the given callable        analyzer.</ins>     stop_words : string {'english'}, list, or None (default=None)         If a string, it is passed to _check_stop_list and the appropriate stop         list is returned. 'english' is currently the only supported string


Question: from ..base import BaseEstimator from .base import SelectorMixin from ..utils import check_arrayfrom ..utils.sparsefuncs import mean_variance_axisfrom ..utils.sparsefuncs import mean_variance_axis, min_max_axis from ..utils.validation import check_is_fitted

VarianceThreshold doesn't remove feature with zero variance When calling VarianceThreshold().fittransform() on certain inputs, it fails to remove a column that has only one unique value. The Variance threshold should produce This issue arose when I was using VarianceThreshold on a real dataset (of which this is a subset). It appears to work correctly in other situations (for instance I can't reproduce this behaviour if I replace the first column with 1's). System python: 3.5.6 (default, Aug 26 2018, 16:30:03) [GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE401/final)] machine: Darwin-18.2.0-x8664-i386-64bit executable: /anaconda3/envs/tensorflow/bin/python3 BLAS macros: NOATLASINFO=3, HAVECBLAS=None libdirs: cblaslibs: cblas Python deps setuptools: 40.2.0 numpy: 1.15.4 sklearn: 0.20.0 Cython: None scipy: 1.1.0 pandas: 0.24.0 pip: 19.0.1


Question: if hasattr(X, "toarray"):   # sparse matrix             _, self.variances_ = mean_variance_axis(X, axis=0)            if self.threshold == 0:                mins, maxes = min_max_axis(X, axis=0)                peak_to_peaks = maxes - mins         else:             self.variances_ = np.var(X, axis=0)            if self.threshold == 0:                peak_to_peaks = np.ptp(X, axis=0)        if self.threshold == 0:            # Use peak-to-peak to avoid numeric precision issues            # for constant features            self.variances_ = np.minimum(self.variances_, peak_to_peaks)          if np.all(self.variances_ <= self.threshold):             msg = "No feature in X meets the variance threshold {0:.5f}"


Question: to be installed. It can be faster on very large, sparse problems,         but may also lead to instabilities. <ins>    n_components : integer, optional, default=n_clusters        Number of eigen vectors to use for the spectral embedding</ins>     random_state : int, RandomState instance or None (default)         A pseudo random number generator used for the initialization of the         lobpcg eigen vectors decomposition when ``eigen_solver='amg'`` and by

ncomponents kwarg missing in SpectralClustering The kwarg defined in the function allow the user to choose how many eigenvalues/eigenvectors should be used in the classification. However this kwarg cannot be accessed/modified when using class, and is set to default ( = ). Since the class calls the spectralclustering function, I guess the user should be able to access this kwarg ?


Question: >>> clustering # doctest: +NORMALIZE_WHITESPACE     SpectralClustering(affinity='rbf', assign_labels='discretize', coef0=1,               degree=3, eigen_solver=None, eigen_tol=0.0, gamma=1.0,              kernel_params=None, n_clusters=2, n_init=10, n_jobs=None,              n_neighbors=10, random_state=0)              kernel_params=None, n_clusters=2, n_components=None, n_init=10,              n_jobs=None, n_neighbors=10, random_state=0)      Notes     -----


Question: https://www1.icsi.berkeley.edu/~stellayu/publication/doc/2003kwayICCV.pdf     """     def __init__(self, n_clusters=8, eigen_solver=None, random_state=None,                 n_init=10, gamma=1., affinity='rbf', n_neighbors=10,                 eigen_tol=0.0, assign_labels='kmeans', degree=3, coef0=1,                 kernel_params=None, n_jobs=None):    def __init__(self, n_clusters=8, eigen_solver=None, n_components=None,                 random_state=None, n_init=10, gamma=1., affinity='rbf',                 n_neighbors=10, eigen_tol=0.0, assign_labels='kmeans',                 degree=3, coef0=1, kernel_params=None, n_jobs=None):         self.n_clusters = n_clusters         self.eigen_solver = eigen_solver        self.n_components = n_components         self.random_state = random_state         self.n_init = n_init         self.gamma = gamma


Question: random_state = check_random_state(self.random_state)         self.labels_ = spectral_clustering(self.affinity_matrix_,                                            n_clusters=self.n_clusters,<ins>                                           n_components=self.n_components,</ins>                                            eigen_solver=self.eigen_solver,                                            random_state=random_state,                                            n_init=self.n_init,


Question: if sample_weight is not None:             for name, step in self.estimators:<ins>                if step is None:                    continue</ins>                 if not has_fit_parameter(step, 'sample_weight'):                     raise ValueError('Underlying estimator \'%s\' does not'                                      ' support sample weights.' % name)

Voting estimator will fail at fit if weights are passed and an estimator is None Because we don't check for an estimator to be in support, is failing`.


Question: def _parallel_fit_estimator(estimator, X, y, sample_weight=None):     """Private function used to fit an estimator within a job."""     if sample_weight is not None:        estimator.fit(X, y, sample_weight=sample_weight)        try:            estimator.fit(X, y, sample_weight=sample_weight)        except TypeError as exc:            if "unexpected keyword argument 'sample_weight'" in str(exc):                raise ValueError(                    "Underlying estimator {} does not support sample weights."                    .format(estimator.__class__.__name__)                ) from exc            raise     else:         estimator.fit(X, y)     return estimator

Handle 'drop' together with None to drop estimator in VotingClassifier/VotingRegressor As mentioned in the following , the and should accept to drop an estimator from the ensemble is the same way that is doing now.


Question: """Get the weights of not `None` estimators"""         if self.weights is None:             return None        return [w for est, w in zip(self.estimators,                                    self.weights) if est[1] is not None]        return [w for est, w in zip(self.estimators, self.weights)                if est[1] not in (None, 'drop')]      def _predict(self, X):         """Collect results from clf.predict calls. """


Question: '; got %d weights, %d estimators'                              % (len(self.weights), len(self.estimators)))         if sample_weight is not None:            for name, step in self.estimators:                if step is None:                    continue                if not has_fit_parameter(step, 'sample_weight'):                    raise ValueError('Underlying estimator \'%s\' does not'                                     ' support sample weights.' % name)         names, clfs = zip(*self.estimators)         self._validate_names(names)         n_isnone = np.sum([clf is None for _, clf in self.estimators])        n_isnone = np.sum(            [clf in (None, 'drop') for _, clf in self.estimators]        )         if n_isnone == len(self.estimators):            raise ValueError('All estimators are None. At least one is '                             'required!')            raise ValueError(                'All estimators are None or "drop". At least one is required!'            )          self.estimators_ = Parallel(n_jobs=self.n_jobs)(                 delayed(_parallel_fit_estimator)(clone(clf), X, y,                                                  sample_weight=sample_weight)                for clf in clfs if clf is not None)                for clf in clfs if clf not in (None, 'drop')            )          self.named_estimators_ = Bunch()         for k, e in zip(self.estimators, self.estimators_):


Question: estimators : list of (string, estimator) tuples         Invoking the ``fit`` method on the ``VotingClassifier`` will fit clones         of those original estimators that will be stored in the class attribute        ``self.estimators_``. An estimator can be set to `None` using        ``set_params``.        ``self.estimators_``. An estimator can be set to ``None`` or ``'drop'``        using ``set_params``.      voting : str, {'hard', 'soft'} (default='hard')         If 'hard', uses predicted class labels for majority rule voting.


Question: Parameters     ----------     estimators : list of (string, estimator) tuples<del>        Invoking the ``fit`` method on the ``VotingRegressor`` will fit        clones of those original estimators that will be stored in the class        attribute ``self.estimators_``. An estimator can be set to `None`</del><ins>        Invoking the ``fit`` method on the ``VotingRegressor`` will fit clones        of those original estimators that will be stored in the class attribute        ``self.estimators_``. An estimator can be set to ``None`` or ``'drop'``</ins>         using ``set_params``.      weights : array-like, shape (n_regressors,), optional (default=`None`)


Question: y : Ignored          """        X = check_array(X, accept_sparse='csr')        if self.affinity == "precomputed":            accept_sparse = False        else:            accept_sparse = 'csr'        X = check_array(X, accept_sparse=accept_sparse)         if self.affinity == "precomputed":             self.affinity_matrix_ = X         elif self.affinity == "euclidean":

sklearn.cluster.AffinityPropagation doesn't support sparse affinity matrix <!-- If your issue is a usage question, submit it here instead: StackOverflow with the scikit-learn tag: Mailing List: For more information, see User Questions: http://scikit- --<!-- Instructions For Filing a Bug: --sklearn.cluster.AffinityPropagation doesn't support sparse affinity matrix. <!-- Example: Joblib Error thrown when calling fit on LatentDirichletAllocation with evaluateevery 0--A similar question is at . It focuses on default affinity. no error raised since it works for dense matrix. Traceback (most recent call last): File "<stdin", line 1, in <moduleFile "D:\Miniconda\lib\site-packages\sklearn\cluster\", line 381, in fit , verbose=self.verbose, returnniter=True) File "D:\Miniconda\lib\site-packages\sklearn\cluster\", line 115, in affinitypropagation preference = (S) File "D:\Miniconda\lib\site-packages\numpy\lib\", line 3336, in median overwriteinput=overwriteinput) File "D:\Miniconda\lib\site-packages\numpy\lib\", line 3250, in ureduce r = func(a, kwargs) File "D:\Miniconda\lib\site-packages\numpy\lib\", line 3395, in median return mean(part[indexer], axis=axis, out=out) File "D:\Miniconda\lib\site-packages\numpy\core\", line 2920, in mean out=out, kwargs) File "D:\Miniconda\lib\site-packages\numpy\core\", line 85, in mean ret = (ret / rcount) ValueError: setting an array element with a sequence.


Question: y : Ignored          """        X = check_array(X, accept_sparse='csr')        if self.affinity == "precomputed":            accept_sparse = False        else:            accept_sparse = 'csr'        X = check_array(X, accept_sparse=accept_sparse)         if self.affinity == "precomputed":             self.affinity_matrix_ = X         elif self.affinity == "euclidean":

System: python: 3.6.7 (default, Oct 28 2018, 19:44:12) [MSC v.1915 64 bit (AMD64)] executable: D:\Miniconda\ machine: Windows-7-6.1.7601-SP1 BLAS: macros: SCIPYMKLH=None, HAVECBLAS=None libdirs: D:/Miniconda\Library\lib cblaslibs: mkl_rt Python deps: pip: 18.1 setuptools: 40.6.2 sklearn: 0.20.1 numpy: 1.15.4 scipy: 1.1.0 Cython: None pandas: 0.23.4 <!-- Thanks for contributing! --


Question: maxmem = max(         ((x_density * n_samples_X + y_density * n_samples_Y) * n_features          + (x_density * n_samples_X * y_density * n_samples_Y)) / 10,        10 * 2**17)        10 * 2 ** 17)      # The increase amount of memory in 8-byte blocks is:     # - x_density * batch_size * n_features (copy of chunk of X)

AttributeError thrown when calling metrics.pairwisedistances with binary metrics and Y is None thrown when calling with binary metrics if is . No error. Should return a of shape containing the pairwise distances. This worked correctly in sklearn version 0.20.3. I think the problem was introduced in there is now a code path where has its checked without any prior check as to whether is .


Question: # Hence x² + (xd+yd)kx = M, where x=batch_size, k=n_features, M=maxmem     #                                 xd=x_density and yd=y_density     tmp = (x_density + y_density) * n_features    batch_size = (-tmp + np.sqrt(tmp**2 + 4 * maxmem)) / 2    batch_size = (-tmp + np.sqrt(tmp ** 2 + 4 * maxmem)) / 2     batch_size = max(int(batch_size), 1)      x_batches = gen_batches(X.shape[0], batch_size)


Question: K = safe_sparse_dot(X, Y.T, dense_output=True)     K *= gamma     K += coef0    np.tanh(K, K)   # compute tanh in-place    np.tanh(K, K)  # compute tanh in-place     return K


Question: K = euclidean_distances(X, Y, squared=True)     K *= -gamma    np.exp(K, K)    # exponentiate K in-place    np.exp(K, K)  # exponentiate K in-place     return K


Question: gamma = 1.0 / X.shape[1]      K = -gamma * manhattan_distances(X, Y)    np.exp(K, K)    # exponentiate K in-place    np.exp(K, K)  # exponentiate K in-place     return K


Question: dtype = bool if metric in PAIRWISE_BOOLEAN_FUNCTIONS else None         if dtype == bool and (X.dtype != bool or Y.dtype != bool):        if (dtype == bool and                (X.dtype != bool or (Y is not None and Y.dtype != bool))):             msg = "Data was converted to boolean for metric %s" % metric             warnings.warn(msg, DataConversionWarning)


Question: 'yule', ] <del></del> # Helper functions - distance PAIRWISE_KERNEL_FUNCTIONS = {     # If updating this dictionary, update the doc in both distance_metrics()


Question: if Y is None:         Y = X    X, Y, dtype = _return_float_dtype(X, Y)      if effective_n_jobs(n_jobs) == 1:         return func(X, Y, **kwds)      # enforce a threading backend to prevent data communication overhead     fd = delayed(_dist_wrapper)    ret = np.empty((X.shape[0], Y.shape[0]), dtype=X.dtype, order='F')    ret = np.empty((X.shape[0], Y.shape[0]), dtype=dtype, order='F')     Parallel(backend="threading", n_jobs=n_jobs)(         fd(func, ret, s, X, Y[s], **kwds)         for s in gen_even_slices(_num_samples(Y), effective_n_jobs(n_jobs)))     if (X is Y or Y is None) and func is euclidean_distances:        # zeroing diagonal for euclidean norm.        # TODO: do it also for other norms.        np.fill_diagonal(ret, 0)     return ret

pairwisedistances returns zeros for metric cosine when executed in parallel returns a list of zeros when calculating with equal to -1 or greater than 2. Using calculates the expected results. Using the metric returns non-zero results, but the values seem to be integers instead of floats. I executed with different values for and . The outputs were as follows:


Question: return distances if squared else np.sqrt(distances, out=distances)  def _euclidean_distances_upcast(X, XX=None, Y=None, YY=None):def _euclidean_distances_upcast(X, XX=None, Y=None, YY=None, batch_size=None):     """Euclidean distances between X and Y      Assumes X and Y have float32 dtype.

Untreated overflow (?) for float32 in euclideandistances new in sklearn 21.1 I am using euclidean distances in a project and after updating, the result is wrong for just one of several datasets. When comparing it to one can see that in version 21.1 it behaves substantially different to 20.3. The matrix is an ndarray with size (100,) with float32. The data are in this zip Can be found when using sklearn 20.3, both behave identical. When using version 21.1 has many 0 entries and some unreasonably high entries Sklearn 21 System: python: 3.6.7 (default, Oct 22 2018, 11:32:17) [GCC 8.2.0] executable: /home/lenz/PycharmProjects/pyrolmm/venvsklearn21/bin/python3 machine: Linux-4.15.0-50-generic-x8664-with-Ubuntu-18.04-bionic BLAS: macros: HAVECBLAS=None, NOATLASINFO=-1 libdirs: /usr/lib/x8664-linux-gnu cblaslibs: cblas Python deps: pip: 9.0.1 setuptools: 39.0.1 sklearn: 0.21.1 numpy: 1.16.3 scipy: 1.3.0 Cython: None pandas: None For sklearn 20.3 the versions are: System: python: 3.6.7 (default, Oct 22 2018, 11:32:17) [GCC 8.2.0] executable: /home/lenz/PycharmProjects/pyrolmm/venvsklearn20/bin/python3 machine: Linux-4.15.0-50-generic-x8664-with-Ubuntu-18.04-bionic BLAS: macros: HAVECBLAS=None, NOATLASINFO=-1 libdirs: /usr/lib/x8664-linux-gnu cblas_libs: cblas Python deps: pip: 9.0.1 setuptools: 39.0.1 sklearn: 0.20.3 numpy: 1.16.3 scipy: 1.3.0 Cython: None pandas: None <!-- Thanks for contributing! --


Question: distances = np.empty((n_samples_X, n_samples_Y), dtype=np.float32)     x_density = X.nnz / np.prod(X.shape) if issparse(X) else 1    y_density = Y.nnz / np.prod(Y.shape) if issparse(Y) else 1    # Allow 10% more memory than X, Y and the distance matrix take (at least    # 10MiB)    maxmem = max(        ((x_density * n_samples_X + y_density * n_samples_Y) * n_features         + (x_density * n_samples_X * y_density * n_samples_Y)) / 10,        10 * 2 ** 17)    # The increase amount of memory in 8-byte blocks is:    # - x_density * batch_size * n_features (copy of chunk of X)    # - y_density * batch_size * n_features (copy of chunk of Y)    # - batch_size * batch_size (chunk of distance matrix)    # Hence x² + (xd+yd)kx = M, where x=batch_size, k=n_features, M=maxmem    #                                 xd=x_density and yd=y_density    tmp = (x_density + y_density) * n_features    batch_size = (-tmp + np.sqrt(tmp ** 2 + 4 * maxmem)) / 2    batch_size = max(int(batch_size), 1)    x_batches = gen_batches(X.shape[0], batch_size)    y_batches = gen_batches(Y.shape[0], batch_size)    if batch_size is None:        x_density = X.nnz / np.prod(X.shape) if issparse(X) else 1        y_density = Y.nnz / np.prod(Y.shape) if issparse(Y) else 1        # Allow 10% more memory than X, Y and the distance matrix take (at        # least 10MiB)        maxmem = max(            ((x_density * n_samples_X + y_density * n_samples_Y) * n_features             + (x_density * n_samples_X * y_density * n_samples_Y)) / 10,            10 * 2 ** 17)        # The increase amount of memory in 8-byte blocks is:        # - x_density * batch_size * n_features (copy of chunk of X)        # - y_density * batch_size * n_features (copy of chunk of Y)        # - batch_size * batch_size (chunk of distance matrix)        # Hence x² + (xd+yd)kx = M, where x=batch_size, k=n_features, M=maxmem        #                                 xd=x_density and yd=y_density        tmp = (x_density + y_density) * n_features        batch_size = (-tmp + np.sqrt(tmp ** 2 + 4 * maxmem)) / 2        batch_size = max(int(batch_size), 1)    x_batches = gen_batches(n_samples_X, batch_size)      for i, x_slice in enumerate(x_batches):         X_chunk = X[x_slice].astype(np.float64)


Question: else:             XX_chunk = XX[x_slice] <ins>        y_batches = gen_batches(n_samples_Y, batch_size)</ins>         for j, y_slice in enumerate(y_batches):             if X is Y and j < i:                 # when X is Y the distance matrix is symmetric so we only need


Question: Returns         -------         X_binned : array-like, shape (n_samples, n_features)            The binned data.            The binned data (fortran-aligned).         """         X = check_array(X, dtype=[X_DTYPE])         check_is_fitted(self, ['bin_thresholds_', 'actual_n_bins_'])diff --git a/sklearn/ensemble/_hist_gradient_boosting/gradient_boosting.py b/sklearn/ensemble/_hist_gradient_boosting/gradient_boosting.py-- a/sklearn/ensemble/_hist_gradient_boosting/gradient_boosting.py++ b/sklearn/ensemble/_hist_gradient_boosting/gradient_boosting.py

GBDTs should bin train and validation data separately? In the new GBDTs we bin the data before calling (for early-stopping). That means that the validation set is also used to find the bin thresholds (it is of course not used to find the split points!). I feel like the "data leak" is very minimal, but it seems more correct to bin Xtrain and Xval separately. WDYT?


Question: # data.         self._in_fit = True <del>        # bin the data        if self.verbose:            print("Binning {:.3f} GB of data: ".format(X.nbytes / 1e9), end="",                  flush=True)        tic = time()        self.bin_mapper_ = _BinMapper(max_bins=self.max_bins, random_state=rng)        X_binned = self.bin_mapper_.fit_transform(X)        toc = time()        if self.verbose:            duration = toc - tic            print("{:.3f} s".format(duration))</del>          self.loss_ = self._get_loss()


Question: # stratify for classification             stratify = y if hasattr(self.loss_, 'predict_proba') else None             X_binned_train, X_binned_val, y_train, y_val = train_test_split(                X_binned, y, test_size=self.validation_fraction,                stratify=stratify, random_state=rng)            X_train, X_val, y_train, y_val = train_test_split(                X, y, test_size=self.validation_fraction, stratify=stratify,                random_state=rng)        else:            X_train, y_train = X, y            X_val, y_val = None, None             # Predicting is faster of C-contiguous arrays, training is faster            # on Fortran arrays.            X_binned_val = np.ascontiguousarray(X_binned_val)            X_binned_train = np.asfortranarray(X_binned_train)        # Bin the data        self.bin_mapper_ = _BinMapper(max_bins=self.max_bins, random_state=rng)        X_binned_train = self._bin_data(X_train, rng, is_training_data=True)        if X_val is not None:            X_binned_val = self._bin_data(X_val, rng, is_training_data=False)         else:            X_binned_train, y_train = X_binned, y            X_binned_val, y_val = None, None            X_binned_val = None          if self.verbose:             print("Fitting gradient boosted rounds:")


Question: for score in recent_scores]         return not any(recent_improvements) <ins>    def _bin_data(self, X, rng, is_training_data):        """Bin data X.        If is_training_data, then set the bin_mapper_ attribute.        Else, the binned data is converted to a C-contiguous array.        """        description = 'training' if is_training_data else 'validation'        if self.verbose:            print("Binning {:.3f} GB of {} data: ".format(                X.nbytes / 1e9, description), end="", flush=True)        tic = time()        if is_training_data:            X_binned = self.bin_mapper_.fit_transform(X)  # F-aligned array        else:            X_binned = self.bin_mapper_.transform(X)  # F-aligned array            # We convert the array to C-contiguous since predicting is faster            # with this layout (training is faster on F-arrays though)            X_binned = np.ascontiguousarray(X_binned)        toc = time()        if self.verbose:            duration = toc - tic            print("{:.3f} s".format(duration))        return X_binned</ins>     def _print_iteration_stats(self, iteration_start_time):         """Print info about the current fitting iteration."""         log_msg = ''


Question: # License: BSD 3 clause  import numpy as npfrom scipy import linalgfrom scipy import linalg, sparse  from .base import _BasePCA from ..utils import check_array, gen_batches

IncrementalPCA should accept sparse input <!-- If your issue is a usage question, submit it here instead: StackOverflow with the scikit-learn tag: Mailing List: For more information, see User Questions: http://scikit- --<!-- Instructions For Filing a Bug: --is by design suited to application to sparse data in a way that most PCA classes are not. However, it is not written to accept this by default. No error should be thrown. I'd be happy to submit this as a PR if it's desirable. <details</details<!-- Thanks for contributing! --


Question: but not scaled for each feature before applying the SVD.      Depending on the size of the input data, this algorithm can be much more    memory efficient than a PCA.    memory efficient than a PCA, and allows sparse input.      This algorithm has constant memory complexity, on the order    of ``batch_size``, enabling use of np.memmap files without loading the    entire file into memory.    of ``batch_size * n_features``, enabling use of np.memmap files without    loading the entire file into memory. For sparse matrices, the input    is converted to dense in batches (in order to be able to subtract the    mean) which avoids storing the entire dense matrix at any one time.      The computational overhead of each SVD is     ``O(batch_size * n_features ** 2)``, but only 2 * batch_size samples


Question: --------     >>> from sklearn.datasets import load_digits     >>> from sklearn.decomposition import IncrementalPCA    >>> from scipy import sparse     >>> X, _ = load_digits(return_X_y=True)     >>> transformer = IncrementalPCA(n_components=7, batch_size=200)     >>> # either partially fit on smaller batches of data     >>> transformer.partial_fit(X[:100, :])     IncrementalPCA(batch_size=200, n_components=7)     >>> # or let the fit function itself divide the data into batches    >>> X_transformed = transformer.fit_transform(X)    >>> X_sparse = sparse.csr_matrix(X)    >>> X_transformed = transformer.fit_transform(X_sparse)     >>> X_transformed.shape     (1797, 7)


Question: Parameters         ----------        X : array-like, shape (n_samples, n_features)        X : array-like or sparse matrix, shape (n_samples, n_features)             Training data, where n_samples is the number of samples and             n_features is the number of features.


Question: self.singular_values_ = None         self.noise_variance_ = None         X = check_array(X, copy=self.copy, dtype=[np.float64, np.float32])        X = check_array(X, accept_sparse=['csr', 'csc', 'lil'],                        copy=self.copy, dtype=[np.float64, np.float32])         n_samples, n_features = X.shape          if self.batch_size is None:


Question: for batch in gen_batches(n_samples, self.batch_size_,                                  min_batch_size=self.n_components or 0):            self.partial_fit(X[batch], check_input=False)            X_batch = X[batch]            if sparse.issparse(X_batch):                X_batch = X_batch.toarray()            self.partial_fit(X_batch, check_input=False)          return self


Question: Returns the instance itself.         """         if check_input:<ins>            if sparse.issparse(X):                raise TypeError(                    "IncrementalPCA.partial_fit does not support "                    "sparse input. Either convert data to dense "                    "or use IncrementalPCA.fit to do so in batches.")</ins>             X = check_array(X, copy=self.copy, dtype=[np.float64, np.float32])         n_samples, n_features = X.shape         if not hasattr(self, 'components_'):


Question: np.sqrt((self.n_samples_seen_ * n_samples) /                         n_total_samples) * (self.mean_ - col_batch_mean)             X = np.vstack((self.singular_values_.reshape((-1, 1)) *                          self.components_, X, mean_correction))                           self.components_, X, mean_correction))          U, S, V = linalg.svd(X, full_matrices=False)         U, V = svd_flip(U, V, u_based_decision=False)


Question: else:             self.noise_variance_ = 0.         return self<ins>    def transform(self, X):        """Apply dimensionality reduction to X.        X is projected on the first principal components previously extracted        from a training set, using minibatches of size batch_size if X is        sparse.        Parameters        ----------        X : array-like, shape (n_samples, n_features)            New data, where n_samples is the number of samples            and n_features is the number of features.        Returns        -------        X_new : array-like, shape (n_samples, n_components)        Examples        --------        >>> import numpy as np        >>> from sklearn.decomposition import IncrementalPCA        >>> X = np.array([[-1, -1], [-2, -1], [-3, -2],        ...               [1, 1], [2, 1], [3, 2]])        >>> ipca = IncrementalPCA(n_components=2, batch_size=3)        >>> ipca.fit(X)        IncrementalPCA(batch_size=3, n_components=2)        >>> ipca.transform(X) # doctest: +SKIP        """        if sparse.issparse(X):            n_samples = X.shape[0]            output = []            for batch in gen_batches(n_samples, self.batch_size_,                                     min_batch_size=self.n_components or 0):                output.append(super().transform(X[batch].toarray()))            return np.vstack(output)        else:            return super().transform(X)</ins>


Question: if self.add_indicator:             self.indicator_ = MissingIndicator(                missing_values=self.missing_values)                missing_values=self.missing_values, error_on_new=False)             self.indicator_.fit(X)         else:             self.indicator_ = Nonediff --git a/sklearn/impute/_iterative.py b/sklearn/impute/_iterative.py-- a/sklearn/impute/_iterative.py++ b/sklearn/impute/_iterative.py

Design of addindicator in SimpleImputer may fail when running cross validation <!-- If your issue is a usage question, submit it here instead: StackOverflow with the scikit-learn tag: Mailing List: For more information, see User Questions: http://scikit- --<!-- Instructions For Filing a Bug: --<!-- Example: Joblib Error thrown when calling fit on LatentDirichletAllocation with evaluateevery 0--The design of depends on missing values exist in the training data. This will break cross validation. <!-- Example: If the code is too long, feel free to put it in a public gist and link it in the issue: --<!-- Example: No error is thrown. Please paste or describe the expected results.--No error is thrown. <!-- Please paste or specifically describe the actual output or traceback. --The original design was adopted because, if the training data has no missing value, there will be a column with all zeros. This type of error will appear when we try to do grid search over the parameter. One way to work around this is to split the data in such a way that missing values are available (for each column that has a missing value) in both the training set and test set. <!-- Thanks for contributing! --


Question: if self.add_indicator:             self.indicator_ = MissingIndicator(                missing_values=self.missing_values)                missing_values=self.missing_values, error_on_new=False)             X_trans_indicator = self.indicator_.fit_transform(X)         else:             self.indicator_ = None


Question: from .predictor import TreePredictor from .utils import sum_parallel from .types import PREDICTOR_RECORD_DTYPE<ins>from .types import Y_DTYPEEPS = np.finfo(Y_DTYPE).eps  # to avoid zero division errors</ins>   class TreeNode:

Zero division error in HistGradientBoosting NaN This dataset is a bit weird in that it has 100 classes with 16 samples each. The default parameter don't work very well but we should fail more gacefully. cc


Question: https://arxiv.org/abs/1603.02754         """         node.value = -self.shrinkage * node.sum_gradients / (            node.sum_hessians + self.splitter.l2_regularization)            node.sum_hessians + self.splitter.l2_regularization + EPS)         self.finalized_leaves.append(node)      def _finalize_splittable_nodes(self):diff --git a/sklearn/utils/estimator_checks.py b/sklearn/utils/estimator_checks.py-- a/sklearn/utils/estimator_checks.py++ b/sklearn/utils/estimator_checks.py


Question: hasattr(estimator, "predict_proba")):          estimator.fit(X, y)        a = estimator.predict_proba(X_test)[:, 1]        b = estimator.decision_function(X_test)        # Since the link function from decision_function() to predict_proba()        # is sometimes not precise enough (typically expit), we round to the        # 10th decimal to avoid numerical issues.        a = estimator.predict_proba(X_test)[:, 1].round(decimals=10)        b = estimator.decision_function(X_test).round(decimals=10)         assert_array_equal(rankdata(a), rankdata(b))


Question: value_fmt = "{}{} value: {}\n"      if feature_names:        feature_names_ = [feature_names[i] for i in tree_.feature]        feature_names_ = [feature_names[i] if i != _tree.TREE_UNDEFINED                          else None for i in tree_.feature]     else:         feature_names_ = ["feature_{}".format(i) for i in tree_.feature]

IndexError: list index out of range in exporttext when the tree only has one feature <!-- If your issue is a usage question, submit it here instead: StackOverflow with the scikit-learn tag: Mailing List: For more information, see User Questions: http://scikit- --<!-- Instructions For Filing a Bug: --returns when there is single feature. <!-- Thanks for contributing! --


Question: # This should remained pinned to version 1.2 and not updated like other# externals."""Copyright (c) 2001-2002 Enthought, Inc.  2003-2019, SciPy Developers.All rights reserved.Redistribution and use in source and binary forms, with or withoutmodification, are permitted provided that the following conditionsare met:1. Redistributions of source code must retain the above copyright   notice, this list of conditions and the following disclaimer.2. Redistributions in binary form must reproduce the above   copyright notice, this list of conditions and the following   disclaimer in the documentation and/or other materials provided   with the distribution.3. Neither the name of the copyright holder nor the names of its   contributors may be used to endorse or promote products derived   from this software without specific prior written permission.THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOTLIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FORA PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHTOWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOTLIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANYTHEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USEOF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE."""import numpy as npimport scipy.linalg.decomp as decompdef pinvh(a, cond=None, rcond=None, lower=True, return_rank=False,          check_finite=True):    """    Compute the (Moore-Penrose) pseudo-inverse of a Hermitian matrix.    Copied in from scipy==1.2.2, in order to preserve the default choice of the    `cond` and `above_cutoff` values which determine which values of the matrix    inversion lie below threshold and are so set to zero. Changes in scipy 1.3    resulted in a smaller default threshold and thus slower convergence of    dependent algorithms in some cases (see Sklearn github issue #14055).    Calculate a generalized inverse of a Hermitian or real symmetric matrix    using its eigenvalue decomposition and including all eigenvalues with    'large' absolute value.    Parameters    ----------    a : (N, N) array_like        Real symmetric or complex hermetian matrix to be pseudo-inverted    cond, rcond : float or None        Cutoff for 'small' eigenvalues.        Singular values smaller than rcond * largest_eigenvalue are considered        zero.        If None or -1, suitable machine precision is used.    lower : bool, optional        Whether the pertinent array data is taken from the lower or upper        triangle of a. (Default: lower)    return_rank : bool, optional        if True, return the effective rank of the matrix    check_finite : bool, optional        Whether to check that the input matrix contains only finite numbers.        Disabling may give a performance gain, but may result in problems        (crashes, non-termination) if the inputs do contain infinities or NaNs.    Returns    -------    B : (N, N) ndarray        The pseudo-inverse of matrix `a`.    rank : int        The effective rank of the matrix.  Returned if return_rank == True    Raises    ------    LinAlgError        If eigenvalue does not converge    Examples    --------    >>> from scipy.linalg import pinvh    >>> a = np.random.randn(9, 6)    >>> a = np.dot(a, a.T)    >>> B = pinvh(a)    >>> np.allclose(a, np.dot(a, np.dot(B, a)))    True    >>> np.allclose(B, np.dot(B, np.dot(a, B)))    True    """    a = decomp._asarray_validated(a, check_finite=check_finite)    s, u = decomp.eigh(a, lower=lower, check_finite=False)    if rcond is not None:        cond = rcond    if cond in [None, -1]:        t = u.dtype.char.lower()        factor = {'f': 1E3, 'd': 1E6}        cond = factor[t] * np.finfo(t).eps    # For Hermitian matrices, singular values equal abs(eigenvalues)    above_cutoff = (abs(s) > cond * np.max(abs(s)))    psigma_diag = 1.0 / s[above_cutoff]    u = u[:, above_cutoff]    B = np.dot(u * psigma_diag, np.conjugate(u).T)    if return_rank:        return B, len(psigma_diag)    else:        return Bdiff --git a/sklearn/linear_model/bayes.py b/sklearn/linear_model/bayes.py-- a/sklearn/linear_model/bayes.py++ b/sklearn/linear_model/bayes.py

ARD Regressor accuracy degrades when upgrading Scipy 1.2.1 -1.3.0 Hi, bit of a tricky one, I'm hoping someone will have some time and/or suggestions for further investigation! There seems to be an often-occurring worsening of performance (i.e. accuracy, although run-time increases too!) from the ARD regressor when upgrading from Scipy 1.2.1 -1.3.0. On a very simple dataset (see code snippets below) where a near-perfect fit should be achievable, typical error seems to degrade from order 1E-5 to 1E-2. Notably, convergence iterations seem to increase also from ~a few (~5) to around 50-200 iterations. Here's the headline plot, plotting absolute co-efficient error when fit across 1000 datasets generated with different random seeds: ! Note how with Scipy==1.2.1, errors are largely constrained to <0.01, while with Scipy==1.3.0 they range up to 0.05 (and in a few rare cases the algorithm produces garbage results, see later). I guess this could be (probably is?) a Scipy rather than Sklearn issue, but probably the only way to confirm / isolate that would be to start here. It's also possible that this worsening of behaviour is a weirdness of my particular toy example, but the difference in behaviour seems large and unexpected enough to warrant further investigation, I'd hope! OK, so here's a short snippet on just a single seed if you're curious to try this yourself. I'm generating three vectors of normally distributed values, 250 samples. Then the target is just a perfect copy of one of those vectors (index=1). We measure the accuracy of the fit by simply checking how close that coefficient is to 1.0 (the other coefficients always shrink to 0., as you'd hope): Scipy 1.2.1: Scipy 1.3.0 It could be that there's some oddity of the random data from a single seed, so I set up some short scripts to first generate a static collection of 1000 of the datasets as seen above, then collate the results from both versions of scipy.


Question: # This should remained pinned to version 1.2 and not updated like other# externals."""Copyright (c) 2001-2002 Enthought, Inc.  2003-2019, SciPy Developers.All rights reserved.Redistribution and use in source and binary forms, with or withoutmodification, are permitted provided that the following conditionsare met:1. Redistributions of source code must retain the above copyright   notice, this list of conditions and the following disclaimer.2. Redistributions in binary form must reproduce the above   copyright notice, this list of conditions and the following   disclaimer in the documentation and/or other materials provided   with the distribution.3. Neither the name of the copyright holder nor the names of its   contributors may be used to endorse or promote products derived   from this software without specific prior written permission.THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOTLIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FORA PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHTOWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOTLIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANYTHEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USEOF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE."""import numpy as npimport scipy.linalg.decomp as decompdef pinvh(a, cond=None, rcond=None, lower=True, return_rank=False,          check_finite=True):    """    Compute the (Moore-Penrose) pseudo-inverse of a Hermitian matrix.    Copied in from scipy==1.2.2, in order to preserve the default choice of the    `cond` and `above_cutoff` values which determine which values of the matrix    inversion lie below threshold and are so set to zero. Changes in scipy 1.3    resulted in a smaller default threshold and thus slower convergence of    dependent algorithms in some cases (see Sklearn github issue #14055).    Calculate a generalized inverse of a Hermitian or real symmetric matrix    using its eigenvalue decomposition and including all eigenvalues with    'large' absolute value.    Parameters    ----------    a : (N, N) array_like        Real symmetric or complex hermetian matrix to be pseudo-inverted    cond, rcond : float or None        Cutoff for 'small' eigenvalues.        Singular values smaller than rcond * largest_eigenvalue are considered        zero.        If None or -1, suitable machine precision is used.    lower : bool, optional        Whether the pertinent array data is taken from the lower or upper        triangle of a. (Default: lower)    return_rank : bool, optional        if True, return the effective rank of the matrix    check_finite : bool, optional        Whether to check that the input matrix contains only finite numbers.        Disabling may give a performance gain, but may result in problems        (crashes, non-termination) if the inputs do contain infinities or NaNs.    Returns    -------    B : (N, N) ndarray        The pseudo-inverse of matrix `a`.    rank : int        The effective rank of the matrix.  Returned if return_rank == True    Raises    ------    LinAlgError        If eigenvalue does not converge    Examples    --------    >>> from scipy.linalg import pinvh    >>> a = np.random.randn(9, 6)    >>> a = np.dot(a, a.T)    >>> B = pinvh(a)    >>> np.allclose(a, np.dot(a, np.dot(B, a)))    True    >>> np.allclose(B, np.dot(B, np.dot(a, B)))    True    """    a = decomp._asarray_validated(a, check_finite=check_finite)    s, u = decomp.eigh(a, lower=lower, check_finite=False)    if rcond is not None:        cond = rcond    if cond in [None, -1]:        t = u.dtype.char.lower()        factor = {'f': 1E3, 'd': 1E6}        cond = factor[t] * np.finfo(t).eps    # For Hermitian matrices, singular values equal abs(eigenvalues)    above_cutoff = (abs(s) > cond * np.max(abs(s)))    psigma_diag = 1.0 / s[above_cutoff]    u = u[:, above_cutoff]    B = np.dot(u * psigma_diag, np.conjugate(u).T)    if return_rank:        return B, len(psigma_diag)    else:        return Bdiff --git a/sklearn/linear_model/bayes.py b/sklearn/linear_model/bayes.py-- a/sklearn/linear_model/bayes.py++ b/sklearn/linear_model/bayes.py

The snippets are as follows: Make data: Test sklearn: Plot results: A little investigating summary statistics of those datasets in notebook gives the following points of comparison:


Question: from math import log import numpy as np from scipy import linalgfrom scipy.linalg import pinvh  from .base import LinearModel, _rescale_data from ..base import RegressorMixin from ..utils.extmath import fast_logdet from ..utils import check_X_yfrom ..utils.fixes import pinvh   ###############################################################################diff --git a/sklearn/utils/fixes.py b/sklearn/utils/fixes.py-- a/sklearn/utils/fixes.py++ b/sklearn/utils/fixes.py


Question: from math import log import numpy as np from scipy import linalgfrom scipy.linalg import pinvh  from .base import LinearModel, _rescale_data from ..base import RegressorMixin from ..utils.extmath import fast_logdet from ..utils import check_X_yfrom ..utils.fixes import pinvh   ###############################################################################diff --git a/sklearn/utils/fixes.py b/sklearn/utils/fixes.py-- a/sklearn/utils/fixes.py++ b/sklearn/utils/fixes.py


Question: # once support for sp_version < (1, 3) is dropped     from ..externals._lobpcg import lobpcg  # noqa <ins>if sp_version >= (1, 3):    # Preserves earlier default choice of pinvh cutoff `cond` value.    # Can be removed once issue #14055 is fully addressed.    from ..externals._scipy_linalg import pinvhelse:    from scipy.linalg import pinvh # noqa</ins> if sp_version >= (0, 19):     def _argmax(arr_or_spmatrix, axis=None):         return arr_or_spmatrix.argmax(axis=axis)


Question: # once support for sp_version < (1, 3) is dropped     from ..externals._lobpcg import lobpcg  # noqa <ins>if sp_version >= (1, 3):    # Preserves earlier default choice of pinvh cutoff `cond` value.    # Can be removed once issue #14055 is fully addressed.    from ..externals._scipy_linalg import pinvhelse:    from scipy.linalg import pinvh # noqa</ins> if sp_version >= (0, 19):     def _argmax(arr_or_spmatrix, axis=None):         return arr_or_spmatrix.argmax(axis=axis)


Question: # Take the best scores across every fold and the average of                 # all coefficients corresponding to the best scores.                 best_indices = np.argmax(scores, axis=1)                if self.multi_class == 'ovr':                if multi_class == 'ovr':                     w = np.mean([coefs_paths[i, best_indices[i], :]                                  for i in range(len(folds))], axis=0)                 else:

IndexError thrown with LogisticRegressionCV and refit=False The following error is thrown when trying to estimate a regularization parameter via cross-validation, without refitting. No error is thrown.


Question: best_indices_C = best_indices % len(self.Cs_)                 self.C_.append(np.mean(self.Cs_[best_indices_C]))                 best_indices_l1 = best_indices // len(self.Cs_)                self.l1_ratio_.append(np.mean(l1_ratios_[best_indices_l1]))                if self.penalty == 'elasticnet':                    best_indices_l1 = best_indices // len(self.Cs_)                    self.l1_ratio_.append(np.mean(l1_ratios_[best_indices_l1]))                else:                    self.l1_ratio_.append(None)              if multi_class == 'multinomial':                 self.C_ = np.tile(self.C_, n_classes)


Question: import numpy as np import sys import time<ins>import numbers</ins> from scipy.optimize import minimize from ..utils.extmath import softmax from ..metrics import pairwise_distances

NCA fails in GridSearch due to too strict parameter checks NCA checks its parameters to have a specific type, which can easily fail in a GridSearch due to how param grid is made. Here is an example: The issue is that for : 1 is not a float, and for : np.int64 is not int Before proposing a fix for this specific situation, I'd like to have your general opinion about parameter checking. I like this idea of common parameter checking tool introduced with the NCA PR. What do you think about extending it across the code-base (or at least for new or recent estimators) ? Currently parameter checking is not always done or often partially done, and is quite redundant. For instance, here is the input validation of lda: most params aren't checked and for those who are there's a lot of duplicated code. A propose to be upgrade the new tool to be able to check open/closed intervals (currently only closed) and list membership. The api would be something like that: where validoptions would be a dict of . e.g for the param of , it can be either a float or a string in a list, which would give Sometimes a parameter can only be positive or within a given interval, e.g. of must be between 0 and 1, which would give positivity of e.g. would be .


Question: # Check the preferred dimensionality of the projected space         if self.n_components is not None:            check_scalar(self.n_components, 'n_components', int, 1)            check_scalar(                self.n_components, 'n_components', numbers.Integral, 1)              if self.n_components > X.shape[1]:                 raise ValueError('The preferred dimensionality of the '


Question: .format(X.shape[1],                                          self.components_.shape[1]))         check_scalar(self.max_iter, 'max_iter', int, 1)        check_scalar(self.tol, 'tol', float, 0.)        check_scalar(self.verbose, 'verbose', int, 0)        check_scalar(self.max_iter, 'max_iter', numbers.Integral, 1)        check_scalar(self.tol, 'tol', numbers.Real, 0.)        check_scalar(self.verbose, 'verbose', numbers.Integral, 0)          if self.callback is not None:             if not callable(self.callback):


Question: from ..tree import DecisionTreeClassifier, DecisionTreeRegressor from ..utils import check_array, check_random_state, check_X_y, safe_indexing<ins>from ..utils.extmath import softmax</ins> from ..utils.extmath import stable_cumsum from ..metrics import accuracy_score, r2_score from ..utils.validation import check_is_fitted

AdaBoost's "SAMME" algorithm uses 'predict' while fitting and 'predictproba' while predicting probas Subj. This seems to me to be a wrong approach, moreover this drives to such mistakes: <preAdaBoostClassifier(algorithm="SAMME", baseestimator=SVC()).fit(trainX, trainY).predictproba(testX) NotImplementedError Traceback (most recent call last) <ipython-input-108-1d666912dadain <module() ----1 AdaBoostClassifier(algorithm="SAMME", baseestimator=SVC()).fit(trainX, trainY).predictproba(testX) /Library/Python/2.7/site- in predictproba(self, X) 716 proba = sum(estimator.predictproba(X) w 717 for estimator, w in zip(self.estimators, --718 self.estimatorweights)) 719 720 proba /= () /Library/Python/2.7/site- in <genexpr((estimator, w)) 715 else: # self.algorithm == "SAMME" 716 proba = sum(estimator.predictproba(X) w --717 for estimator, w in zip(self.estimators, 718 self.estimatorweights)) 719 /Library/Python/2.7/site- in predictproba(self, X) 493 if not self.probability: 494 raise NotImplementedError( --495 "probability estimates must be enabled to use this method") 496 497 if not in ('csvc', 'nusvc'): NotImplementedError: probability estimates must be enabled to use this method </pre


Question: else:                 yield pred / norm <ins>    @staticmethod    def _compute_proba_from_decision(decision, n_classes):        """Compute probabilities from the decision function.        This is based eq. (4) of [1] where:            p(y=c|X) = exp((1 / K-1) f_c(X)) / sum_k(exp((1 / K-1) f_k(X)))                     = softmax((1 / K-1) * f(X))        References        ----------        .. [1] J. Zhu, H. Zou, S. Rosset, T. Hastie, "Multi-class AdaBoost",               2009.        """        if n_classes == 2:            decision = np.vstack([-decision, decision]).T / 2        else:            decision /= (n_classes - 1)        return softmax(decision, copy=False)</ins>     def predict_proba(self, X):         """Predict class probabilities for X.


Question: if n_classes == 1:             return np.ones((_num_samples(X), 1))         if self.algorithm == 'SAMME.R':            # The weights are all 1. for SAMME.R            proba = sum(_samme_proba(estimator, n_classes, X)                        for estimator in self.estimators_)        else:  # self.algorithm == "SAMME"            proba = sum(estimator.predict_proba(X) * w                        for estimator, w in zip(self.estimators_,                                                self.estimator_weights_))        proba /= self.estimator_weights_.sum()        proba = np.exp((1. / (n_classes - 1)) * proba)        normalizer = proba.sum(axis=1)[:, np.newaxis]        normalizer[normalizer == 0.0] = 1.0        proba /= normalizer        return proba        decision = self.decision_function(X)        return self._compute_proba_from_decision(decision, n_classes)      def staged_predict_proba(self, X):         """Predict class probabilities for X.


Question: X = self._validate_data(X)          n_classes = self.n_classes_        proba = None        norm = 0.        for weight, estimator in zip(self.estimator_weights_,                                     self.estimators_):            norm += weight            if self.algorithm == 'SAMME.R':                # The weights are all 1. for SAMME.R                current_proba = _samme_proba(estimator, n_classes, X)            else:  # elif self.algorithm == "SAMME":                current_proba = estimator.predict_proba(X) * weight            if proba is None:                proba = current_proba            else:                proba += current_proba            real_proba = np.exp((1. / (n_classes - 1)) * (proba / norm))            normalizer = real_proba.sum(axis=1)[:, np.newaxis]            normalizer[normalizer == 0.0] = 1.0            real_proba /= normalizer             yield real_proba        for decision in self.staged_decision_function(X):            yield self._compute_proba_from_decision(decision, n_classes)      def predict_log_proba(self, X):         """Predict class log-probabilities for X.


Question: raise ValueError('Expected array-like (array or non-string sequence), '                          'got %r' % y)     sparseseries = (y.__class__.__name__ == 'SparseSeries')    if sparseseries:        raise ValueError("y cannot be class 'SparseSeries'.")    sparse_pandas = (y.__class__.__name__ in ['SparseSeries', 'SparseArray'])    if sparse_pandas:        raise ValueError("y cannot be class 'SparseSeries' or 'SparseArray'")      if is_multilabel(y):         return 'multilabel-indicator'

[MRG] Fix 'SparseSeries deprecated: scipy-dev failing on travis' <!-- Thanks for contributing a pull request! Please ensure you have taken a look at the contribution guidelines: -- Issue: SparseSeries deprecated: scipy-dev failing on travis <!-- Example: See also . Please use keywords (e.g., Fixes) to create link to the issues or pull requests you resolved, so that they will automatically be closed when your pull request is merged. See --Use a Series with sparse values instead instead of . <!-- Please be aware that we are a loose team of volunteers so patience is necessary; assistance handling other issues is very welcome. We value all user contributions, no matter how minor they are. If we are slow to review, either the pull request needs some benchmarking, tinkering, convincing, etc. or more likely the reviewers are simply busy. In either case, we ask for your understanding during the review process. For more information, see our FAQ on this topic: http://scikit- Thanks for contributing! --


Question: "Cython",         "pandas",         "matplotlib",<ins>        "joblib",</ins>     ]      def get_version(module):

Add joblib in showversions joblib should be to the dependencies listed in showversions or to the issue template when sklearn version is 0.20.


Question: from ..utils import Bunch from ..utils import safe_indexing from ..utils import _get_column_indices<ins>from ..utils import _check_key_type</ins> from ..utils.metaestimators import _BaseComposition from ..utils.validation import check_array, check_is_fitted

Named col indexing fails with ColumnTransformer remainder on changing DataFrame column ordering I am using ColumnTransformer to prepare (impute etc.) heterogenous data. I use a DataFrame to have more control on the different (types of) columns by their name. I had some really cryptic problems when downstream transformers complained of data of the wrong type, while the ColumnTransformer should have divided them up properly. I found out that ColumnTransformer silently passes the wrong columns along as when specifying columns by name, using the option, and using DataFrames where column ordering can differ between and In this case, the wrong columns are passed on to the downstream transformers, as the example demonstrates: When using ColumnTransformer with a DataFrame and specifying columns by name, should reference the same columns when fitting and when transforming (['b'] in above example), regardless of the column positions in the data during fitting and transforming. appears to, during fitting, remember remaining named DataFrame columns by their numeric index (not by their names), which (silently) leads to the wrong columns being handled downstream if the transformed DataFrame's column ordering differs from that of the fitted DataFrame. Position in module where the index is determined: My current workaround is to not use the option but specify all columns by name explicitly. System: python: 3.7.3 (default, Mar 30 2019, 03:44:34) [Clang 9.1.0 (clang-902.0.39.2)] executable: machine: Darwin-17.7.0-x8664-i386-64bit BLAS: macros: NOATLASINFO=3, HAVECBLAS=None libdirs: cblaslibs: cblas Python deps: pip: 19.1.1 setuptools: 41.0.1 sklearn: 0.21.2 numpy: 1.16.4 scipy: 1.3.0 Cython: None pandas: 0.24.2


Question: By setting ``remainder`` to be an estimator, the remaining         non-specified columns will use the ``remainder`` estimator. The         estimator must support :term:`fit` and :term:`transform`.<ins>        Note that using this feature requires that the DataFrame columns        input at :term:`fit` and :term:`transform` have identical order.</ins>      sparse_threshold : float, default = 0.3         If the output of the different transformers contains sparse matrices,


Question: "'passthrough', or estimator. '%s' was passed instead" %                 self.remainder)         n_columns = X.shape[1]        # Make it possible to check for reordered named columns on transform        if (hasattr(X, 'columns') and                any(_check_key_type(cols, str) for cols in self._columns)):            self._df_columns = X.columns        self._n_features = X.shape[1]         cols = []         for columns in self._columns:             cols.extend(_get_column_indices(X, columns))        remaining_idx = sorted(list(set(range(n_columns)) - set(cols))) or None        remaining_idx = list(set(range(self._n_features)) - set(cols))        remaining_idx = sorted(remaining_idx) or None          self._remainder = ('remainder', self.remainder, remaining_idx)


Question: """         check_is_fitted(self, 'transformers_')         X = _check_X(X)        if self._n_features > X.shape[1]:            raise ValueError('Number of features of the input must be equal '                             'to or greater than that of the fitted '                             'transformer. Transformer n_features is {0} '                             'and input n_features is {1}.'                             .format(self._n_features, X.shape[1]))        # No column reordering allowed for named cols combined with remainder        if (self._remainder[2] is not None and                hasattr(self, '_df_columns') and                hasattr(X, 'columns')):            n_cols_fit = len(self._df_columns)            n_cols_transform = len(X.columns)            if (n_cols_transform >= n_cols_fit and                    any(X.columns[:n_cols_fit] != self._df_columns)):                raise ValueError('Column ordering must be equal for fit '                                 'and for transform when using the '                                 'remainder keyword')         Xs = self._fit_transform(X, None, _transform_one, fitted=True)         self._validate_output(Xs)


Question: raise ValueError(             "'estimator' must be a fitted regressor or classifier.")     if (hasattr(estimator, 'classes_') and            isinstance(estimator.classes_[0], np.ndarray)):        raise ValueError('Multiclass-multioutput estimators are not supported')    if is_classifier(estimator):        if not hasattr(estimator, 'classes_'):            raise ValueError(                "'estimator' parameter must be a fitted estimator"            )        if isinstance(estimator.classes_[0], np.ndarray):            raise ValueError(                'Multiclass-multioutput estimators are not supported'            )      X = check_array(X)

plotpartialdependence() fails when used on DecisionTreeRegressor <!-- If your issue is a usage question, submit it here instead: StackOverflow with the scikit-learn tag: Mailing List: For more information, see User Questions: http://scikit- --<!-- Instructions For Filing a Bug: --fails when using a as the estimator. The problem appears to be related to the presence of a attribute (with a value of ) on the estimator, despite it being a regressor and not a classifier. Deleting the attribute from the estimator allows to successfully run. <!-- Example: If the code is too long, feel free to put it in a public gist and link it in the issue: --No error is thrown. <!-- Example: No error is thrown. Please paste or describe the expected results.--<!-- Please paste or specifically describe the actual output or traceback. --A is thrown: <!-- Please run the following snippet and paste the output below. For scikit-learn = 0.20: import sklearn; sklearn.show_versions() For scikit-learn < 0.20: import platform; print(platform.platform()) import sys; print("Python", sys.version) import numpy; print("NumPy", numpy.version) import scipy; print("SciPy", scipy.version) import sklearn; print("Scikit-Learn", sklearn.version) --<!-- Thanks for contributing! --