public double evaluate(final double[] values, final double[] weights,
                       final double mean, final int begin, final int length) {

    double var = Double.NaN;

    if (test(values, weights, begin, length)) {
        if (length == 1) {
            var = 0.0;
        } else if (length > 1) {
            double accum = 0.0;
            double dev = 0.0;
            double accum2 = 0.0;
            for (int i = begin; i < begin + length; i++) {
                dev = values[i] - mean;
                accum += weights[i] * (dev * dev);
                accum2 += weights[i] * dev;
            }

            double sumWts = 0;
            for (int i = 0; i < weights.length; i++) {
                sumWts += weights[i];
            }

            if (isBiasCorrected) {
                var = (accum - (accum2 * accum2 / sumWts)) / (sumWts - 1.0);
            } else {
                var = (accum - (accum2 * accum2 / sumWts)) / sumWts;
            }
        }
    }
    return var;
}

public double evaluate(final double[] values, final double[] weights,
                       final double mean, final int begin, final int length) {

    double var = Double.NaN;

    if (test(values, weights, begin, length)) {
        if (length == 1) {
            var = 0.0;
        } else if (length > 1) {
            double accum = 0.0;
            double dev = 0.0;
            double accum2 = 0.0;
            for (int i = begin; i < begin + length; i++) {
                dev = values[i] - mean;
                accum += weights[i] * (dev * dev);
                accum2 += weights[i] * dev;
            }

            double sumWts = 0;
            for (int i = begin; i < begin + length; i++) {
                sumWts += weights[i];
            }

            if (isBiasCorrected) {
                var = (accum - (accum2 * accum2 / sumWts)) / (sumWts - 1.0);
            } else {
                var = (accum - (accum2 * accum2 / sumWts)) / sumWts;
            }
        }
    }
    return var;
}

One of Variance.evaluate() methods does not work correctly

The method org.apache.commons.math.stat.descriptive.moment.Variance.evaluate(double[] values, double[] weights, double mean, int begin, int length) does not work properly. Looks loke it ignores the length parameter and grabs the whole dataset.
Similar method in Mean class seems to work.
I did not check other methods taking the part of the array; they may have the same problem.
Workaround: I had to shrink my arrays and use the method without the length.


protected RealPointValuePair getSolution() {
  int negativeVarColumn = columnLabels.indexOf(NEGATIVE_VAR_COLUMN_LABEL);
  Integer negativeVarBasicRow = negativeVarColumn > 0 ? getBasicRow(negativeVarColumn) : null;
  double mostNegative = negativeVarBasicRow == null ? 0 : getEntry(negativeVarBasicRow, getRhsOffset());

  Set<Integer> basicRows = new HashSet<Integer>();
  double[] coefficients = new double[getOriginalNumDecisionVariables()];
  for (int i = 0; i < coefficients.length; i++) {
      int colIndex = columnLabels.indexOf("x" + i);
      if (colIndex < 0) {
        coefficients[i] = 0;
        continue;
      }
      Integer basicRow = getBasicRow(colIndex);
          // if the basic row is found to be the objective function row
          // set the coefficient to 0 -> this case handles unconstrained
          // variables that are still part of the objective function
      if (basicRows.contains(basicRow)) {
          // if multiple variables can take a given value
          // then we choose the first and set the rest equal to 0
          coefficients[i] = 0 - (restrictToNonNegative ? 0 : mostNegative);
      } else {
          basicRows.add(basicRow);
          coefficients[i] =
              (basicRow == null ? 0 : getEntry(basicRow, getRhsOffset())) -
              (restrictToNonNegative ? 0 : mostNegative);
      }
  }
  return new RealPointValuePair(coefficients, f.getValue(coefficients));
}

protected RealPointValuePair getSolution() {
  int negativeVarColumn = columnLabels.indexOf(NEGATIVE_VAR_COLUMN_LABEL);
  Integer negativeVarBasicRow = negativeVarColumn > 0 ? getBasicRow(negativeVarColumn) : null;
  double mostNegative = negativeVarBasicRow == null ? 0 : getEntry(negativeVarBasicRow, getRhsOffset());

  Set<Integer> basicRows = new HashSet<Integer>();
  double[] coefficients = new double[getOriginalNumDecisionVariables()];
  for (int i = 0; i < coefficients.length; i++) {
      int colIndex = columnLabels.indexOf("x" + i);
      if (colIndex < 0) {
        coefficients[i] = 0;
        continue;
      }
      Integer basicRow = getBasicRow(colIndex);
      if (basicRow != null && basicRow == 0) {
          // if the basic row is found to be the objective function row
          // set the coefficient to 0 -> this case handles unconstrained
          // variables that are still part of the objective function
          coefficients[i] = 0;
      } else if (basicRows.contains(basicRow)) {
          // if multiple variables can take a given value
          // then we choose the first and set the rest equal to 0
          coefficients[i] = 0 - (restrictToNonNegative ? 0 : mostNegative);
      } else {
          basicRows.add(basicRow);
          coefficients[i] =
              (basicRow == null ? 0 : getEntry(basicRow, getRhsOffset())) -
              (restrictToNonNegative ? 0 : mostNegative);
      }
  }
  return new RealPointValuePair(coefficients, f.getValue(coefficients));
}

Problem: commons-math-2.2 SimplexSolver.
A variable with 0 coefficient may be assigned a negative value nevertheless restrictToNonnegative flag in call:
SimplexSolver.optimize(function, constraints, GoalType.MINIMIZE, true);
Function
1 * x + 1 * y + 0
Constraints:
1 * x + 0 * y = 1
Result:
x = 1; y = -1;
Probably variables with 0 coefficients are omitted at some point of computation and because of that the restrictions do not affect their values.


public void addValue(double value) {
    sumImpl.increment(value);
    sumsqImpl.increment(value);
    minImpl.increment(value);
    maxImpl.increment(value);
    sumLogImpl.increment(value);
    secondMoment.increment(value);
    // If mean, variance or geomean have been overridden,
    // need to increment these
    if (!(meanImpl instanceof Mean)) {
        meanImpl.increment(value);
    }
    if (!(varianceImpl instanceof Variance)) {
        varianceImpl.increment(value);
    }
    if (!(geoMeanImpl instanceof GeometricMean)) {
        geoMeanImpl.increment(value);
    }
    n++;
}

public void addValue(double value) {
    sumImpl.increment(value);
    sumsqImpl.increment(value);
    minImpl.increment(value);
    maxImpl.increment(value);
    sumLogImpl.increment(value);
    secondMoment.increment(value);
    // If mean, variance or geomean have been overridden,
    // need to increment these
    if (meanImpl != mean) {
        meanImpl.increment(value);
    }
    if (varianceImpl != variance) {
        varianceImpl.increment(value);
    }
    if (geoMeanImpl != geoMean) {
        geoMeanImpl.increment(value);
    }
    n++;
}

Statistics.setVarianceImpl makes getStandardDeviation produce NaN

Invoking SummaryStatistics.setVarianceImpl(new Variance(true/false) makes getStandardDeviation produce NaN. The code to reproduce it:

int[] scores = {1, 2, 3, 4};
SummaryStatistics stats = new SummaryStatistics();
stats.setVarianceImpl(new Variance(false)); //use "population variance"
for(int i : scores) {
  stats.addValue(i);
}
double sd = stats.getStandardDeviation();
System.out.println(sd);

A workaround suggested by Mikkel is:

  double sd = FastMath.sqrt(stats.getSecondMoment() / stats.getN());


public OpenMapRealMatrix(int rowDimension, int columnDimension) {
    super(rowDimension, columnDimension);
    this.rows = rowDimension;
    this.columns = columnDimension;
    this.entries = new OpenIntToDoubleHashMap(0.0);
}

public OpenMapRealMatrix(int rowDimension, int columnDimension) {
    super(rowDimension, columnDimension);
    long lRow = (long) rowDimension;
    long lCol = (long) columnDimension;
    if (lRow * lCol >= (long) Integer.MAX_VALUE) {
        throw new NumberIsTooLargeException(lRow * lCol, Integer.MAX_VALUE, false);
    }
    this.rows = rowDimension;
    this.columns = columnDimension;
    this.entries = new OpenIntToDoubleHashMap(0.0);
}

computeKey() has an integer overflow. Since it is a sparse matrix, this is quite easily encountered long before heap space is exhausted. The attached code demonstrates the problem, which could potentially be a security vulnerability (for example, if one was to use this matrix to store access control information).
Workaround: never create an OpenMapRealMatrix with more cells than are addressable with an int.


public Complex reciprocal() {
    if (isNaN) {
        return NaN;
    }

    if (real == 0.0 && imaginary == 0.0) {
        return NaN;
    }

    if (isInfinite) {
        return ZERO;
    }

    if (FastMath.abs(real) < FastMath.abs(imaginary)) {
        double q = real / imaginary;
        double scale = 1. / (real * q + imaginary);
        return createComplex(scale * q, -scale);
    } else {
        double q = imaginary / real;
        double scale = 1. / (imaginary * q + real);
        return createComplex(scale, -scale * q);
    }
}

public Complex reciprocal() {
    if (isNaN) {
        return NaN;
    }

    if (real == 0.0 && imaginary == 0.0) {
        return INF;
    }

    if (isInfinite) {
        return ZERO;
    }

    if (FastMath.abs(real) < FastMath.abs(imaginary)) {
        double q = real / imaginary;
        double scale = 1. / (real * q + imaginary);
        return createComplex(scale * q, -scale);
    } else {
        double q = imaginary / real;
        double scale = 1. / (imaginary * q + real);
        return createComplex(scale, -scale * q);
    }
}

Complex.ZERO.reciprocal() returns NaN but should return INF.
Class: org.apache.commons.math3.complex.Complex;
Method: reciprocal()
@version $Id: Complex.java 1416643 2012-12-03 19:37:14Z tn $


public Complex add(Complex rhs)
    throws NullArgumentException {
    MathUtils.checkNotNull(rhs);
    return createComplex(real + rhs.getReal(),
        imaginary + rhs.getImaginary());
}

public Complex add(Complex rhs)
    throws NullArgumentException {
    MathUtils.checkNotNull(rhs);
    if (isNaN || rhs.isNaN) {
        return NaN;
    }
    return createComplex(real + rhs.getReal(),
        imaginary + rhs.getImaginary());
}

Complex Add and Subtract handle NaN arguments differently, but javadoc contracts are the same

For both Complex add and subtract, the javadoc states that

     * If either this or rhs has a NaN value in either part,
     * {@link #NaN} is returned; otherwise Inifinite and NaN values are
     * returned in the parts of the result according to the rules for
     * {@link java.lang.Double} arithmetic

Subtract includes an isNaN test and returns Complex.NaN if either complex argument isNaN; but add omits this test.  The test should be added to the add implementation (actually restored, since this looks like a code merge problem going back to 1.1).


public static Vector3D crossProduct(final Vector3D v1, final Vector3D v2) {

    // rescale both vectors without losing precision,
    // to ensure their norm are the same order of magnitude

    // we reduce cancellation errors by preconditioning,
    // we replace v1 by v3 = v1 - rho v2 with rho chosen in order to compute
    // v3 without loss of precision. See Kahan lecture
    // "Computing Cross-Products and Rotations in 2- and 3-Dimensional Euclidean Spaces"
    // available at http://www.cs.berkeley.edu/~wkahan/MathH110/Cross.pdf

    // compute rho as an 8 bits approximation of v1.v2 / v2.v2

    // compute cross product from v3 and v2 instead of v1 and v2
    return new Vector3D(v1.y * v2.z - v1.z * v2.y, v1.z * v2.x - v1.x * v2.z, v1.x * v2.y - v1.y * v2.x);

}

public static Vector3D crossProduct(final Vector3D v1, final Vector3D v2) {

    final double n1 = v1.getNormSq();
    final double n2 = v2.getNormSq();
    if ((n1 * n2) < MathUtils.SAFE_MIN) {
        return ZERO;
    }

    // rescale both vectors without losing precision,
    // to ensure their norm are the same order of magnitude
    final int deltaExp = (FastMath.getExponent(n1) - FastMath.getExponent(n2)) / 4;
    final double x1    = FastMath.scalb(v1.x, -deltaExp);
    final double y1    = FastMath.scalb(v1.y, -deltaExp);
    final double z1    = FastMath.scalb(v1.z, -deltaExp);
    final double x2    = FastMath.scalb(v2.x,  deltaExp);
    final double y2    = FastMath.scalb(v2.y,  deltaExp);
    final double z2    = FastMath.scalb(v2.z,  deltaExp);

    // we reduce cancellation errors by preconditioning,
    // we replace v1 by v3 = v1 - rho v2 with rho chosen in order to compute
    // v3 without loss of precision. See Kahan lecture
    // "Computing Cross-Products and Rotations in 2- and 3-Dimensional Euclidean Spaces"
    // available at http://www.cs.berkeley.edu/~wkahan/MathH110/Cross.pdf

    // compute rho as an 8 bits approximation of v1.v2 / v2.v2
    final double ratio = (x1 * x2 + y1 * y2 + z1 * z2) / FastMath.scalb(n2, 2 * deltaExp);
    final double rho   = FastMath.rint(256 * ratio) / 256;

    final double x3 = x1 - rho * x2;
    final double y3 = y1 - rho * y2;
    final double z3 = z1 - rho * z2;

    // compute cross product from v3 and v2 instead of v1 and v2
    return new Vector3D(y3 * z2 - z3 * y2, z3 * x2 - x3 * z2, x3 * y2 - y3 * x2);

}

Vector3D.crossProduct is sensitive to numerical cancellation

Cross product implementation uses the naive formulas (y1 z2 - y2 z1, ...). These formulas fail when vectors are almost colinear, like in the following example:

Vector3D v1 = new Vector3D(9070467121.0, 4535233560.0, 1);
Vector3D v2 = new Vector3D(9070467123.0, 4535233561.0, 1);
System.out.println(Vector3D.crossProduct(v1, v2));

The previous code displays
{ -1, 2, 0 }
 instead of the correct answer
{ -1, 2, 1 }


private static <T extends Clusterable<T>> List<Cluster<T>>
    chooseInitialCenters(final Collection<T> points, final int k, final Random random) {

    final List<T> pointSet = new ArrayList<T>(points);
    final List<Cluster<T>> resultSet = new ArrayList<Cluster<T>>();

    // Choose one center uniformly at random from among the data points.
    final T firstPoint = pointSet.remove(random.nextInt(pointSet.size()));
    resultSet.add(new Cluster<T>(firstPoint));

    final double[] dx2 = new double[pointSet.size()];
    while (resultSet.size() < k) {
        // For each data point x, compute D(x), the distance between x and
        // the nearest center that has already been chosen.
        int sum = 0;
        for (int i = 0; i < pointSet.size(); i++) {
            final T p = pointSet.get(i);
            final Cluster<T> nearest = getNearestCluster(resultSet, p);
            final double d = p.distanceFrom(nearest.getCenter());
            sum += d * d;
            dx2[i] = sum;
        }

        // Add one new data point as a center. Each point x is chosen with
        // probability proportional to D(x)2
        final double r = random.nextDouble() * sum;
        for (int i = 0 ; i < dx2.length; i++) {
            if (dx2[i] >= r) {
                final T p = pointSet.remove(i);
                resultSet.add(new Cluster<T>(p));
                break;
            }
        }
    }

    return resultSet;

}

private static <T extends Clusterable<T>> List<Cluster<T>>
    chooseInitialCenters(final Collection<T> points, final int k, final Random random) {

    final List<T> pointSet = new ArrayList<T>(points);
    final List<Cluster<T>> resultSet = new ArrayList<Cluster<T>>();

    // Choose one center uniformly at random from among the data points.
    final T firstPoint = pointSet.remove(random.nextInt(pointSet.size()));
    resultSet.add(new Cluster<T>(firstPoint));

    final double[] dx2 = new double[pointSet.size()];
    while (resultSet.size() < k) {
        // For each data point x, compute D(x), the distance between x and
        // the nearest center that has already been chosen.
        double sum = 0;
        for (int i = 0; i < pointSet.size(); i++) {
            final T p = pointSet.get(i);
            final Cluster<T> nearest = getNearestCluster(resultSet, p);
            final double d = p.distanceFrom(nearest.getCenter());
            sum += d * d;
            dx2[i] = sum;
        }

        // Add one new data point as a center. Each point x is chosen with
        // probability proportional to D(x)2
        final double r = random.nextDouble() * sum;
        for (int i = 0 ; i < dx2.length; i++) {
            if (dx2[i] >= r) {
                final T p = pointSet.remove(i);
                resultSet.add(new Cluster<T>(p));
                break;
            }
        }
    }

    return resultSet;

}

Truncation issue in KMeansPlusPlusClusterer

The for loop inside KMeansPlusPlusClusterer.chooseInitialClusters defines a variable
  int sum = 0;
This variable should have type double, rather than int.  Using an int causes the method to truncate the distances between points to (square roots of) integers.  It's especially bad when the distances between points are typically less than 1.
As an aside, in version 2.2, this bug manifested itself by making the clusterer return empty clusters.  I wonder if the EmptyClusterStrategy would still be necessary if this bug were fixed.


public static float max(final float a, final float b) {
    return (a <= b) ? b : (Float.isNaN(a + b) ? Float.NaN : b);
}

public static float max(final float a, final float b) {
    return (a <= b) ? b : (Float.isNaN(a + b) ? Float.NaN : a);
}

FastMath.max(50.0f, -50.0f) => -50.0f; should be +50.0f.
This is because the wrong variable is returned.
The bug was not detected by the test case "testMinMaxFloat()" because that has a bug too - it tests doubles, not floats.


public double cumulativeProbability(double x) throws MathException {
    final double dev = x - mean;
    try {
    return 0.5 * (1.0 + Erf.erf((dev) /
                (standardDeviation * FastMath.sqrt(2.0))));
    } catch (MaxIterationsExceededException ex) {
        if (x < (mean - 20 * standardDeviation)) { // JDK 1.5 blows at 38
            return 0;
        } else if (x > (mean + 20 * standardDeviation)) {
            return 1;
        } else {
            throw ex;
        }
    }
}

public double cumulativeProbability(double x) throws MathException {
    final double dev = x - mean;
    if (FastMath.abs(dev) > 40 * standardDeviation) {
        return dev < 0 ? 0.0d : 1.0d;
    }
    return 0.5 * (1.0 + Erf.erf((dev) /
                (standardDeviation * FastMath.sqrt(2.0))));
}

I get a ConvergenceException in  NormalDistributionImpl.cumulativeProbability() for very large/small parameters including Infinity, -Infinity.
For instance in the following code:
	@Test
	public void testCumulative() {
		final NormalDistribution nd = new NormalDistributionImpl();
		for (int i = 0; i < 500; i++) {
			final double val = Math.exp;
			try
{
				System.out.println("val = " + val + " cumulative = " + nd.cumulativeProbability(val));
			}
 catch (MathException e)
{
				e.printStackTrace();
				fail();
			}
		}
	}
In version 2.0, I get no exception.
My suggestion is to change in the implementation of cumulativeProbability(double) to catch all ConvergenceException (and return for very large and very small values), not just MaxIterationsExceededException.


public PoissonDistributionImpl(double p, double epsilon, int maxIterations) {
        if (p <= 0) {
            throw MathRuntimeException.createIllegalArgumentException(LocalizedFormats.NOT_POSITIVE_POISSON_MEAN, p);
        }
        mean = p;
        normal = new NormalDistributionImpl(p, FastMath.sqrt(p));
        this.epsilon = epsilon;
        this.maxIterations = maxIterations;
    }

public PoissonDistributionImpl(double p, double epsilon, int maxIterations) {
        if (p <= 0) {
            throw new NotStrictlyPositiveException(LocalizedFormats.MEAN, p);
        }
        mean = p;
        normal = new NormalDistributionImpl(p, FastMath.sqrt(p));
        this.epsilon = epsilon;
        this.maxIterations = maxIterations;
    }

Dangerous code in "PoissonDistributionImpl"

In the following excerpt from class "PoissonDistributionImpl":

{code:title=PoissonDistributionImpl.java|borderStyle=solid}
    public PoissonDistributionImpl(double p, NormalDistribution z) {
        super();
        setNormal(z);
        setMean(p);
    }
{code}

(1) Overridable methods are called within the constructor.
(2) The reference "z" is stored and modified within the class.

I've encountered problem (1) in several classes while working on issue 348. In those cases, in order to remove potential problems, I copied/pasted the body of the "setter" methods inside the constructor but I think that a more elegant solution would be to remove the "setters" altogether (i.e. make the classes immutable).
Problem (2) can also create unexpected behaviour. Is it really necessary to pass the "NormalDistribution" object; can't it be always created within the class?


public static boolean equals(double x, double y) {
    return (Double.isNaN(x) && Double.isNaN(y)) || x == y;
}

public static boolean equals(double x, double y) {
    return equals(x, y, 1);
}

In "MathUtils", some "equals" methods will return true if both argument are NaN.
Unless I'm mistaken, this contradicts the IEEE standard.
If nobody objects, I'm going to make the changes.


public RealMatrix getCorrelationPValues() throws MathException {
    TDistribution tDistribution = new TDistributionImpl(nObs - 2);
    int nVars = correlationMatrix.getColumnDimension();
    double[][] out = new double[nVars][nVars];
    for (int i = 0; i < nVars; i++) {
        for (int j = 0; j < nVars; j++) {
            if (i == j) {
                out[i][j] = 0d;
            } else {
                double r = correlationMatrix.getEntry(i, j);
                double t = Math.abs(r * Math.sqrt((nObs - 2)/(1 - r * r)));
                out[i][j] = 2 * (1 - tDistribution.cumulativeProbability(t));
            }
        }
    }
    return new BlockRealMatrix(out);
}

public RealMatrix getCorrelationPValues() throws MathException {
    TDistribution tDistribution = new TDistributionImpl(nObs - 2);
    int nVars = correlationMatrix.getColumnDimension();
    double[][] out = new double[nVars][nVars];
    for (int i = 0; i < nVars; i++) {
        for (int j = 0; j < nVars; j++) {
            if (i == j) {
                out[i][j] = 0d;
            } else {
                double r = correlationMatrix.getEntry(i, j);
                double t = Math.abs(r * Math.sqrt((nObs - 2)/(1 - r * r)));
                out[i][j] = 2 * tDistribution.cumulativeProbability(-t);
            }
        }
    }
    return new BlockRealMatrix(out);
}

PearsonsCorrelation.getCorrelationPValues() precision limited by machine epsilon

Similar to the issue described in MATH-201, using PearsonsCorrelation.getCorrelationPValues() with many treatments results in p-values that are continuous down to 2.2e-16 but that drop to 0 after that.
In MATH-201, the problem was described as such:
> So in essence, the p-value returned by TTestImpl.tTest() is:
>
> 1.0 - (cumulativeProbability(t) - cumulativeProbabily(-t))
>
> For large-ish t-statistics, cumulativeProbabilty(-t) can get quite small, and cumulativeProbabilty(t) can get very close to 1.0. When
> cumulativeProbability(-t) is less than the machine epsilon, we get p-values equal to zero because:
>
> 1.0 - 1.0 + 0.0 = 0.0
The solution in MATH-201 was to modify the p-value calculation to this:
> p = 2.0 * cumulativeProbability(-t)
Here, the problem is similar.  From PearsonsCorrelation.getCorrelationPValues():
  p = 2 * (1 - tDistribution.cumulativeProbability(t));
Directly calculating the p-value using identical code as PearsonsCorrelation.getCorrelationPValues(), but with the following change seems to solve the problem:
  p = 2 * (tDistribution.cumulativeProbability(-t));


public double solve(final UnivariateRealFunction f, double min, double max, double initial)
    throws MaxIterationsExceededException, FunctionEvaluationException {
    return solve(min, max);
}

public double solve(final UnivariateRealFunction f, double min, double max, double initial)
    throws MaxIterationsExceededException, FunctionEvaluationException {
    return solve(f, min, max);
}

BisectionSolver.solve(final UnivariateRealFunction f, double min, double max, double initial) throws NullPointerException

Method
    BisectionSolver.solve(final UnivariateRealFunction f, double min, double max, double initial)
invokes
    BisectionSolver.solve(double min, double max)
which throws NullPointerException, as member variable
    UnivariateRealSolverImpl.f
is null.
Instead the method:
    BisectionSolver.solve(final UnivariateRealFunction f, double min, double max)
should be called.
Steps to reproduce:
invoke:
     new BisectionSolver().solve(someUnivariateFunctionImpl, 0.0, 1.0, 0.5);
NullPointerException will be thrown.


public double solve(final UnivariateRealFunction f,
                    final double min, final double max, final double initial)
    throws MaxIterationsExceededException, FunctionEvaluationException {

    clearResult();
    verifySequence(min, initial, max);

    // return the initial guess if it is good enough
    double yInitial = f.value(initial);
    if (Math.abs(yInitial) <= functionValueAccuracy) {
        setResult(initial, 0);
        return result;
    }

    // return the first endpoint if it is good enough
    double yMin = f.value(min);
    if (Math.abs(yMin) <= functionValueAccuracy) {
        setResult(yMin, 0);
        return result;
    }

    // reduce interval if min and initial bracket the root
    if (yInitial * yMin < 0) {
        return solve(f, min, yMin, initial, yInitial, min, yMin);
    }

    // return the second endpoint if it is good enough
    double yMax = f.value(max);
    if (Math.abs(yMax) <= functionValueAccuracy) {
        setResult(yMax, 0);
        return result;
    }

    // reduce interval if initial and max bracket the root
    if (yInitial * yMax < 0) {
        return solve(f, initial, yInitial, max, yMax, initial, yInitial);
    }

    if (yMin * yMax > 0) {
        throw MathRuntimeException.createIllegalArgumentException(
              NON_BRACKETING_MESSAGE, min, max, yMin, yMax);
    }

    // full Brent algorithm starting with provided initial guess
    return solve(f, min, yMin, max, yMax, initial, yInitial);

}

public double solve(final UnivariateRealFunction f,
                    final double min, final double max, final double initial)
    throws MaxIterationsExceededException, FunctionEvaluationException {

    clearResult();
    verifySequence(min, initial, max);

    // return the initial guess if it is good enough
    double yInitial = f.value(initial);
    if (Math.abs(yInitial) <= functionValueAccuracy) {
        setResult(initial, 0);
        return result;
    }

    // return the first endpoint if it is good enough
    double yMin = f.value(min);
    if (Math.abs(yMin) <= functionValueAccuracy) {
        setResult(min, 0);
        return result;
    }

    // reduce interval if min and initial bracket the root
    if (yInitial * yMin < 0) {
        return solve(f, min, yMin, initial, yInitial, min, yMin);
    }

    // return the second endpoint if it is good enough
    double yMax = f.value(max);
    if (Math.abs(yMax) <= functionValueAccuracy) {
        setResult(max, 0);
        return result;
    }

    // reduce interval if initial and max bracket the root
    if (yInitial * yMax < 0) {
        return solve(f, initial, yInitial, max, yMax, initial, yInitial);
    }

    if (yMin * yMax > 0) {
        throw MathRuntimeException.createIllegalArgumentException(
              NON_BRACKETING_MESSAGE, min, max, yMin, yMax);
    }

    // full Brent algorithm starting with provided initial guess
    return solve(f, min, yMin, max, yMax, initial, yInitial);

}

Brent solver returns the wrong value if either bracket endpoint is root

The solve(final UnivariateRealFunction f, final double min, final double max, final double initial) function returns yMin or yMax if min or max are deemed to be roots, respectively, instead of min or max.


public double solve(final UnivariateRealFunction f,
                    final double min, final double max, final double initial)
    throws MaxIterationsExceededException, FunctionEvaluationException {

    clearResult();
    verifySequence(min, initial, max);

    // return the initial guess if it is good enough
    double yInitial = f.value(initial);
    if (Math.abs(yInitial) <= functionValueAccuracy) {
        setResult(initial, 0);
        return result;
    }

    // return the first endpoint if it is good enough
    double yMin = f.value(min);
    if (Math.abs(yMin) <= functionValueAccuracy) {
        setResult(yMin, 0);
        return result;
    }

    // reduce interval if min and initial bracket the root
    if (yInitial * yMin < 0) {
        return solve(f, min, yMin, initial, yInitial, min, yMin);
    }

    // return the second endpoint if it is good enough
    double yMax = f.value(max);
    if (Math.abs(yMax) <= functionValueAccuracy) {
        setResult(yMax, 0);
        return result;
    }

    // reduce interval if initial and max bracket the root
    if (yInitial * yMax < 0) {
        return solve(f, initial, yInitial, max, yMax, initial, yInitial);
    }

    // full Brent algorithm starting with provided initial guess
    return solve(f, min, yMin, max, yMax, initial, yInitial);

}

public double solve(final UnivariateRealFunction f,
                    final double min, final double max, final double initial)
    throws MaxIterationsExceededException, FunctionEvaluationException {

    clearResult();
    verifySequence(min, initial, max);

    // return the initial guess if it is good enough
    double yInitial = f.value(initial);
    if (Math.abs(yInitial) <= functionValueAccuracy) {
        setResult(initial, 0);
        return result;
    }

    // return the first endpoint if it is good enough
    double yMin = f.value(min);
    if (Math.abs(yMin) <= functionValueAccuracy) {
        setResult(yMin, 0);
        return result;
    }

    // reduce interval if min and initial bracket the root
    if (yInitial * yMin < 0) {
        return solve(f, min, yMin, initial, yInitial, min, yMin);
    }

    // return the second endpoint if it is good enough
    double yMax = f.value(max);
    if (Math.abs(yMax) <= functionValueAccuracy) {
        setResult(yMax, 0);
        return result;
    }

    // reduce interval if initial and max bracket the root
    if (yInitial * yMax < 0) {
        return solve(f, initial, yInitial, max, yMax, initial, yInitial);
    }

    if (yMin * yMax > 0) {
        throw MathRuntimeException.createIllegalArgumentException(
              NON_BRACKETING_MESSAGE, min, max, yMin, yMax);
    }

    // full Brent algorithm starting with provided initial guess
    return solve(f, min, yMin, max, yMax, initial, yInitial);

}

Brent solver doesn't throw IllegalArgumentException when initial guess has the wrong sign

Javadoc for "public double solve(final UnivariateRealFunction f, final double min, final double max, final double initial)" claims that "if the values of the function at the three points have the same sign" an IllegalArgumentException is thrown. This case isn't even checked.


public double getPct(Object v) {
    return getPct((Comparable<?>) v);
}

In stat.Frequency, getPct(Object) uses getCumPct(Comparable) instead of getPct(Comparable)

Drop in Replacement of 1.2 with 2.0 not possible because all getPct calls will be cummulative without code change
Frequency.java
   /**

Returns the percentage of values that are equal to v
@deprecated replaced by
{@link #getPct(Comparable)}
 as of 2.0
     */
    @Deprecated
    public double getPct(Object v)
{
        return getCumPct((Comparable<?>) v);
    }


public static double distance(int[] p1, int[] p2) {
  int sum = 0;
  for (int i = 0; i < p1.length; i++) {
      final int dp = p1[i] - p2[i];
      sum += dp * dp;
  }
  return Math.sqrt(sum);
}

public static double distance(int[] p1, int[] p2) {
  double sum = 0;
  for (int i = 0; i < p1.length; i++) {
      final double dp = p1[i] - p2[i];
      sum += dp * dp;
  }
  return Math.sqrt(sum);
}

NPE in  KMeansPlusPlusClusterer unittest

When running this unittest, I am facing this NPE:
java.lang.NullPointerException
	at org.apache.commons.math.stat.clustering.KMeansPlusPlusClusterer.assignPointsToClusters(KMeansPlusPlusClusterer.java:91)
This is the unittest:
package org.fao.fisheries.chronicles.calcuation.cluster;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertTrue;
import java.util.Arrays;
import java.util.List;
import java.util.Random;
import org.apache.commons.math.stat.clustering.Cluster;
import org.apache.commons.math.stat.clustering.EuclideanIntegerPoint;
import org.apache.commons.math.stat.clustering.KMeansPlusPlusClusterer;
import org.fao.fisheries.chronicles.input.CsvImportProcess;
import org.fao.fisheries.chronicles.input.Top200Csv;
import org.junit.Test;
public class ClusterAnalysisTest {
	@Test
	public void testPerformClusterAnalysis2() {
		KMeansPlusPlusClusterer<EuclideanIntegerPoint> transformer = new KMeansPlusPlusClusterer<EuclideanIntegerPoint>(
				new Random(1746432956321l));
		EuclideanIntegerPoint[] points = new EuclideanIntegerPoint[] {
				new EuclideanIntegerPoint(new int[]
{ 1959, 325100 }
),
				new EuclideanIntegerPoint(new int[]
{ 1960, 373200 }
), };
		List<Cluster<EuclideanIntegerPoint>> clusters = transformer.cluster(Arrays.asList(points), 1, 1);
		assertEquals(1, clusters.size());
	}
}


public T[] sample(int sampleSize) throws NotStrictlyPositiveException {
    if (sampleSize <= 0) {
        throw new NotStrictlyPositiveException(LocalizedFormats.NUMBER_OF_SAMPLES,
                sampleSize);
    }

    final T[]out = (T[]) java.lang.reflect.Array.newInstance(singletons.get(0).getClass(), sampleSize);

    for (int i = 0; i < sampleSize; i++) {
        out[i] = sample();
    }

    return out;

}

public Object[] sample(int sampleSize) throws NotStrictlyPositiveException {
    if (sampleSize <= 0) {
        throw new NotStrictlyPositiveException(LocalizedFormats.NUMBER_OF_SAMPLES,
                sampleSize);
    }

    final Object[] out = new Object[sampleSize];

    for (int i = 0; i < sampleSize; i++) {
        out[i] = sample();
    }

    return out;

}

DiscreteDistribution.sample(int) may throw an exception if first element of singletons of sub-class type

Creating an array with Array.newInstance(singletons.get(0).getClass(), sampleSize) in DiscreteDistribution.sample(int) is risky. An exception will be thrown if:

singleons.get(0) is of type T1, an sub-class of T, and
DiscreteDistribution.sample() returns an object which is of type T, but not of type T1.

To reproduce:

List<Pair<Object,Double>> list = new ArrayList<Pair<Object, Double>>();
list.add(new Pair<Object, Double>(new Object() {}, new Double(0)));
list.add(new Pair<Object, Double>(new Object() {}, new Double(1)));
new DiscreteDistribution<Object>(list).sample(1);

Attaching a patch.


private Integer getPivotRow(final int col, final SimplexTableau tableau) {
    double minRatio = Double.MAX_VALUE;
    Integer minRatioPos = null;
    for (int i = tableau.getNumObjectiveFunctions(); i < tableau.getHeight(); i++) {
        final double rhs = tableau.getEntry(i, tableau.getWidth() - 1);
        final double entry = tableau.getEntry(i, col);
        if (MathUtils.compareTo(entry, 0, epsilon) >= 0) {
            final double ratio = rhs / entry;
            if (ratio < minRatio) {
                minRatio = ratio;
                minRatioPos = i;
            }
        }
    }
    return minRatioPos;
}

private Integer getPivotRow(final int col, final SimplexTableau tableau) {
    double minRatio = Double.MAX_VALUE;
    Integer minRatioPos = null;
    for (int i = tableau.getNumObjectiveFunctions(); i < tableau.getHeight(); i++) {
        final double rhs = tableau.getEntry(i, tableau.getWidth() - 1);
        final double entry = tableau.getEntry(i, col);
        if (MathUtils.compareTo(entry, 0, epsilon) > 0) {
            final double ratio = rhs / entry;
            if (ratio < minRatio) {
                minRatio = ratio;
                minRatioPos = i;
            }
        }
    }
    return minRatioPos;
}

SimplexSolver didn't find the optimal solution.
Program for Lpsolve:
=====================
/* Objective function */
max: 7 a 3 b;
/* Constraints */
R1: +3 a -5 c <= 0;
R2: +2 a -5 d <= 0;
R3: +2 b -5 c <= 0;
R4: +3 b -5 d <= 0;
R5: +3 a +2 b <= 5;
R6: +2 a +3 b <= 5;
/* Variable bounds */
a <= 1;
b <= 1;
=====================
Results(correct): a = 1, b = 1, value = 10
Program for SimplexSolve:
=====================
LinearObjectiveFunction kritFcia = new LinearObjectiveFunction(new double[]
{7, 3, 0, 0}
, 0);
Collection<LinearConstraint> podmienky = new ArrayList<LinearConstraint>();
podmienky.add(new LinearConstraint(new double[]
{1, 0, 0, 0}
, Relationship.LEQ, 1));
podmienky.add(new LinearConstraint(new double[]
{0, 1, 0, 0}
, Relationship.LEQ, 1));
podmienky.add(new LinearConstraint(new double[]
{3, 0, -5, 0}
, Relationship.LEQ, 0));
podmienky.add(new LinearConstraint(new double[]
{2, 0, 0, -5}
, Relationship.LEQ, 0));
podmienky.add(new LinearConstraint(new double[]
{0, 2, -5, 0}
, Relationship.LEQ, 0));
podmienky.add(new LinearConstraint(new double[]
{0, 3, 0, -5}
, Relationship.LEQ, 0));
podmienky.add(new LinearConstraint(new double[]
{3, 2, 0, 0}
, Relationship.LEQ, 5));
podmienky.add(new LinearConstraint(new double[]
{2, 3, 0, 0}
, Relationship.LEQ, 5));
SimplexSolver solver = new SimplexSolver();
RealPointValuePair result = solver.optimize(kritFcia, podmienky, GoalType.MAXIMIZE, true);
=====================
Results(incorrect): a = 1, b = 0.5, value = 8.5
P.S. I used the latest software from the repository (including MATH-286 fix).


protected void iterateSimplex(final Comparator<RealPointValuePair> comparator)
    throws FunctionEvaluationException, OptimizationException, IllegalArgumentException {

    while (true) {

        incrementIterationsCounter();

        // save the original vertex
        final RealPointValuePair[] original = simplex;
        final RealPointValuePair best = original[0];

        // perform a reflection step
        final RealPointValuePair reflected = evaluateNewSimplex(original, 1.0, comparator);
        if (comparator.compare(reflected, best) < 0) {

            // compute the expanded simplex
            final RealPointValuePair[] reflectedSimplex = simplex;
            final RealPointValuePair expanded = evaluateNewSimplex(original, khi, comparator);
            if (comparator.compare(reflected, expanded) <= 0) {
                // accept the reflected simplex
                simplex = reflectedSimplex;
            }

            return;

        }

        // compute the contracted simplex
        final RealPointValuePair contracted = evaluateNewSimplex(original, gamma, comparator);
        if (comparator.compare(contracted, best) < 0) {
            // accept the contracted simplex

        // check convergence
            return;
        }

    }

}

protected void iterateSimplex(final Comparator<RealPointValuePair> comparator)
    throws FunctionEvaluationException, OptimizationException, IllegalArgumentException {

    final RealConvergenceChecker checker = getConvergenceChecker();
    while (true) {

        incrementIterationsCounter();

        // save the original vertex
        final RealPointValuePair[] original = simplex;
        final RealPointValuePair best = original[0];

        // perform a reflection step
        final RealPointValuePair reflected = evaluateNewSimplex(original, 1.0, comparator);
        if (comparator.compare(reflected, best) < 0) {

            // compute the expanded simplex
            final RealPointValuePair[] reflectedSimplex = simplex;
            final RealPointValuePair expanded = evaluateNewSimplex(original, khi, comparator);
            if (comparator.compare(reflected, expanded) <= 0) {
                // accept the reflected simplex
                simplex = reflectedSimplex;
            }

            return;

        }

        // compute the contracted simplex
        final RealPointValuePair contracted = evaluateNewSimplex(original, gamma, comparator);
        if (comparator.compare(contracted, best) < 0) {
            // accept the contracted simplex
            return;
        }

        // check convergence
        final int iter = getIterations();
        boolean converged = true;
        for (int i = 0; i < simplex.length; ++i) {
            converged &= checker.converged(iter, original[i], simplex[i]);
        }
        if (converged) {
            return;
        }

    }

}

MultiDirectional optimzation loops forver if started at the correct solution

MultiDirectional.iterateSimplex loops forever if the starting point is the correct solution.
see the attached test case (testMultiDirectionalCorrectStart) as an example.


private Integer getBasicRow(final int col) {
    Integer row = null;
    for (int i = getNumObjectiveFunctions(); i < getHeight(); i++) {
        if (!MathUtils.equals(getEntry(i, col), 0.0, epsilon)) {
            if (row == null) {
            row = i;
            } else {
            return null;
            }
        }
    }
    return row;
}

private Integer getBasicRow(final int col) {
    Integer row = null;
    for (int i = getNumObjectiveFunctions(); i < getHeight(); i++) {
        if (MathUtils.equals(getEntry(i, col), 1.0, epsilon) && (row == null)) {
            row = i;
        } else if (!MathUtils.equals(getEntry(i, col), 0.0, epsilon)) {
            return null;
        }
    }
    return row;
}

Basic variable is not found correctly in simplex tableau

The last patch to SimplexTableau caused an automated test suite I'm running at work to go down a new code path and uncover what is hopefully the last bug remaining in the Simplex code.
SimplexTableau was assuming an entry in the tableau had to be nonzero to indicate a basic variable, which is incorrect - the entry should have a value equal to 1.


protected RealPointValuePair getSolution() {
    double[] coefficients = new double[getOriginalNumDecisionVariables()];
    Integer basicRow =
        getBasicRow(getNumObjectiveFunctions() + getOriginalNumDecisionVariables());
    double mostNegative = basicRow == null ? 0 : getEntry(basicRow, getRhsOffset());
    for (int i = 0; i < coefficients.length; i++) {
        basicRow = getBasicRow(getNumObjectiveFunctions() + i);
            // if multiple variables can take a given value
            // then we choose the first and set the rest equal to 0
            coefficients[i] =
                (basicRow == null ? 0 : getEntry(basicRow, getRhsOffset())) -
                (restrictToNonNegative ? 0 : mostNegative);
        if (basicRow != null) {
            for (int j = getNumObjectiveFunctions(); j < getNumObjectiveFunctions() + i; j++) {
                if (tableau.getEntry(basicRow, j) == 1) {
                     coefficients[i] = 0;
                }
            }
        }
    }
    return new RealPointValuePair(coefficients, f.getValue(coefficients));
}

protected RealPointValuePair getSolution() {
    double[] coefficients = new double[getOriginalNumDecisionVariables()];
    Integer basicRow =
        getBasicRow(getNumObjectiveFunctions() + getOriginalNumDecisionVariables());
    double mostNegative = basicRow == null ? 0 : getEntry(basicRow, getRhsOffset());
    Set<Integer> basicRows = new HashSet<Integer>();
    for (int i = 0; i < coefficients.length; i++) {
        basicRow = getBasicRow(getNumObjectiveFunctions() + i);
        if (basicRows.contains(basicRow)) {
            // if multiple variables can take a given value
            // then we choose the first and set the rest equal to 0
            coefficients[i] = 0;
        } else {
            basicRows.add(basicRow);
            coefficients[i] =
                (basicRow == null ? 0 : getEntry(basicRow, getRhsOffset())) -
                (restrictToNonNegative ? 0 : mostNegative);
        }
    }
    return new RealPointValuePair(coefficients, f.getValue(coefficients));
}

Simplex Solver arrives at incorrect solution

I have reduced the problem reported to me down to a minimal test case which I will attach.


public void addValue(Object v) {
        addValue((Comparable<?>) v);
}

public void addValue(Object v) {
    if (v instanceof Comparable<?>){
        addValue((Comparable<?>) v);
    } else {
        throw new IllegalArgumentException("Object must implement Comparable");
    }
}

I think the existing Frequency API has some bugs in it.
The addValue(Object v) method allows one to add a plain Object, but one cannot add anything further to the instance, as the second add fails with IllegalArgumentException.
In fact, the problem is with the first call to addValue(Object) which should not allow a plain Object to be added - it should only allow Comparable objects.
This could be fixed by checking that the object is Comparable.
Similar considerations apply to the getCumFreq(Object) and getCumPct(Object) methods - they will only work with objects that implement Comparable.
The getCount(Object) and getPct(Object) methods don't fail when given a non-Comparable object (because the class cast exception is caught), however they just return 0 as if the object was not present:

        final Object OBJ = new Object();
        f.addValue(OBJ); // This ought to fail, but doesn't, causing the unexpected behaviour below
        System.out.println(f.getCount(OBJ)); // 0
        System.out.println(f.getPct(OBJ)); // 0.0

Rather than adding extra checks for Comparable, it seems to me that the API would be much improved by using Comparable instead of Object.
Also, it should make it easier to implement generics.
However, this would cause compilation failures for some programs that pass Object rather than Comparable to the class.
These would need recoding, but I think they would continue to run OK against the new API.
It would also affect the run-time behaviour slightly, as the first attempt to add a non-Comparable object would fail, rather than the second add of a possibly valid object.
But is that a viable program? It can only add one object, and any attempt to get statistics will either return 0 or an Exception, and applying the instanceof fix would also cause it to fail.


public Line revert() {
    final Line reverted = new Line(zero, zero.subtract(direction));
    return reverted;
}

public Line revert() {
    final Line reverted = new Line(this);
    reverted.direction = reverted.direction.negate();
    return reverted;
}

Line.revert() only maintains ~10 digits for the direction. This becomes an issue when the line's position is evaluated far from the origin. A simple fix would be to use Vector3D.negate() for the direction.
Also, is there a reason why Line is not immutable? It is just comprised of two vectors.


public void addValue(Object v) {

/**
 * Adds 1 to the frequency count for v.
 *

 * If other objects have already been added to this Frequency, v must
 * be comparable to those that have already been added.
 *

 *
 * @param v the value to add.
 * @throws IllegalArgumentException if v is not comparable with previous entries
 */
    Object obj = v;
    if (v instanceof Integer) {
       obj = Long.valueOf(((Integer) v).longValue());
    }
    try {
        Long count = (Long) freqTable.get(obj);
        if (count == null) {
            freqTable.put(obj, Long.valueOf(1));
        } else {
            freqTable.put(obj, Long.valueOf(count.longValue() + 1));
        }
    } catch (ClassCastException ex) {
        //TreeMap will throw ClassCastException if v is not comparable
        throw new IllegalArgumentException("Value not comparable to existing values.");
    }
}

public void addValue(Object v) {
    addValue((Comparable<?>) v);
}


public int compareTo(Fraction object) {
    double nOd = doubleValue();
    double dOn = object.doubleValue();
    return (nOd < dOn) ? -1 : ((nOd > dOn) ? +1 : 0);
}

public int compareTo(Fraction object) {
    long nOd = ((long) numerator) * object.denominator;
    long dOn = ((long) denominator) * object.numerator;
    return (nOd < dOn) ? -1 : ((nOd > dOn) ? +1 : 0);
}

Fraction.comparTo returns 0 for some differente fractions

If two different fractions evaluate to the same double due to limited precision,
the compareTo methode returns 0 as if they were identical.

// value is roughly PI - 3.07e-18
Fraction pi1 = new Fraction(1068966896, 340262731);

// value is roughly PI + 1.936e-17
Fraction pi2 = new Fraction( 411557987, 131002976);

System.out.println(pi1.doubleValue() - pi2.doubleValue()); // exactly 0.0 due to limited IEEE754 precision
System.out.println(pi1.compareTo(pi2)); // display 0 instead of a negative value


public static int gcd(int u, int v) {
    if (u * v == 0) {
        return (Math.abs(u) + Math.abs(v));
    }
    // keep u and v negative, as negative integers range down to
    // -2^31, while positive numbers can only be as large as 2^31-1
    // (i.e. we can't necessarily negate a negative number without
    // overflow)
    /* assert u!=0 && v!=0; */
    if (u > 0) {
        u = -u;
    } // make u negative
    if (v > 0) {
        v = -v;
    } // make v negative
    // B1. [Find power of 2]
    int k = 0;
    while ((u & 1) == 0 && (v & 1) == 0 && k < 31) { // while u and v are
                                                        // both even...
        u /= 2;
        v /= 2;
        k++; // cast out twos.
    }
    if (k == 31) {
        throw new ArithmeticException("overflow: gcd is 2^31");
    }
    // B2. Initialize: u and v have been divided by 2^k and at least
    // one is odd.
    int t = ((u & 1) == 1) ? v : -(u / 2)/* B3 */;
    // t negative: u was odd, v may be even (t replaces v)
    // t positive: u was even, v is odd (t replaces u)
    do {
        /* assert u<0 && v<0; */
        // B4/B3: cast out twos from t.
        while ((t & 1) == 0) { // while t is even..
            t /= 2; // cast out twos
        }
        // B5 [reset max(u,v)]
        if (t > 0) {
            u = -t;
        } else {
            v = t;
        }
        // B6/B3. at this point both u and v should be odd.
        t = (v - u) / 2;
        // |u| larger: t positive (replace u)
        // |v| larger: t negative (replace v)
    } while (t != 0);
    return -u * (1 << k); // gcd is u*2^k
}

public static int gcd(int u, int v) {
    if ((u == 0) || (v == 0)) {
        return (Math.abs(u) + Math.abs(v));
    }
    // keep u and v negative, as negative integers range down to
    // -2^31, while positive numbers can only be as large as 2^31-1
    // (i.e. we can't necessarily negate a negative number without
    // overflow)
    /* assert u!=0 && v!=0; */
    if (u > 0) {
        u = -u;
    } // make u negative
    if (v > 0) {
        v = -v;
    } // make v negative
    // B1. [Find power of 2]
    int k = 0;
    while ((u & 1) == 0 && (v & 1) == 0 && k < 31) { // while u and v are
                                                        // both even...
        u /= 2;
        v /= 2;
        k++; // cast out twos.
    }
    if (k == 31) {
        throw new ArithmeticException("overflow: gcd is 2^31");
    }
    // B2. Initialize: u and v have been divided by 2^k and at least
    // one is odd.
    int t = ((u & 1) == 1) ? v : -(u / 2)/* B3 */;
    // t negative: u was odd, v may be even (t replaces v)
    // t positive: u was even, v is odd (t replaces u)
    do {
        /* assert u<0 && v<0; */
        // B4/B3: cast out twos from t.
        while ((t & 1) == 0) { // while t is even..
            t /= 2; // cast out twos
        }
        // B5 [reset max(u,v)]
        if (t > 0) {
            u = -t;
        } else {
            v = t;
        }
        // B6/B3. at this point both u and v should be odd.
        t = (v - u) / 2;
        // |u| larger: t positive (replace u)
        // |v| larger: t negative (replace v)
    } while (t != 0);
    return -u * (1 << k); // gcd is u*2^k
}

MathUtils.gcd(u, v) fails when u and v both contain a high power of 2

The test at the beginning of MathUtils.gcd(u, v) for arguments equal to zero fails when u and v contain high enough powers of 2 so that their product overflows to zero.
        assertEquals(3 * (1<<15), MathUtils.gcd(3 * (1<<20), 9 * (1<<15)));
Fix: Replace the test at the start of MathUtils.gcd()
        if (u * v == 0) {
by
        if (u == 0 || v == 0) {


protected double getInitialDomain(double p) {
    double ret;
    double d = getDenominatorDegreesOfFreedom();
        // use mean
        ret = d / (d - 2.0);
    return ret;
}

protected double getInitialDomain(double p) {
    double ret = 1.0;
    double d = getDenominatorDegreesOfFreedom();
    if (d > 2.0) {
        // use mean
        ret = d / (d - 2.0);
    }
    return ret;
}

denominatorDegreeOfFreedom in FDistribution leads to IllegalArgumentsException in UnivariateRealSolverUtils.bracket

We are using the FDistributionImpl from the commons.math project to do
some statistical calculations, namely receiving the upper and lower
boundaries of a confidence interval. Everything is working fine and the
results are matching our reference calculations.
However, the FDistribution behaves strange if a
denominatorDegreeOfFreedom of 2 is used, with an alpha-value of 0.95.
This results in an IllegalArgumentsException, stating:
Invalid endpoint parameters:  lowerBound=0.0 initial=Infinity
upperBound=1.7976931348623157E308
coming from
org.apache.commons.math.analysis.UnivariateRealSolverUtils.bracket
The problem is the 'initial' parameter to that function, wich is
POSITIVE_INFINITY and therefore not within the boundaries. I already
pinned down the problem to the FDistributions getInitialDomain()-method,
wich goes like:
        return getDenominatorDegreesOfFreedom() /
                    (getDenominatorDegreesOfFreedom() - 2.0);
Obviously, in case of denominatorDegreesOfFreedom == 2, this must lead
to a division-by-zero, resulting in POSTIVE_INFINITY. The result of this
operation is then directly passed into the
UnivariateRealSolverUtils.bracket() - method as second argument.


public double solve(double min, double max) throws MaxIterationsExceededException,
    FunctionEvaluationException {

    clearResult();
    verifyInterval(min, max);

    double ret = Double.NaN;

    double yMin = f.value(min);
    double yMax = f.value(max);

    // Verify bracketing
    double sign = yMin * yMax;
    if (sign >= 0) {
        // check if either value is close to a zero
            // neither value is close to zero and min and max do not bracket root.
            throw new IllegalArgumentException
            ("Function values at endpoints do not have different signs." +
                    "  Endpoints: [" + min + "," + max + "]" +
                    "  Values: [" + yMin + "," + yMax + "]");
    } else {
        // solve using only the first endpoint as initial guess
        ret = solve(min, yMin, max, yMax, min, yMin);
        // either min or max is a root
    }

    return ret;
}

public double solve(double min, double max) throws MaxIterationsExceededException,
    FunctionEvaluationException {

    clearResult();
    verifyInterval(min, max);

    double ret = Double.NaN;

    double yMin = f.value(min);
    double yMax = f.value(max);

    // Verify bracketing
    double sign = yMin * yMax;
    if (sign > 0) {
        // check if either value is close to a zero
        if (Math.abs(yMin) <= functionValueAccuracy) {
            setResult(min, 0);
            ret = min;
        } else if (Math.abs(yMax) <= functionValueAccuracy) {
            setResult(max, 0);
            ret = max;
        } else {
            // neither value is close to zero and min and max do not bracket root.
            throw new IllegalArgumentException
            ("Function values at endpoints do not have different signs." +
                    "  Endpoints: [" + min + "," + max + "]" +
                    "  Values: [" + yMin + "," + yMax + "]");
        }
    } else if (sign < 0){
        // solve using only the first endpoint as initial guess
        ret = solve(min, yMin, max, yMax, min, yMin);
    } else {
        // either min or max is a root
        if (yMin == 0.0) {
            ret = min;
        } else {
            ret = max;
        }
    }

    return ret;
}

I am getting this exception:
java.lang.IllegalArgumentException: Function values at endpoints do not have different signs.  Endpoints: [-100000.0,1.7976931348623157E308]  Values: [0.0,-101945.04630982173]
at org.apache.commons.math.analysis.BrentSolver.solve(BrentSolver.java:99)
at org.apache.commons.math.analysis.BrentSolver.solve(BrentSolver.java:62)
The exception should not be thrown with values  [0.0,-101945.04630982173] because 0.0 is positive.
According to Brent Worden, the algorithm should stop and return 0 as the root instead of throwing an exception.
The problem comes from this method:
    public double solve(double min, double max) throws MaxIterationsExceededException,
        FunctionEvaluationException {
        clearResult();
        verifyInterval(min, max);
        double yMin = f.value(min);
        double yMax = f.value(max);
        // Verify bracketing
        if (yMin * yMax >= 0)
{
            throw new IllegalArgumentException
            ("Function values at endpoints do not have different signs." +
                    "  Endpoints: [" + min + "," + max + "]" +
                    "  Values: [" + yMin + "," + yMax + "]");
        }

        // solve using only the first endpoint as initial guess
        return solve(min, yMin, max, yMax, min, yMin);
    }
One way to fix it would be to add this code after the assignment of yMin and yMax:
        if (yMin ==0 || yMax == 0)
{
        	return 0;
       	}


public Class getGenericType(Field field) {
    Type generic = field.getGenericType();
    if (generic != null && generic instanceof ParameterizedType) {
        Type actual = ((ParameterizedType) generic).getActualTypeArguments()[0];
            return (Class) actual;
            //in case of nested generics we don't go deep
    }

    return Object.class;
}

public Class getGenericType(Field field) {
    Type generic = field.getGenericType();
    if (generic != null && generic instanceof ParameterizedType) {
        Type actual = ((ParameterizedType) generic).getActualTypeArguments()[0];
        if (actual instanceof Class) {
            return (Class) actual;
        } else if (actual instanceof ParameterizedType) {
            //in case of nested generics we don't go deep
            return (Class) ((ParameterizedType) actual).getRawType();
        }
    }

    return Object.class;
}

ArgumentCaptor no longer working for varargs

I ran into the issue described here: http://stackoverflow.com/questions/27303562/why-does-upgrading-mockito-from-1-9-5-to-1-10-8-break-this-captor


public Object handle(Invocation invocation) throws Throwable {
    if (invocationContainerImpl.hasAnswersForStubbing()) {
        // stubbing voids with stubVoid() or doAnswer() style
        InvocationMatcher invocationMatcher = matchersBinder.bindMatchers(mockingProgress
                        .getArgumentMatcherStorage(), invocation);
        invocationContainerImpl.setMethodForStubbing(invocationMatcher);
        return null;
    }
    VerificationMode verificationMode = mockingProgress.pullVerificationMode();

    InvocationMatcher invocationMatcher = matchersBinder.bindMatchers(mockingProgress.getArgumentMatcherStorage(),
                    invocation);

    mockingProgress.validateState();

    //if verificationMode is not null then someone is doing verify()
    if (verificationMode != null) {
        //We need to check if verification was started on the correct mock
        // - see VerifyingWithAnExtraCallToADifferentMockTest (bug 138)
        if (verificationMode instanceof MockAwareVerificationMode && ((MockAwareVerificationMode) verificationMode).getMock() == invocation.getMock()) {
            VerificationDataImpl data = new VerificationDataImpl(invocationContainerImpl, invocationMatcher);
            verificationMode.verify(data);
            return null;
            // this means there is an invocation on a different mock. Re-adding verification mode
            // - see VerifyingWithAnExtraCallToADifferentMockTest (bug 138)
        }
    }

    invocationContainerImpl.setInvocationForPotentialStubbing(invocationMatcher);
    OngoingStubbingImpl<T> ongoingStubbing = new OngoingStubbingImpl<T>(invocationContainerImpl);
    mockingProgress.reportOngoingStubbing(ongoingStubbing);

    StubbedInvocationMatcher stubbedInvocation = invocationContainerImpl.findAnswerFor(invocation);

    if (stubbedInvocation != null) {
        stubbedInvocation.captureArgumentsFrom(invocation);
        return stubbedInvocation.answer(invocation);
    } else {
        Object ret = mockSettings.getDefaultAnswer().answer(invocation);

        // redo setting invocation for potential stubbing in case of partial
        // mocks / spies.
        // Without it, the real method inside 'when' might have delegated
        // to other self method and overwrite the intended stubbed method
        // with a different one. The reset is required to avoid runtime exception that validates return type with stubbed method signature.
        invocationContainerImpl.resetInvocationForPotentialStubbing(invocationMatcher);
        return ret;
    }
}

public Object handle(Invocation invocation) throws Throwable {
    if (invocationContainerImpl.hasAnswersForStubbing()) {
        // stubbing voids with stubVoid() or doAnswer() style
        InvocationMatcher invocationMatcher = matchersBinder.bindMatchers(mockingProgress
                        .getArgumentMatcherStorage(), invocation);
        invocationContainerImpl.setMethodForStubbing(invocationMatcher);
        return null;
    }
    VerificationMode verificationMode = mockingProgress.pullVerificationMode();

    InvocationMatcher invocationMatcher = matchersBinder.bindMatchers(mockingProgress.getArgumentMatcherStorage(),
                    invocation);

    mockingProgress.validateState();

    //if verificationMode is not null then someone is doing verify()
    if (verificationMode != null) {
        //We need to check if verification was started on the correct mock
        // - see VerifyingWithAnExtraCallToADifferentMockTest (bug 138)
        if (((MockAwareVerificationMode) verificationMode).getMock() == invocation.getMock()) {
            VerificationDataImpl data = new VerificationDataImpl(invocationContainerImpl, invocationMatcher);
            verificationMode.verify(data);
            return null;
        } else {
            // this means there is an invocation on a different mock. Re-adding verification mode
            // - see VerifyingWithAnExtraCallToADifferentMockTest (bug 138)
            mockingProgress.verificationStarted(verificationMode);
        }
    }

    invocationContainerImpl.setInvocationForPotentialStubbing(invocationMatcher);
    OngoingStubbingImpl<T> ongoingStubbing = new OngoingStubbingImpl<T>(invocationContainerImpl);
    mockingProgress.reportOngoingStubbing(ongoingStubbing);

    StubbedInvocationMatcher stubbedInvocation = invocationContainerImpl.findAnswerFor(invocation);

    if (stubbedInvocation != null) {
        stubbedInvocation.captureArgumentsFrom(invocation);
        return stubbedInvocation.answer(invocation);
    } else {
        Object ret = mockSettings.getDefaultAnswer().answer(invocation);

        // redo setting invocation for potential stubbing in case of partial
        // mocks / spies.
        // Without it, the real method inside 'when' might have delegated
        // to other self method and overwrite the intended stubbed method
        // with a different one. The reset is required to avoid runtime exception that validates return type with stubbed method signature.
        invocationContainerImpl.resetInvocationForPotentialStubbing(invocationMatcher);
        return ret;
    }
}

@bric3, can you take a look at this one? If you don't have time I'll just merge it. All existing tests are passing.
Thanks for the fix!!!


public OngoingInjecter filterCandidate(final Collection<Object> mocks, final Field field, final Object fieldInstance) {
        if(mocks.size() == 1) {
            final Object matchingMock = mocks.iterator().next();

            return new OngoingInjecter() {
                public boolean thenInject() {
                    try {
                            new FieldSetter(fieldInstance, field).set(matchingMock);
                    } catch (Exception e) {
                        throw new MockitoException("Problems injecting dependency in " + field.getName(), e);
                    }
                    return true;
                }
            };
        }

        return new OngoingInjecter() {
            public boolean thenInject() {
                return false;
            }
        };

    }

public OngoingInjecter filterCandidate(final Collection<Object> mocks, final Field field, final Object fieldInstance) {
        if(mocks.size() == 1) {
            final Object matchingMock = mocks.iterator().next();

            return new OngoingInjecter() {
                public boolean thenInject() {
                    try {
                        if (!new BeanPropertySetter(fieldInstance, field).set(matchingMock)) {
                            new FieldSetter(fieldInstance, field).set(matchingMock);
                        }
                    } catch (Exception e) {
                        throw new MockitoException("Problems injecting dependency in " + field.getName(), e);
                    }
                    return true;
                }
            };
        }

        return new OngoingInjecter() {
            public boolean thenInject() {
                return false;
            }
        };

    }

Fixes #188 . These commits should fix issue with capturing varargs.


Object returnValueFor(Class<?> type) {
    if (Primitives.isPrimitiveOrWrapper(type)) {
        return Primitives.defaultValueForPrimitiveOrWrapper(type);
        //new instances are used instead of Collections.emptyList(), etc.
        //to avoid UnsupportedOperationException if code under test modifies returned collection
    } else if (type == Collection.class) {
        return new LinkedList<Object>();
    } else if (type == Set.class) {
        return new HashSet<Object>();
    } else if (type == HashSet.class) {
        return new HashSet<Object>();
    } else if (type == SortedSet.class) {
        return new TreeSet<Object>();
    } else if (type == TreeSet.class) {
        return new TreeSet<Object>();
    } else if (type == LinkedHashSet.class) {
        return new LinkedHashSet<Object>();
    } else if (type == List.class) {
        return new LinkedList<Object>();
    } else if (type == LinkedList.class) {
        return new LinkedList<Object>();
    } else if (type == ArrayList.class) {
        return new ArrayList<Object>();
    } else if (type == Map.class) {
        return new HashMap<Object, Object>();
    } else if (type == HashMap.class) {
        return new HashMap<Object, Object>();
    } else if (type == SortedMap.class) {
        return new TreeMap<Object, Object>();
    } else if (type == TreeMap.class) {
        return new TreeMap<Object, Object>();
    } else if (type == LinkedHashMap.class) {
        return new LinkedHashMap<Object, Object>();
    }
    //Let's not care about the rest of collections.
    return null;
}

Object returnValueFor(Class<?> type) {
    if (Primitives.isPrimitiveOrWrapper(type)) {
        return Primitives.defaultValueForPrimitiveOrWrapper(type);
        //new instances are used instead of Collections.emptyList(), etc.
        //to avoid UnsupportedOperationException if code under test modifies returned collection
    } else if (type == Iterable.class) {
        return new ArrayList<Object>(0);
    } else if (type == Collection.class) {
        return new LinkedList<Object>();
    } else if (type == Set.class) {
        return new HashSet<Object>();
    } else if (type == HashSet.class) {
        return new HashSet<Object>();
    } else if (type == SortedSet.class) {
        return new TreeSet<Object>();
    } else if (type == TreeSet.class) {
        return new TreeSet<Object>();
    } else if (type == LinkedHashSet.class) {
        return new LinkedHashSet<Object>();
    } else if (type == List.class) {
        return new LinkedList<Object>();
    } else if (type == LinkedList.class) {
        return new LinkedList<Object>();
    } else if (type == ArrayList.class) {
        return new ArrayList<Object>();
    } else if (type == Map.class) {
        return new HashMap<Object, Object>();
    } else if (type == HashMap.class) {
        return new HashMap<Object, Object>();
    } else if (type == SortedMap.class) {
        return new TreeMap<Object, Object>();
    } else if (type == TreeMap.class) {
        return new TreeMap<Object, Object>();
    } else if (type == LinkedHashMap.class) {
        return new LinkedHashMap<Object, Object>();
    }
    //Let's not care about the rest of collections.
    return null;
}

http://code.google.com/p/mockito/issues/detail?id=175
I expect an Iterable to be mocked by default with an empty Iterable. I understand from the initial issue this behavior would be introduced in Mockito 2, but beta-8 still returns null.
Could we return null for Iterables ?
Should we have the same behavior for Iterator ?
Thanks


public Timer(long durationMillis) {
        this.durationMillis = durationMillis;
    }

public Timer(long durationMillis) {
        validateInput(durationMillis);
        this.durationMillis = durationMillis;
    }

Mockito.after() method accepts negative timeperiods and subsequent verifications always pass

e.g.

```
Runnable runnable = Mockito.mock(Runnable.class);
Mockito.verify(runnable, Mockito.never()).run(); // passes as expected
Mockito.verify(runnable, Mockito.after(1000).never()).run(); // passes as expected
Mockito.verify(runnable, Mockito.after(-1000).atLeastOnce()).run(); // passes incorrectly
```


public Object answer(InvocationOnMock invocation) {
    if (methodsGuru.isToString(invocation.getMethod())) {
        Object mock = invocation.getMock();
        MockName name = mockUtil.getMockName(mock);
        if (name.isDefault()) {
            return "Mock for " + mockUtil.getMockSettings(mock).getTypeToMock().getSimpleName() + ", hashCode: " + mock.hashCode();
        } else {
            return name.toString();
        }
    } else if (methodsGuru.isCompareToMethod(invocation.getMethod())) {
        //see issue 184.
        //mocks by default should return 0 if references are the same, otherwise some other value because they are not the same. Hence we return 1 (anything but 0 is good).
        //Only for compareTo() method by the Comparable interface
        return 1;
    }

    Class<?> returnType = invocation.getMethod().getReturnType();
    return returnValueFor(returnType);
}

public Object answer(InvocationOnMock invocation) {
    if (methodsGuru.isToString(invocation.getMethod())) {
        Object mock = invocation.getMock();
        MockName name = mockUtil.getMockName(mock);
        if (name.isDefault()) {
            return "Mock for " + mockUtil.getMockSettings(mock).getTypeToMock().getSimpleName() + ", hashCode: " + mock.hashCode();
        } else {
            return name.toString();
        }
    } else if (methodsGuru.isCompareToMethod(invocation.getMethod())) {
        //see issue 184.
        //mocks by default should return 0 if references are the same, otherwise some other value because they are not the same. Hence we return 1 (anything but 0 is good).
        //Only for compareTo() method by the Comparable interface
        return invocation.getMock() == invocation.getArguments()[0] ? 0 : 1;
    }

    Class<?> returnType = invocation.getMethod().getReturnType();
    return returnValueFor(returnType);
}


public <T> void resetMock(T mock) {
    MockHandlerInterface<T> oldMockHandler = getMockHandler(mock);
    MockHandler<T> newMockHandler = new MockHandler<T>(oldMockHandler);
    MethodInterceptorFilter newFilter = new MethodInterceptorFilter(newMockHandler, (MockSettingsImpl) org.mockito.Mockito.withSettings().defaultAnswer(org.mockito.Mockito.RETURNS_DEFAULTS));
    ((Factory) mock).setCallback(0, newFilter);
}

public <T> void resetMock(T mock) {
    MockHandlerInterface<T> oldMockHandler = getMockHandler(mock);
    MethodInterceptorFilter newFilter = newMethodInterceptorFilter(oldMockHandler.getMockSettings());
    ((Factory) mock).setCallback(0, newFilter);
}

Exception when stubbing more than once with when...thenThrow

If I create a mock and stub a method so it throws an exception and do that twice the first exception will be thrown upon invoking the second stub instruction.
Example:
@Test
public void testThrowException() {
    Object o = Mockito.mock(Object.class);
    // test behavior with Runtimeexception
    Mockito.when(o.toString()).thenThrow(RuntimeException.class);
    // ...
    // test behavior with another exception
    // this throws a RuntimeException
    Mockito.when(o.toString()).thenThrow(IllegalArgumentException.class);
    // ...
}

I can work around this if I do it the other way around with doThrow...when. But I lose type safety then. Can you fix this?


private void injectMockCandidate(Class<?> awaitingInjectionClazz, Set<Object> mocks, Object fieldInstance) {
    for(Field field : orderedInstanceFieldsFrom(awaitingInjectionClazz)) {
        mockCandidateFilter.filterCandidate(mocks, field, fieldInstance).thenInject();
    }
}

private void injectMockCandidate(Class<?> awaitingInjectionClazz, Set<Object> mocks, Object fieldInstance) {
    for(Field field : orderedInstanceFieldsFrom(awaitingInjectionClazz)) {
        Object injected = mockCandidateFilter.filterCandidate(mocks, field, fieldInstance).thenInject();
        mocks.remove(injected);
    }
}

nicer textual printing of typed parameters

When matchers fail but yield the same toString(), Mockito prints extra type information. However, the type information is awkwardly printed for Strings. I've encountered this issue while working on removing hard dependency to hamcrest.
//current:
someMethod(1, (Integer) 2);
someOther(1, "(String) 2");
//desired:
someOther(1, (String) "2");


public void describeTo(Description description) {
    description.appendText("same(");
    appendQuoting(description);
    description.appendText(wanted.toString());
    appendQuoting(description);
    description.appendText(")");
}

public void describeTo(Description description) {
    description.appendText("same(");
    appendQuoting(description);
    description.appendText(wanted == null ? "null" : wanted.toString());
    appendQuoting(description);
    description.appendText(")");
}

Fixes #228: fixed a verify() call example in @Captor javadoc


public void captureArgumentsFrom(Invocation invocation) {
        if (invocation.getMethod().isVarArgs()) {
            int indexOfVararg = invocation.getRawArguments().length - 1;
            for (int position = 0; position < indexOfVararg; position++) {
                Matcher m = matchers.get(position);
                if (m instanceof CapturesArguments) {
                    ((CapturesArguments) m).captureFrom(invocation.getArgumentAt(position, Object.class));
                }
            }
            for (int position = indexOfVararg; position < matchers.size(); position++) {
                Matcher m = matchers.get(position);
                if (m instanceof CapturesArguments) {
                    ((CapturesArguments) m).captureFrom(invocation.getRawArguments()[position - indexOfVararg]);
                }
            }
        } else {
            for (int position = 0; position < matchers.size(); position++) {
                Matcher m = matchers.get(position);
                if (m instanceof CapturesArguments) {
                    ((CapturesArguments) m).captureFrom(invocation.getArgumentAt(position, Object.class));
                }
            }
        }
    }

public void captureArgumentsFrom(Invocation invocation) {
        if (invocation.getMethod().isVarArgs()) {
            int indexOfVararg = invocation.getRawArguments().length - 1;
            for (int position = 0; position < indexOfVararg; position++) {
                Matcher m = matchers.get(position);
                if (m instanceof CapturesArguments) {
                    ((CapturesArguments) m).captureFrom(invocation.getArgumentAt(position, Object.class));
                }
            }
            for (Matcher m : uniqueMatcherSet(indexOfVararg)) {
                if (m instanceof CapturesArguments) {
                    Object rawArgument = invocation.getRawArguments()[indexOfVararg];
                    for (int i = 0; i < Array.getLength(rawArgument); i++) {
                        ((CapturesArguments) m).captureFrom(Array.get(rawArgument, i));
                    }
                }
            }
        } else {
            for (int position = 0; position < matchers.size(); position++) {
                Matcher m = matchers.get(position);
                if (m instanceof CapturesArguments) {
                    ((CapturesArguments) m).captureFrom(invocation.getArgumentAt(position, Object.class));
                }
            }
        }
    }


@SuppressWarnings("deprecation")
    public void process(Class<?> context, Object testClass) {
        Field[] fields = context.getDeclaredFields();
        for (Field field : fields) {
            if (field.isAnnotationPresent(Spy.class)) {
                assertNoAnnotations(Spy.class, field, Mock.class, org.mockito.MockitoAnnotations.Mock.class, Captor.class);
                boolean wasAccessible = field.isAccessible();
                field.setAccessible(true);
                try {
                    Object instance = field.get(testClass);
                    if (instance == null) {
                        throw new MockitoException("Cannot create a @Spy for '" + field.getName() + "' field because the *instance* is missing\n" +
                        		  "The instance must be created *before* initMocks();\n" +
                                  "Example of correct usage of @Spy:\n" +
                            	  "   @Spy List mock = new LinkedList();\n" +
                            	  "   //also, don't forget about MockitoAnnotations.initMocks();");

                    }
                    if (new MockUtil().isMock(instance)) {
                        // instance has been spied earlier
                        Mockito.reset(instance);
                    } else {
                        field.set(testClass, Mockito.spy(instance));
                    }
                } catch (IllegalAccessException e) {
                    throw new MockitoException("Problems initiating spied field " + field.getName(), e);
                } finally {
                    field.setAccessible(wasAccessible);
                }
            }
        }
    }

@SuppressWarnings("deprecation")
    public void process(Class<?> context, Object testClass) {
        Field[] fields = context.getDeclaredFields();
        for (Field field : fields) {
            if (field.isAnnotationPresent(Spy.class)) {
                assertNoAnnotations(Spy.class, field, Mock.class, org.mockito.MockitoAnnotations.Mock.class, Captor.class);
                boolean wasAccessible = field.isAccessible();
                field.setAccessible(true);
                try {
                    Object instance = field.get(testClass);
                    if (instance == null) {
                        throw new MockitoException("Cannot create a @Spy for '" + field.getName() + "' field because the *instance* is missing\n" +
                        		  "The instance must be created *before* initMocks();\n" +
                                  "Example of correct usage of @Spy:\n" +
                            	  "   @Spy List mock = new LinkedList();\n" +
                            	  "   //also, don't forget about MockitoAnnotations.initMocks();");

                    }
                    if (new MockUtil().isMock(instance)) {
                        // instance has been spied earlier
                        Mockito.reset(instance);
                    } else {
                        field.set(testClass, Mockito.mock(instance.getClass(), withSettings()
                                .spiedInstance(instance)
                                .defaultAnswer(Mockito.CALLS_REAL_METHODS)
                                .name(field.getName())));
                    }
                } catch (IllegalAccessException e) {
                    throw new MockitoException("Problems initiating spied field " + field.getName(), e);
                } finally {
                    field.setAccessible(wasAccessible);
                }
            }
        }
    }

Mockito can't create mock on public class that extends package-private class

I created simple project to demonstrate this:
https://github.com/astafev/mockito-package-private-class/

Please take a look. Even if it can't be implemented, I think that mockito should throw some normal exception at time of creation.
In my variant on first creation it returns wrong-working mock (invokes real method instead of stubbed). On second creation throws exception that doesn't really connected with problem.

Everything works fine if you mock package-private parent.


public boolean hasSameMethod(Invocation candidate) {
    //not using method.equals() for 1 good reason:
    //sometimes java generates forwarding methods when generics are in play see JavaGenericsForwardingMethodsTest
    Method m1 = invocation.getMethod();
    Method m2 = candidate.getMethod();

    	/* Avoid unnecessary cloning */
    return m1.equals(m2);
}

public boolean hasSameMethod(Invocation candidate) {
    //not using method.equals() for 1 good reason:
    //sometimes java generates forwarding methods when generics are in play see JavaGenericsForwardingMethodsTest
    Method m1 = invocation.getMethod();
    Method m2 = candidate.getMethod();

    if (m1.getName() != null && m1.getName().equals(m2.getName())) {
    	/* Avoid unnecessary cloning */
    	Class[] params1 = m1.getParameterTypes();
    	Class[] params2 = m2.getParameterTypes();
    	if (params1.length == params2.length) {
    	    for (int i = 0; i < params1.length; i++) {
    		if (params1[i] != params2[i])
    		    return false;
    	    }
    	    return true;
    	}
    }
    return false;
}

ArgumentCaptor.fromClass's return type should match a parameterized type.  I.e. the expression ArgumentCaptor.fromClass(Class<S>) should be of type ArgumentCaptor<U> where S is a subtype of U.
For example:
ArgumentCaptor<Consumer<String>> captor = ArgumentCaptor.fromClass(Consumer.class)

does not type check (i.e. it is a compile time error). It should type check.
The reasons that it is desirable for ArgumentCaptor.fromClass to allow expressions such as the example above to type check are:

ArgumentCaptor.fromClass is intended to be a convenience method to allow the user to construct an ArgumentCaptor without casting the returned value.

Currently, the user can devise a workaround such as:
ArgumentCaptor<? extends Consumer<String>> captor
= ArgumentCaptor.fromClass(Consumer.class)

This workaround is inconvenient, and so contrary to ArgumentCaptor.fromClass being a convenience method.

It is inconsistent with @Captor, which can be applied to a field with a paramterized type.  I.e.

@Captor ArgumentCaptor<Consumer<String>> captor

type checks.


public void captureArgumentsFrom(Invocation i) {
    int k = 0;
    for (Matcher m : matchers) {
        if (m instanceof CapturesArguments) {
            ((CapturesArguments) m).captureFrom(i.getArguments()[k]);
        }
        k++;
    }
}

public void captureArgumentsFrom(Invocation i) {
    int k = 0;
    for (Matcher m : matchers) {
        if (m instanceof CapturesArguments && i.getArguments().length > k) {
            ((CapturesArguments) m).captureFrom(i.getArguments()[k]);
        }
        k++;
    }
}

Source files should not be put in binary JAR

Source files (*.java) should not be put into binary mockito-core.jar. It stupefies Idea to show decompiled file even when source jar is available.


public Object callRealMethod() throws Throwable {
        return realMethod.invoke(mock, rawArguments);
    }

public Object callRealMethod() throws Throwable {
        if (this.getMethod().getDeclaringClass().isInterface()) {
            new Reporter().cannotCallRealMethodOnInterface();
        }
        return realMethod.invoke(mock, rawArguments);
    }

- Mockito JUnit rule easier to use by avoiding the need to pass test instance
- Make it compatible with JUnit 4.7+ instead of 4.9+


public void validate(Answer<?> answer, Invocation invocation) {
        if (answer instanceof ThrowsException) {
            validateException((ThrowsException) answer, invocation);
        }

        if (answer instanceof Returns) {
            validateReturnValue((Returns) answer, invocation);
        }

        if (answer instanceof DoesNothing) {
            validateDoNothing((DoesNothing) answer, invocation);
        }

    }

public void validate(Answer<?> answer, Invocation invocation) {
        if (answer instanceof ThrowsException) {
            validateException((ThrowsException) answer, invocation);
        }

        if (answer instanceof Returns) {
            validateReturnValue((Returns) answer, invocation);
        }

        if (answer instanceof DoesNothing) {
            validateDoNothing((DoesNothing) answer, invocation);
        }

        if (answer instanceof CallsRealMethods) {
            validateMockingConcreteClass((CallsRealMethods) answer, invocation);
        }
    }


private boolean toStringEquals(Matcher m, Object arg) {
    return StringDescription.toString(m).equals(arg.toString());
}

private boolean toStringEquals(Matcher m, Object arg) {
    return StringDescription.toString(m).equals(arg == null? "null" : arg.toString());
}

Generate change list separated by types using labels

Changes Unknown when pulling 47a7016 on szpak:topic/releaseLabels into * on mockito:master*.


private void readTypeVariables() {
    for (Type type : typeVariable.getBounds()) {
        registerTypeVariablesOn(type);
    }
    registerTypeVariablesOn(getActualTypeArgumentFor(typeVariable));
}

private void readTypeVariables() {
    for (Type type : typeVariable.getBounds()) {
        registerTypeVariablesOn(type);
    }
    registerTypeParametersOn(new TypeVariable[] { typeVariable });
    registerTypeVariablesOn(getActualTypeArgumentFor(typeVariable));
}

Deep stubbing with generic responses in the call chain is not working

Deep stubbing will throw an Exception if multiple generics occur in the call chain. For instance, consider having a mock myMock1 that provides a function that returns a generic T. If T also has a function that returns a generic, an Exception with the message "Raw extraction not supported for : 'null'" will be thrown.
As an example the following test will throw an Exception:
public class MockitoGenericsDeepStubTest {

    @Test
    public void discoverDeepMockingOfGenerics() {
        MyClass1 myMock1 = mock(MyClass1.class, RETURNS_DEEP_STUBS);

        when(myMock1.getNested().getNested().returnSomething()).thenReturn("Hello World.");
    }

    public static interface MyClass1 <MC2 extends MyClass2> {
        public MC2 getNested();
    }

    public static interface MyClass2<MC3 extends MyClass3> {
        public MC3 getNested();
    }

    public static interface MyClass3 {
        public String returnSomething();
    }
}
You can make this test run if you step into the class ReturnsDeepStubs and change the method withSettingsUsing to return MockSettings with ReturnsDeepStubs instead of ReturnsDeepStubsSerializationFallback as default answer:
private MockSettings withSettingsUsing(GenericMetadataSupport returnTypeGenericMetadata, MockCreationSettings parentMockSettings) {
    MockSettings mockSettings = returnTypeGenericMetadata.hasRawExtraInterfaces() ?
            withSettings().extraInterfaces(returnTypeGenericMetadata.rawExtraInterfaces())
            : withSettings();

    return propagateSerializationSettings(mockSettings, parentMockSettings)
            .defaultAnswer(this);
}
However, this breaks other tests and features.
I think, the issue is that further generics are not possible to be mocked by ReturnsDeepStubsSerializationFallback since the GenericMetadataSupport is "closed" at this point.
Thanks and kind regards
Tobias


protected void registerTypeVariablesOn(Type classType) {
    if (!(classType instanceof ParameterizedType)) {
        return;
    }
    ParameterizedType parameterizedType = (ParameterizedType) classType;
    TypeVariable[] typeParameters = ((Class<?>) parameterizedType.getRawType()).getTypeParameters();
    Type[] actualTypeArguments = parameterizedType.getActualTypeArguments();
    for (int i = 0; i < actualTypeArguments.length; i++) {
        TypeVariable typeParameter = typeParameters[i];
        Type actualTypeArgument = actualTypeArguments[i];

        if (actualTypeArgument instanceof WildcardType) {
            contextualActualTypeParameters.put(typeParameter, boundsOf((WildcardType) actualTypeArgument));
        } else {
            contextualActualTypeParameters.put(typeParameter, actualTypeArgument);
        }
        // logger.log("For '" + parameterizedType + "' found type variable : { '" + typeParameter + "(" + System.identityHashCode(typeParameter) + ")" + "' : '" + actualTypeArgument + "(" + System.identityHashCode(typeParameter) + ")" + "' }");
    }
}

protected void registerTypeVariablesOn(Type classType) {
    if (!(classType instanceof ParameterizedType)) {
        return;
    }
    ParameterizedType parameterizedType = (ParameterizedType) classType;
    TypeVariable[] typeParameters = ((Class<?>) parameterizedType.getRawType()).getTypeParameters();
    Type[] actualTypeArguments = parameterizedType.getActualTypeArguments();
    for (int i = 0; i < actualTypeArguments.length; i++) {
        TypeVariable typeParameter = typeParameters[i];
        Type actualTypeArgument = actualTypeArguments[i];

        if (actualTypeArgument instanceof WildcardType) {
            contextualActualTypeParameters.put(typeParameter, boundsOf((WildcardType) actualTypeArgument));
        } else if (typeParameter != actualTypeArgument) {
            contextualActualTypeParameters.put(typeParameter, actualTypeArgument);
        }
        // logger.log("For '" + parameterizedType + "' found type variable : { '" + typeParameter + "(" + System.identityHashCode(typeParameter) + ")" + "' : '" + actualTypeArgument + "(" + System.identityHashCode(typeParameter) + ")" + "' }");
    }
}

1.10 regression (StackOverflowError) with interface where generic type has itself as upper bound

Add this to GenericMetadataSupportTest:
    interface GenericsSelfReference<T extends GenericsSelfReference<T>> {
        T self();
    }

    @Test
    public void typeVariable_of_self_type() {
        GenericMetadataSupport genericMetadata = inferFrom(GenericsSelfReference.class).resolveGenericReturnType(firstNamedMethod("self", GenericsSelfReference.class));

        assertThat(genericMetadata.rawType()).isEqualTo(GenericsSelfReference.class);
    }
It fails on master and 1.10.8 with this:
java.lang.StackOverflowError
    at sun.reflect.generics.reflectiveObjects.TypeVariableImpl.hashCode(TypeVariableImpl.java:201)
    at java.util.HashMap.hash(HashMap.java:338)
    at java.util.HashMap.get(HashMap.java:556)
    at org.mockito.internal.util.reflection.GenericMetadataSupport.getActualTypeArgumentFor(GenericMetadataSupport.java:193)
    at org.mockito.internal.util.reflection.GenericMetadataSupport.getActualTypeArgumentFor(GenericMetadataSupport.java:196)
    at org.mockito.internal.util.reflection.GenericMetadataSupport.getActualTypeArgumentFor(GenericMetadataSupport.java:196)

It worked on 1.9.5. May be caused by the changes in ab9e9f3 (cc @bric3).
(Also note that while the above interface looks strange, it is commonly used for builder hierarchies, where base class methods want to return this with a more specific type.)


public Object answer(InvocationOnMock invocation) throws Throwable {
        return invocation.callRealMethod();
    }

public Object answer(InvocationOnMock invocation) throws Throwable {
    	if (Modifier.isAbstract(invocation.getMethod().getModifiers())) {
    		return new GloballyConfiguredAnswer().answer(invocation);
    	}
        return invocation.callRealMethod();
    }

There's a problem with spying on abstract classes when the real implementation calls out to the abstract method. More details: #121


public int[] add(ReadablePartial partial, int fieldIndex, int[] values, int valueToAdd) {
    // overridden as superclass algorithm can't handle
    // 2004-02-29 + 48 months -> 2008-02-29 type dates
    if (valueToAdd == 0) {
        return values;
    }
        // month is largest field and being added to, such as month-day
    if (DateTimeUtils.isContiguous(partial)) {
        long instant = 0L;
        for (int i = 0, isize = partial.size(); i < isize; i++) {
            instant = partial.getFieldType(i).getField(iChronology).set(instant, values[i]);
        }
        instant = add(instant, valueToAdd);
        return iChronology.get(partial, instant);
    } else {
        return super.add(partial, fieldIndex, values, valueToAdd);
    }
}

public int[] add(ReadablePartial partial, int fieldIndex, int[] values, int valueToAdd) {
    // overridden as superclass algorithm can't handle
    // 2004-02-29 + 48 months -> 2008-02-29 type dates
    if (valueToAdd == 0) {
        return values;
    }
    if (partial.size() > 0 && partial.getFieldType(0).equals(DateTimeFieldType.monthOfYear()) && fieldIndex == 0) {
        // month is largest field and being added to, such as month-day
        int curMonth0 = partial.getValue(0) - 1;
        int newMonth = ((curMonth0 + (valueToAdd % 12) + 12) % 12) + 1;
        return set(partial, 0, values, newMonth);
    }
    if (DateTimeUtils.isContiguous(partial)) {
        long instant = 0L;
        for (int i = 0, isize = partial.size(); i < isize; i++) {
            instant = partial.getFieldType(i).getField(iChronology).set(instant, values[i]);
        }
        instant = add(instant, valueToAdd);
        return iChronology.get(partial, instant);
    } else {
        return super.add(partial, fieldIndex, values, valueToAdd);
    }
}

#151 Unable to add days to a MonthDay set to the ISO leap date

It's not possible to add days to a MonthDay set to the ISO leap date (February 29th). This is even more bizarre given the exact error message thrown.
Sample snippet:
final MonthDay isoLeap = new MonthDay(DateTimeConstants.FEBRUARY, 29, ISOChronology.getInstanceUTC());
System.out.println(isoLeap);
System.out.println(isoLeap.plusDays(2));

Which generates the following combined console output and stack trace:
--02-29
Exception in thread "main" org.joda.time.IllegalFieldValueException: Value 29 for dayOfMonth must be in the range [1,28]
at org.joda.time.field.FieldUtils.verifyValueBounds(FieldUtils.java:215)
at org.joda.time.field.PreciseDurationDateTimeField.set(PreciseDurationDateTimeField.java:78)
at org.joda.time.chrono.BasicMonthOfYearDateTimeField.add(BasicMonthOfYearDateTimeField.java:212)
at org.joda.time.field.BaseDateTimeField.add(BaseDateTimeField.java:324)
at org.joda.time.MonthDay.withFieldAdded(MonthDay.java:519)
at org.joda.time.MonthDay.minusDays(MonthDay.java:672)
at ext.site.time.chrono.Main.m7(Main.java:191)
at ext.site.time.chrono.Main.main(Main.java:27)
The follwing method calls and parameters also generate the same or related error:
isoLeap.plusMonths(1);
isoLeap.plusMonths(-1);
isoLeap.minusMonths(1);
isoLeap.minusMonths(-1);
isoLeap.minusDays(-1);

However, the following methods work:
isoLeap.minusDays(1);
isoLeap.plusDays(-1);

Performing operations on dates around the ISO leap date react as if it exists, ie:
System.out.println(isoLeap.minusDays(1).plusDays(2));

Prints out '--03-01' as expected.


public static long safeMultiply(long val1, int val2) {
    switch (val2) {
        case -1:
            return -val1;
        case 0:
            return 0L;
        case 1:
            return val1;
    }
    long total = val1 * val2;
    if (total / val2 != val1) {
      throw new ArithmeticException("Multiplication overflows a long: " + val1 + " * " + val2);
    }
    return total;
}

public static long safeMultiply(long val1, int val2) {
    switch (val2) {
        case -1:
            if (val1 == Long.MIN_VALUE) {
                throw new ArithmeticException("Multiplication overflows a long: " + val1 + " * " + val2);
            }
            return -val1;
        case 0:
            return 0L;
        case 1:
            return val1;
    }
    long total = val1 * val2;
    if (total / val2 != val1) {
      throw new ArithmeticException("Multiplication overflows a long: " + val1 + " * " + val2);
    }
    return total;
}

#147 possibly a bug in org.joda.time.field.FieldUtils.safeMultipl

It seems to me that as currently written in joda-time-2.1.jar
org.joda.time.field.FieldUtils.safeMultiply(long val1, int scalar)
doesn't detect the overflow if the long val1 == Long.MIN_VALUE and the int scalar == -1.
The attached file demonstrates what I think is the bug and suggests a patch.
I looked at the Joda Time bugs list in SourceForge but couldn't see anything that looked relevant: my apologies if I've missed something, or if I'm making a mistake with this bug report.
Colin Bartlett