private static int greatestCommonDivisor(int u, int v) {
    // From Commons Math:
    //if either operand is abs 1, return 1:
    if (Math.abs(u) <= 1 || Math.abs(v) <= 1) {
        return 1;
    }
    // keep u and v negative, as negative integers range down to
    // -2^31, while positive numbers can only be as large as 2^31-1
    // (i.e. we can't necessarily negate a negative number without
    // overflow)
    if (u>0) { u=-u; } // make u negative
    if (v>0) { v=-v; } // make v negative
    // B1. [Find power of 2]
    int k=0;
    while ((u&1)==0 && (v&1)==0 && k<31) { // while u and v are both even...
        u/=2; v/=2; k++; // cast out twos.
    }
    if (k==31) {
        throw new ArithmeticException("overflow: gcd is 2^31");
    }
    // B2. Initialize: u and v have been divided by 2^k and at least
    //     one is odd.
    int t = ((u&1)==1) ? v : -(u/2)/*B3*/;
    // t negative: u was odd, v may be even (t replaces v)
    // t positive: u was even, v is odd (t replaces u)
    do {
        /* assert u<0 && v<0; */
        // B4/B3: cast out twos from t.
        while ((t&1)==0) { // while t is even..
            t/=2; // cast out twos
        }
        // B5 [reset max(u,v)]
        if (t>0) {
            u = -t;
        } else {
            v = t;
        }
        // B6/B3. at this point both u and v should be odd.
        t = (v - u)/2;
        // |u| larger: t positive (replace u)
        // |v| larger: t negative (replace v)
    } while (t!=0);
    return -u*(1<<k); // gcd is u*2^k
}

private static int greatestCommonDivisor(int u, int v) {
    // From Commons Math:
    if ((u == 0) || (v == 0)) {
        if ((u == Integer.MIN_VALUE) || (v == Integer.MIN_VALUE)) {
            throw new ArithmeticException("overflow: gcd is 2^31");
        }
        return Math.abs(u) + Math.abs(v);
    }
    //if either operand is abs 1, return 1:
    if (Math.abs(u) == 1 || Math.abs(v) == 1) {
        return 1;
    }
    // keep u and v negative, as negative integers range down to
    // -2^31, while positive numbers can only be as large as 2^31-1
    // (i.e. we can't necessarily negate a negative number without
    // overflow)
    if (u>0) { u=-u; } // make u negative
    if (v>0) { v=-v; } // make v negative
    // B1. [Find power of 2]
    int k=0;
    while ((u&1)==0 && (v&1)==0 && k<31) { // while u and v are both even...
        u/=2; v/=2; k++; // cast out twos.
    }
    if (k==31) {
        throw new ArithmeticException("overflow: gcd is 2^31");
    }
    // B2. Initialize: u and v have been divided by 2^k and at least
    //     one is odd.
    int t = ((u&1)==1) ? v : -(u/2)/*B3*/;
    // t negative: u was odd, v may be even (t replaces v)
    // t positive: u was even, v is odd (t replaces u)
    do {
        /* assert u<0 && v<0; */
        // B4/B3: cast out twos from t.
        while ((t&1)==0) { // while t is even..
            t/=2; // cast out twos
        }
        // B5 [reset max(u,v)]
        if (t>0) {
            u = -t;
        } else {
            v = t;
        }
        // B6/B3. at this point both u and v should be odd.
        t = (v - u)/2;
        // |u| larger: t positive (replace u)
        // |v| larger: t negative (replace v)
    } while (t!=0);
    return -u*(1<<k); // gcd is u*2^k
}

org.apache.commons.lang3.math.Fraction does not reduce (Integer.MIN_VALUE, 2^k)

The greatestCommonDivisor method in class Fraction does not find the gcd of Integer.MIN_VALUE and 2^k, and this case can be triggered by taking Integer.MIN_VALUE as the numerator. Note that the case of taking Integer.MIN_VALUE as the denominator is handled explicitly in the getReducedFraction factory method.
FractionTest.java
	// additional test cases
	public void testReducedFactory_int_int() {
		// ...
		f = Fraction.getReducedFraction(Integer.MIN_VALUE, 2);
		assertEquals(Integer.MIN_VALUE / 2, f.getNumerator());
		assertEquals(1, f.getDenominator());

	public void testReduce() {
		// ...
		f = Fraction.getFraction(Integer.MIN_VALUE, 2);
		result = f.reduce();
		assertEquals(Integer.MIN_VALUE / 2, result.getNumerator());
		assertEquals(1, result.getDenominator());


public String format(Date date) {
    Calendar c = new GregorianCalendar(mTimeZone);
    c.setTime(date);
    return applyRules(c, new StringBuffer(mMaxLengthEstimate)).toString();
}

public String format(Date date) {
    Calendar c = new GregorianCalendar(mTimeZone, mLocale);
    c.setTime(date);
    return applyRules(c, new StringBuffer(mMaxLengthEstimate)).toString();
}

FastDateFormat.format() outputs incorrect week of year because locale isn't respected

FastDateFormat apparently doesn't respect the locale it was sent on creation when outputting week in year (e.g. "ww") in format(). It seems to use the settings of the system locale for firstDayOfWeek and minimalDaysInFirstWeek, which (depending on the year) may result in the incorrect week number being output.
Here is a simple test program to demonstrate the problem by comparing with SimpleDateFormat, which gets the week number right:

import java.util.Calendar;
import java.util.Date;
import java.util.Locale;
import java.text.SimpleDateFormat;

import org.apache.commons.lang.time.FastDateFormat;

public class FastDateFormatWeekBugDemo {
    public static void main(String[] args) {
        Locale.setDefault(new Locale("en", "US"));
        Locale locale = new Locale("sv", "SE");

        Calendar cal = Calendar.getInstance(); // setting locale here doesn't change outcome
        cal.set(2010, 0, 1, 12, 0, 0);
        Date d = cal.getTime();
        System.out.println("Target date: " + d);

        FastDateFormat fdf = FastDateFormat.getInstance("EEEE', week 'ww", locale);
        SimpleDateFormat sdf = new SimpleDateFormat("EEEE', week 'ww", locale);
        System.out.println("FastDateFormat:   " + fdf.format(d)); // will output "FastDateFormat:   fredag, week 01"
        System.out.println("SimpleDateFormat: " + sdf.format(d)); // will output "SimpleDateFormat: fredag, week 53"
    }
}

If sv/SE is passed to Locale.setDefault() instead of en/US, both FastDateFormat and SimpleDateFormat output the correct week number.


public int translate(CharSequence input, int index, Writer out) throws IOException {
    // TODO: Protect from ArrayIndexOutOfBounds
    if(input.charAt(index) == '&' && input.charAt(index + 1) == '#') {
        int start = index + 2;
        boolean isHex = false;

        char firstChar = input.charAt(start);
        if(firstChar == 'x' || firstChar == 'X') {
            start++;
            isHex = true;
        }

        int end = start;
        while(input.charAt(end) != ';') {
            end++;
        }

        int entityValue;
        try {
            if(isHex) {
                entityValue = Integer.parseInt(input.subSequence(start, end).toString(), 16);
            } else {
                entityValue = Integer.parseInt(input.subSequence(start, end).toString(), 10);
            }
        } catch(NumberFormatException nfe) {
            return 0;
        }

            out.write(entityValue);
        return 2 + (end - start) + (isHex ? 1 : 0) + 1;
    }
    return 0;
}

public int translate(CharSequence input, int index, Writer out) throws IOException {
    // TODO: Protect from ArrayIndexOutOfBounds
    if(input.charAt(index) == '&' && input.charAt(index + 1) == '#') {
        int start = index + 2;
        boolean isHex = false;

        char firstChar = input.charAt(start);
        if(firstChar == 'x' || firstChar == 'X') {
            start++;
            isHex = true;
        }

        int end = start;
        while(input.charAt(end) != ';') {
            end++;
        }

        int entityValue;
        try {
            if(isHex) {
                entityValue = Integer.parseInt(input.subSequence(start, end).toString(), 16);
            } else {
                entityValue = Integer.parseInt(input.subSequence(start, end).toString(), 10);
            }
        } catch(NumberFormatException nfe) {
            return 0;
        }

        if(entityValue > 0xFFFF) {
            char[] chrs = Character.toChars(entityValue);
            out.write(chrs[0]);
            out.write(chrs[1]);
        } else {
            out.write(entityValue);
        }
        return 2 + (end - start) + (isHex ? 1 : 0) + 1;
    }
    return 0;
}

StringEscapeUtils.escapeXML() can't process UTF-16 supplementary characters

Supplementary characters in UTF-16 are those whose code points are above 0xffff, that is, require more than 1 Java char to be encoded, as explained here: http://java.sun.com/developer/technicalArticles/Intl/Supplementary/
Currently, StringEscapeUtils.escapeXML() isn't aware of this coding scheme and treats each char as one character, which is not always right.
A possible solution in class Entities would be:
    public void escape(Writer writer, String str) throws IOException {
        int len = str.length();
        for (int i = 0; i < len; i++) {
            int code = str.codePointAt;
            String entityName = this.entityName(code);
            if (entityName != null)
{
                writer.write('&');
                writer.write(entityName);
                writer.write(';');
            }
 else if (code > 0x7F)
{
                    writer.write("&#");
                    writer.write(code);
                    writer.write(';');
            }
 else
{
                    writer.write((char) code);
            }

            if (code > 0xffff)
{
                    i++;
            }
        }
    }
Besides fixing escapeXML(), this will also affect HTML escaping functions. I guess that's a good thing, but please remember I have only tested escapeXML().


static float toJavaVersionInt(String version) {
    return toVersionInt(toJavaVersionIntArray(version, JAVA_VERSION_TRIM_SIZE));
}

static int toJavaVersionInt(String version) {
    return toVersionInt(toJavaVersionIntArray(version, JAVA_VERSION_TRIM_SIZE));
}

SystemUtils.getJavaVersionAsFloat throws StringIndexOutOfBoundsException on Android runtime/Dalvik VM

Can be replicated in the Android emulator quite easily.
Stack trace:

at org.apache.commons.lang.builder.ToStringBuilder.<clinit>(ToStringBuilder.java:98)
E/AndroidRuntime( 1681): 	... 17 more
E/AndroidRuntime( 1681): Caused by: java.lang.ExceptionInInitializerError
E/AndroidRuntime( 1681): 	at org.apache.commons.lang.builder.ToStringStyle$MultiLineToStringStyle.<init>(ToStringStyle.java:2276)
E/AndroidRuntime( 1681): 	at org.apache.commons.lang.builder.ToStringStyle.<clinit>(ToStringStyle.java:94)
E/AndroidRuntime( 1681): 	... 18 more
E/AndroidRuntime( 1681): Caused by: java.lang.StringIndexOutOfBoundsException
E/AndroidRuntime( 1681): 	at java.lang.String.substring(String.java:1571)
E/AndroidRuntime( 1681): 	at org.apache.commons.lang.SystemUtils.getJavaVersionAsFloat(SystemUtils.java:1153)
E/AndroidRuntime( 1681): 	at org.apache.commons.lang.SystemUtils.<clinit>(SystemUtils.java:818)


public static boolean containsAny(CharSequence cs, char[] searchChars) {
	if (isEmpty(cs) || ArrayUtils.isEmpty(searchChars)) {
		return false;
	}
	int csLength = cs.length();
	int searchLength = searchChars.length;
	for (int i = 0; i < csLength; i++) {
		char ch = cs.charAt(i);
		for (int j = 0; j < searchLength; j++) {
			if (searchChars[j] == ch) {
					// ch is a supplementary character
					// ch is in the Basic Multilingual Plane
					return true;
			}
		}
	}
	return false;
}

public static boolean containsAny(CharSequence cs, char[] searchChars) {
	if (isEmpty(cs) || ArrayUtils.isEmpty(searchChars)) {
		return false;
	}
	int csLength = cs.length();
	int searchLength = searchChars.length;
	int csLastIndex = csLength - 1;
	int searchLastIndex = searchLength - 1;
	for (int i = 0; i < csLength; i++) {
		char ch = cs.charAt(i);
		for (int j = 0; j < searchLength; j++) {
			if (searchChars[j] == ch) {
				if (i < csLastIndex && j < searchLastIndex && ch >= Character.MIN_HIGH_SURROGATE && ch <= Character.MAX_HIGH_SURROGATE) {
					// ch is a supplementary character
					if (searchChars[j + 1] == cs.charAt(i + 1)) {
						return true;
					}
				} else {
					// ch is in the Basic Multilingual Plane
					return true;
				}
			}
		}
	}
	return false;
}

StringUtils methods do not handle Unicode 2.0+ supplementary characters correctly.

StringUtils.containsAny methods incorrectly matches Unicode 2.0+ supplementary characters.
For example, define a test fixture to be the Unicode character U+20000 where U+20000 is written in Java source as "\uD840\uDC00"
	private static final String CharU20000 = "\uD840\uDC00";
	private static final String CharU20001 = "\uD840\uDC01";
You can see Unicode supplementary characters correctly implemented in the JRE call:
	assertEquals(-1, CharU20000.indexOf(CharU20001));
But this is broken:
	assertEquals(false, StringUtils.containsAny(CharU20000, CharU20001));
	assertEquals(false, StringUtils.containsAny(CharU20001, CharU20000));
This is fine:
	assertEquals(true, StringUtils.contains(CharU20000 + CharU20001, CharU20000));
	assertEquals(true, StringUtils.contains(CharU20000 + CharU20001, CharU20001));
	assertEquals(true, StringUtils.contains(CharU20000, CharU20000));
	assertEquals(false, StringUtils.contains(CharU20000, CharU20001));
because the method calls the JRE to perform the match.
More than you want to know:

http://java.sun.com/developer/technicalArticles/Intl/Supplementary/


public static Class<?>[] toClass(Object[] array) {
    if (array == null) {
        return null;
    } else if (array.length == 0) {
        return ArrayUtils.EMPTY_CLASS_ARRAY;
    }
    Class<?>[] classes = new Class[array.length];
    for (int i = 0; i < array.length; i++) {
        classes[i] = array[i].getClass();
    }
    return classes;
}

public static Class<?>[] toClass(Object[] array) {
    if (array == null) {
        return null;
    } else if (array.length == 0) {
        return ArrayUtils.EMPTY_CLASS_ARRAY;
    }
    Class<?>[] classes = new Class[array.length];
    for (int i = 0; i < array.length; i++) {
        classes[i] = array[i] == null ? null : array[i].getClass();
    }
    return classes;
}

ClassUtils.toClass(Object[]) throws NPE on null array element


public static <T> T[] addAll(T[] array1, T... array2) {
    if (array1 == null) {
        return clone(array2);
    } else if (array2 == null) {
        return clone(array1);
    }
    final Class<?> type1 = array1.getClass().getComponentType();
    T[] joinedArray = (T[]) Array.newInstance(type1, array1.length + array2.length);
    System.arraycopy(array1, 0, joinedArray, 0, array1.length);
        System.arraycopy(array2, 0, joinedArray, array1.length, array2.length);
        // Check if problem is incompatible types
    return joinedArray;
}

public static <T> T[] addAll(T[] array1, T... array2) {
    if (array1 == null) {
        return clone(array2);
    } else if (array2 == null) {
        return clone(array1);
    }
    final Class<?> type1 = array1.getClass().getComponentType();
    T[] joinedArray = (T[]) Array.newInstance(type1, array1.length + array2.length);
    System.arraycopy(array1, 0, joinedArray, 0, array1.length);
    try {
        System.arraycopy(array2, 0, joinedArray, array1.length, array2.length);
    } catch (ArrayStoreException ase) {
        // Check if problem is incompatible types
        final Class<?> type2 = array2.getClass().getComponentType();
        if (!type1.isAssignableFrom(type2)){
            throw new IllegalArgumentException("Cannot store "+type2.getName()+" in an array of "+type1.getName());
        }
        throw ase; // No, so rethrow original
    }
    return joinedArray;
}

ArrayUtils.addAll(T[] array1, T... array2) does not handle mixed types very well

ArrayUtils.addAll(T[] array1, T... array2) does not handle mixed array types very well.
The stack trace for
Number[] st = ArrayUtils.addAll(new Integer[]
{1}
, new Long[]
{2L}
);
starts:
java.lang.ArrayStoreException
	at java.lang.System.arraycopy(Native Method)
	at org.apache.commons.lang3.ArrayUtils.addAll(ArrayUtils.java:2962)
which is not all that obvious.
It would be a lot clearer if the method threw an IlegalArgumentException or similar.


public StringBuffer format(Calendar calendar, StringBuffer buf) {
    if (mTimeZoneForced) {
        calendar = (Calendar) calendar.clone();
        calendar.setTimeZone(mTimeZone);
    }
    return applyRules(calendar, buf);
}

public StringBuffer format(Calendar calendar, StringBuffer buf) {
    if (mTimeZoneForced) {
        calendar.getTime(); /// LANG-538
        calendar = (Calendar) calendar.clone();
        calendar.setTimeZone(mTimeZone);
    }
    return applyRules(calendar, buf);
}

DateFormatUtils.format does not correctly change Calendar TimeZone in certain situations

If a Calendar object is constructed in certain ways a call to Calendar.setTimeZone does not correctly change the Calendars fields.  Calling Calenar.getTime() seems to fix this problem.  While this is probably a bug in the JDK, it would be nice if DateFormatUtils was smart enough to detect/resolve this problem.
For example, the following unit test fails:

  public void testFormat_CalendarIsoMsZulu() {
    final String dateTime = "2009-10-16T16:42:16.000Z";

    // more commonly constructed with: cal = new GregorianCalendar(2009, 9, 16, 8, 42, 16)
    // for the unit test to work in any time zone, constructing with GMT-8 rather than default locale time zone
    GregorianCalendar cal = new GregorianCalendar(TimeZone.getTimeZone("GMT-8"));
    cal.clear();
    cal.set(2009, 9, 16, 8, 42, 16);

    FastDateFormat format = FastDateFormat.getInstance("yyyy-MM-dd'T'HH:mm:ss.SSS'Z'", TimeZone.getTimeZone("GMT"));
    assertEquals("dateTime", dateTime, format.format(cal));
  }

However, this unit test passes:

  public void testFormat_CalendarIsoMsZulu() {
    final String dateTime = "2009-10-16T16:42:16.000Z";
    GregorianCalendar cal = new GregorianCalendar(TimeZone.getTimeZone("GMT-8"));
    cal.clear();
    cal.set(2009, 9, 16, 8, 42, 16);
    cal.getTime();

    FastDateFormat format = FastDateFormat.getInstance("yyyy-MM-dd'T'HH:mm:ss.SSS'Z'", TimeZone.getTimeZone("GMT"));
    assertEquals("dateTime", dateTime, format.format(cal));
  }


public static boolean containsIgnoreCase(String str, String searchStr) {
    if (str == null || searchStr == null) {
        return false;
    }
    return contains(str.toUpperCase(), searchStr.toUpperCase());
}

public static boolean containsIgnoreCase(String str, String searchStr) {
    if (str == null || searchStr == null) {
        return false;
    }
    int len = searchStr.length();
    int max = str.length() - len;
    for (int i = 0; i <= max; i++) {
        if (str.regionMatches(true, i, searchStr, 0, len)) {
            return true;
        }
    }
    return false;
}

String.to*Case() is locale-sensitive, this is usually not intended for case-insensitive comparisions. Please see Common Bug #3 for details.


public void escape(Writer writer, String str) throws IOException {
    int len = str.length();
    for (int i = 0; i < len; i++) {
        char c = str.charAt(i);
        String entityName = this.entityName(c);
        if (entityName == null) {
            if (c > 0x7F) {
                writer.write("&#");
                writer.write(Integer.toString(c, 10));
                writer.write(';');
            } else {
                writer.write(c);
            }
        } else {
            writer.write('&');
            writer.write(entityName);
            writer.write(';');
        }
    }
}

public void escape(Writer writer, String str) throws IOException {
    int len = str.length();
    for (int i = 0; i < len; i++) {
        int c = Character.codePointAt(str, i);
        String entityName = this.entityName(c);
        if (entityName == null) {
            if (c >= 0x010000 && i < len - 1) {
                writer.write("&#");
                writer.write(Integer.toString(c, 10));
                writer.write(';');
                i++;
            } else if (c > 0x7F) {
                writer.write("&#");
                writer.write(Integer.toString(c, 10));
                writer.write(';');
            } else {
                writer.write(c);
            }
        } else {
            writer.write('&');
            writer.write(entityName);
            writer.write(';');
        }
    }
}

StringEscapeUtils.escapeHtml incorrectly converts unicode characters above U+00FFFF into 2 characters

Characters that are represented as a 2 characters internaly by java are incorrectly converted by the function. The following test displays the problem quite nicely:
import org.apache.commons.lang.*;
public class J2 {
    public static void main(String[] args) throws Exception {
        // this is the utf8 representation of the character:
        // COUNTING ROD UNIT DIGIT THREE
        // in unicode
        // codepoint: U+1D362
        byte[] data = new byte[]
{ (byte)0xF0, (byte)0x9D, (byte)0x8D, (byte)0xA2 }
;
        //output is: ��
        // should be: 𝍢
        System.out.println("'" + StringEscapeUtils.escapeHtml(new String(data, "UTF8")) + "'");
    }
}
Should be very quick to fix, feel free to drop me an email if you want a patch.


private StringBuffer appendQuotedString(String pattern, ParsePosition pos,
        StringBuffer appendTo, boolean escapingOn) {
    int start = pos.getIndex();
    char[] c = pattern.toCharArray();
    if (escapingOn && c[start] == QUOTE) {
        return appendTo == null ? null : appendTo.append(QUOTE);
    }
    int lastHold = start;
    for (int i = pos.getIndex(); i < pattern.length(); i++) {
        if (escapingOn && pattern.substring(i).startsWith(ESCAPED_QUOTE)) {
            appendTo.append(c, lastHold, pos.getIndex() - lastHold).append(
                    QUOTE);
            pos.setIndex(i + ESCAPED_QUOTE.length());
            lastHold = pos.getIndex();
            continue;
        }
        switch (c[pos.getIndex()]) {
        case QUOTE:
            next(pos);
            return appendTo == null ? null : appendTo.append(c, lastHold,
                    pos.getIndex() - lastHold);
        default:
            next(pos);
        }
    }
    throw new IllegalArgumentException(
            "Unterminated quoted string at position " + start);
}

private StringBuffer appendQuotedString(String pattern, ParsePosition pos,
        StringBuffer appendTo, boolean escapingOn) {
    int start = pos.getIndex();
    char[] c = pattern.toCharArray();
    if (escapingOn && c[start] == QUOTE) {
        next(pos);
        return appendTo == null ? null : appendTo.append(QUOTE);
    }
    int lastHold = start;
    for (int i = pos.getIndex(); i < pattern.length(); i++) {
        if (escapingOn && pattern.substring(i).startsWith(ESCAPED_QUOTE)) {
            appendTo.append(c, lastHold, pos.getIndex() - lastHold).append(
                    QUOTE);
            pos.setIndex(i + ESCAPED_QUOTE.length());
            lastHold = pos.getIndex();
            continue;
        }
        switch (c[pos.getIndex()]) {
        case QUOTE:
            next(pos);
            return appendTo == null ? null : appendTo.append(c, lastHold,
                    pos.getIndex() - lastHold);
        default:
            next(pos);
        }
    }
    throw new IllegalArgumentException(
            "Unterminated quoted string at position " + start);
}

ExtendedMessageFormat: OutOfMemory with custom format registry and a pattern containing single quotes

When using ExtendedMessageFormat with a custom format registry and a pattern conatining single quotes, an OutOfMemoryError will occur.
Example that will cause error:
ExtendedMessageFormatTest.java

private static Map<String, Object> formatRegistry = new HashMap<String, Object>();
    static {
        formatRegistry.put(DummyFormatFactory.DUMMY_FORMAT, new DummyFormatFactory());
    }

    public static void main(String[] args) {
        ExtendedMessageFormat mf = new ExtendedMessageFormat("it''s a {dummy} 'test'!", formatRegistry);
        String formattedPattern = mf.format(new String[] {"great"});
        System.out.println(formattedPattern);
    }
}

The following change starting at line 421 on the 2.4 release seems to fix the problem:
ExtendedMessageFormat.java
CURRENT (Broken):
if (escapingOn && c[start] == QUOTE) {
        return appendTo == null ? null : appendTo.append(QUOTE);
}

WORKING:
if (escapingOn && c[start] == QUOTE) {
        next(pos);
        return appendTo == null ? null : appendTo.append(QUOTE);
}


public static String abbreviate(String str, int lower, int upper, String appendToEnd) {
    // initial parameter checks
    if (str == null) {
        return null;
    }
    if (str.length() == 0) {
        return StringUtils.EMPTY;
    }

    // if the lower value is greater than the length of the string,
    // set to the length of the string
    // if the upper value is -1 (i.e. no limit) or is greater
    // than the length of the string, set to the length of the string
    if (upper == -1 || upper > str.length()) {
        upper = str.length();
    }
    // if upper is less than lower, raise it to lower
    if (upper < lower) {
        upper = lower;
    }

    StringBuffer result = new StringBuffer();
    int index = StringUtils.indexOf(str, " ", lower);
    if (index == -1) {
        result.append(str.substring(0, upper));
        // only if abbreviation has occured do we append the appendToEnd value
        if (upper != str.length()) {
            result.append(StringUtils.defaultString(appendToEnd));
        }
    } else if (index > upper) {
        result.append(str.substring(0, upper));
        result.append(StringUtils.defaultString(appendToEnd));
    } else {
        result.append(str.substring(0, index));
        result.append(StringUtils.defaultString(appendToEnd));
    }
    return result.toString();
}

public static String abbreviate(String str, int lower, int upper, String appendToEnd) {
    // initial parameter checks
    if (str == null) {
        return null;
    }
    if (str.length() == 0) {
        return StringUtils.EMPTY;
    }

    // if the lower value is greater than the length of the string,
    // set to the length of the string
    if (lower > str.length()) {
        lower = str.length();
    }
    // if the upper value is -1 (i.e. no limit) or is greater
    // than the length of the string, set to the length of the string
    if (upper == -1 || upper > str.length()) {
        upper = str.length();
    }
    // if upper is less than lower, raise it to lower
    if (upper < lower) {
        upper = lower;
    }

    StringBuffer result = new StringBuffer();
    int index = StringUtils.indexOf(str, " ", lower);
    if (index == -1) {
        result.append(str.substring(0, upper));
        // only if abbreviation has occured do we append the appendToEnd value
        if (upper != str.length()) {
            result.append(StringUtils.defaultString(appendToEnd));
        }
    } else if (index > upper) {
        result.append(str.substring(0, upper));
        result.append(StringUtils.defaultString(appendToEnd));
    } else {
        result.append(str.substring(0, index));
        result.append(StringUtils.defaultString(appendToEnd));
    }
    return result.toString();
}

WordUtils.abbreviate bug when lower is greater than str.length

In WordUtils.abbreviate, upper is adjusted to the length of the string, then to lower.
But lower is never adjusted to the length of the string, so if lower is greater than str.lengt(), upper will be too...
Then, str.substring(0, upper) throw a StringIndexOutOfBoundsException
The fix is to adjust lower to the length of the string


public Fraction reduce() {
    int gcd = greatestCommonDivisor(Math.abs(numerator), denominator);
    if (gcd == 1) {
        return this;
    }
    return Fraction.getFraction(numerator / gcd, denominator / gcd);
}

public Fraction reduce() {
    if (numerator == 0) {
        return equals(ZERO) ? this : ZERO;
    }
    int gcd = greatestCommonDivisor(Math.abs(numerator), denominator);
    if (gcd == 1) {
        return this;
    }
    return Fraction.getFraction(numerator / gcd, denominator / gcd);
}

infinite loop in Fraction.reduce when numerator == 0


public static Locale toLocale(final String str) {
    if (str == null) {
        return null;
    }
    final int len = str.length();
    if (len < 2) {
        throw new IllegalArgumentException("Invalid locale format: " + str);
    }
    final char ch0 = str.charAt(0);
        final char ch1 = str.charAt(1);
        if (!Character.isLowerCase(ch0) || !Character.isLowerCase(ch1)) {
            throw new IllegalArgumentException("Invalid locale format: " + str);
        }
        if (len == 2) {
            return new Locale(str);
        }
        if (len < 5) {
            throw new IllegalArgumentException("Invalid locale format: " + str);
        }
        if (str.charAt(2) != '_') {
            throw new IllegalArgumentException("Invalid locale format: " + str);
        }
        final char ch3 = str.charAt(3);
        if (ch3 == '_') {
            return new Locale(str.substring(0, 2), "", str.substring(4));
        }
        final char ch4 = str.charAt(4);
        if (!Character.isUpperCase(ch3) || !Character.isUpperCase(ch4)) {
            throw new IllegalArgumentException("Invalid locale format: " + str);
        }
        if (len == 5) {
            return new Locale(str.substring(0, 2), str.substring(3, 5));
        }
        if (len < 7) {
            throw new IllegalArgumentException("Invalid locale format: " + str);
        }
        if (str.charAt(5) != '_') {
            throw new IllegalArgumentException("Invalid locale format: " + str);
        }
        return new Locale(str.substring(0, 2), str.substring(3, 5), str.substring(6));
}

public static Locale toLocale(final String str) {
    if (str == null) {
        return null;
    }
    final int len = str.length();
    if (len < 2) {
        throw new IllegalArgumentException("Invalid locale format: " + str);
    }
    final char ch0 = str.charAt(0);
    if (ch0 == '_') {
        if (len < 3) {
            throw new IllegalArgumentException("Invalid locale format: " + str);
        }
        final char ch1 = str.charAt(1);
        final char ch2 = str.charAt(2);
        if (!Character.isUpperCase(ch1) || !Character.isUpperCase(ch2)) {
            throw new IllegalArgumentException("Invalid locale format: " + str);
        }
        if (len == 3) {
            return new Locale("", str.substring(1, 3));
        }
        if (len < 5) {
            throw new IllegalArgumentException("Invalid locale format: " + str);
        }
        if (str.charAt(3) != '_') {
            throw new IllegalArgumentException("Invalid locale format: " + str);
        }
        return new Locale("", str.substring(1, 3), str.substring(4));
    } else {
        final char ch1 = str.charAt(1);
        if (!Character.isLowerCase(ch0) || !Character.isLowerCase(ch1)) {
            throw new IllegalArgumentException("Invalid locale format: " + str);
        }
        if (len == 2) {
            return new Locale(str);
        }
        if (len < 5) {
            throw new IllegalArgumentException("Invalid locale format: " + str);
        }
        if (str.charAt(2) != '_') {
            throw new IllegalArgumentException("Invalid locale format: " + str);
        }
        final char ch3 = str.charAt(3);
        if (ch3 == '_') {
            return new Locale(str.substring(0, 2), "", str.substring(4));
        }
        final char ch4 = str.charAt(4);
        if (!Character.isUpperCase(ch3) || !Character.isUpperCase(ch4)) {
            throw new IllegalArgumentException("Invalid locale format: " + str);
        }
        if (len == 5) {
            return new Locale(str.substring(0, 2), str.substring(3, 5));
        }
        if (len < 7) {
            throw new IllegalArgumentException("Invalid locale format: " + str);
        }
        if (str.charAt(5) != '_') {
            throw new IllegalArgumentException("Invalid locale format: " + str);
        }
        return new Locale(str.substring(0, 2), str.substring(3, 5), str.substring(6));
    }
}

LocaleUtils.toLocale does not parse strings starting with an underscore

Hi,
Javadocs of Locale.toString() states that "If the language is missing, the string will begin with an underbar.". This is not handled in the LocaleUtils.toLocale method if it is meant to be the inversion method of Locale.toString().
The fix for the ticket 328 does not handle well the case "fr__P", which I found out during fixing the first bug.
I am attaching the patch for both problems.


public static boolean toBoolean(String str) {
    // Previously used equalsIgnoreCase, which was fast for interned 'true'.
    // Non interned 'true' matched 15 times slower.
    //
    // Optimisation provides same performance as before for interned 'true'.
    // Similar performance for null, 'false', and other strings not length 2/3/4.
    // 'true'/'TRUE' match 4 times slower, 'tRUE'/'True' 7 times slower.
    if (str == "true") {
        return true;
    }
    if (str == null) {
        return false;
    }
    switch (str.length()) {
        case 2: {
            char ch0 = str.charAt(0);
            char ch1 = str.charAt(1);
            return
                (ch0 == 'o' || ch0 == 'O') &&
                (ch1 == 'n' || ch1 == 'N');
        }
        case 3: {
            char ch = str.charAt(0);
            if (ch == 'y') {
                return
                    (str.charAt(1) == 'e' || str.charAt(1) == 'E') &&
                    (str.charAt(2) == 's' || str.charAt(2) == 'S');
            }
            if (ch == 'Y') {
                return
                    (str.charAt(1) == 'E' || str.charAt(1) == 'e') &&
                    (str.charAt(2) == 'S' || str.charAt(2) == 's');
            }
        }
        case 4: {
            char ch = str.charAt(0);
            if (ch == 't') {
                return
                    (str.charAt(1) == 'r' || str.charAt(1) == 'R') &&
                    (str.charAt(2) == 'u' || str.charAt(2) == 'U') &&
                    (str.charAt(3) == 'e' || str.charAt(3) == 'E');
            }
            if (ch == 'T') {
                return
                    (str.charAt(1) == 'R' || str.charAt(1) == 'r') &&
                    (str.charAt(2) == 'U' || str.charAt(2) == 'u') &&
                    (str.charAt(3) == 'E' || str.charAt(3) == 'e');
            }
        }
    }
    return false;
}

public static boolean toBoolean(String str) {
    // Previously used equalsIgnoreCase, which was fast for interned 'true'.
    // Non interned 'true' matched 15 times slower.
    //
    // Optimisation provides same performance as before for interned 'true'.
    // Similar performance for null, 'false', and other strings not length 2/3/4.
    // 'true'/'TRUE' match 4 times slower, 'tRUE'/'True' 7 times slower.
    if (str == "true") {
        return true;
    }
    if (str == null) {
        return false;
    }
    switch (str.length()) {
        case 2: {
            char ch0 = str.charAt(0);
            char ch1 = str.charAt(1);
            return
                (ch0 == 'o' || ch0 == 'O') &&
                (ch1 == 'n' || ch1 == 'N');
        }
        case 3: {
            char ch = str.charAt(0);
            if (ch == 'y') {
                return
                    (str.charAt(1) == 'e' || str.charAt(1) == 'E') &&
                    (str.charAt(2) == 's' || str.charAt(2) == 'S');
            }
            if (ch == 'Y') {
                return
                    (str.charAt(1) == 'E' || str.charAt(1) == 'e') &&
                    (str.charAt(2) == 'S' || str.charAt(2) == 's');
            }
            return false;
        }
        case 4: {
            char ch = str.charAt(0);
            if (ch == 't') {
                return
                    (str.charAt(1) == 'r' || str.charAt(1) == 'R') &&
                    (str.charAt(2) == 'u' || str.charAt(2) == 'U') &&
                    (str.charAt(3) == 'e' || str.charAt(3) == 'E');
            }
            if (ch == 'T') {
                return
                    (str.charAt(1) == 'R' || str.charAt(1) == 'r') &&
                    (str.charAt(2) == 'U' || str.charAt(2) == 'u') &&
                    (str.charAt(3) == 'E' || str.charAt(3) == 'e');
            }
        }
    }
    return false;
}

BooleanUtils.toBoolean() - invalid drop-thru in case statement causes StringIndexOutOfBoundsException

The method BooleanUtils.toBoolean() has a case statement; case 3 drops through to case 4; this can cause StringIndexOutOfBoundsException, for example with the test:
assertEquals(false, BooleanUtils.toBoolean("tru"));
The end of case 3 should return false.
Patch to follow for source and unit test.


private static void escapeJavaStyleString(Writer out, String str, boolean escapeSingleQuote) throws IOException {
    if (out == null) {
        throw new IllegalArgumentException("The Writer must not be null");
    }
    if (str == null) {
        return;
    }
    int sz;
    sz = str.length();
    for (int i = 0; i < sz; i++) {
        char ch = str.charAt(i);

        // handle unicode
        if (ch > 0xfff) {
            out.write("\\u" + hex(ch));
        } else if (ch > 0xff) {
            out.write("\\u0" + hex(ch));
        } else if (ch > 0x7f) {
            out.write("\\u00" + hex(ch));
        } else if (ch < 32) {
            switch (ch) {
                case '\b':
                    out.write('\\');
                    out.write('b');
                    break;
                case '\n':
                    out.write('\\');
                    out.write('n');
                    break;
                case '\t':
                    out.write('\\');
                    out.write('t');
                    break;
                case '\f':
                    out.write('\\');
                    out.write('f');
                    break;
                case '\r':
                    out.write('\\');
                    out.write('r');
                    break;
                default :
                    if (ch > 0xf) {
                        out.write("\\u00" + hex(ch));
                    } else {
                        out.write("\\u000" + hex(ch));
                    }
                    break;
            }
        } else {
            switch (ch) {
                case '\'':
                    if (escapeSingleQuote) {
                      out.write('\\');
                    }
                    out.write('\'');
                    break;
                case '"':
                    out.write('\\');
                    out.write('"');
                    break;
                case '\\':
                    out.write('\\');
                    out.write('\\');
                    break;
                default :
                    out.write(ch);
                    break;
            }
        }
    }
}

private static void escapeJavaStyleString(Writer out, String str, boolean escapeSingleQuote) throws IOException {
    if (out == null) {
        throw new IllegalArgumentException("The Writer must not be null");
    }
    if (str == null) {
        return;
    }
    int sz;
    sz = str.length();
    for (int i = 0; i < sz; i++) {
        char ch = str.charAt(i);

        // handle unicode
        if (ch > 0xfff) {
            out.write("\\u" + hex(ch));
        } else if (ch > 0xff) {
            out.write("\\u0" + hex(ch));
        } else if (ch > 0x7f) {
            out.write("\\u00" + hex(ch));
        } else if (ch < 32) {
            switch (ch) {
                case '\b':
                    out.write('\\');
                    out.write('b');
                    break;
                case '\n':
                    out.write('\\');
                    out.write('n');
                    break;
                case '\t':
                    out.write('\\');
                    out.write('t');
                    break;
                case '\f':
                    out.write('\\');
                    out.write('f');
                    break;
                case '\r':
                    out.write('\\');
                    out.write('r');
                    break;
                default :
                    if (ch > 0xf) {
                        out.write("\\u00" + hex(ch));
                    } else {
                        out.write("\\u000" + hex(ch));
                    }
                    break;
            }
        } else {
            switch (ch) {
                case '\'':
                    if (escapeSingleQuote) {
                      out.write('\\');
                    }
                    out.write('\'');
                    break;
                case '"':
                    out.write('\\');
                    out.write('"');
                    break;
                case '\\':
                    out.write('\\');
                    out.write('\\');
                    break;
                case '/':
                    out.write('\\');
                    out.write('/');
                    break;
                default :
                    out.write(ch);
                    break;
            }
        }
    }
}

StringEscapeUtils.escapeJavaScript() method did not escape '/' into '\/', it will make IE render page uncorrectly

If Javascripts including'/', IE will parse the scripts uncorrectly, actually '/' should be escaped to '\/'.
For example, document.getElementById("test").value = '<script>alert(\'aaa\');</script>';this expression will make IE render page uncorrect, it should be document.getElementById("test").value = '<script>alert(\'aaa\');<\/script>';
Btw, Spring's JavascriptEscape behavor is correct.
Try  to run below codes, you will find the difference:
  String s = "<script>alert('aaa');</script>";
  String str = org.springframework.web.util.JavaScriptUtils.javaScriptEscape(s);
  System.out.println("Spring JS Escape : "+str);
  str = org.apache.commons.lang.StringEscapeUtils.escapeJavaScript(s);
  System.out.println("Apache Common Lang JS Escape : "+ str);


public static Locale toLocale(String str) {
    if (str == null) {
        return null;
    }
    int len = str.length();
    if (len != 2 && len != 5 && len < 7) {
        throw new IllegalArgumentException("Invalid locale format: " + str);
    }
    char ch0 = str.charAt(0);
    char ch1 = str.charAt(1);
    if (ch0 < 'a' || ch0 > 'z' || ch1 < 'a' || ch1 > 'z') {
        throw new IllegalArgumentException("Invalid locale format: " + str);
    }
    if (len == 2) {
        return new Locale(str, "");
    } else {
        if (str.charAt(2) != '_') {
            throw new IllegalArgumentException("Invalid locale format: " + str);
        }
        char ch3 = str.charAt(3);
        char ch4 = str.charAt(4);
        if (ch3 < 'A' || ch3 > 'Z' || ch4 < 'A' || ch4 > 'Z') {
            throw new IllegalArgumentException("Invalid locale format: " + str);
        }
        if (len == 5) {
            return new Locale(str.substring(0, 2), str.substring(3, 5));
        } else {
            if (str.charAt(5) != '_') {
                throw new IllegalArgumentException("Invalid locale format: " + str);
            }
            return new Locale(str.substring(0, 2), str.substring(3, 5), str.substring(6));
        }
    }
}

public static Locale toLocale(String str) {
    if (str == null) {
        return null;
    }
    int len = str.length();
    if (len != 2 && len != 5 && len < 7) {
        throw new IllegalArgumentException("Invalid locale format: " + str);
    }
    char ch0 = str.charAt(0);
    char ch1 = str.charAt(1);
    if (ch0 < 'a' || ch0 > 'z' || ch1 < 'a' || ch1 > 'z') {
        throw new IllegalArgumentException("Invalid locale format: " + str);
    }
    if (len == 2) {
        return new Locale(str, "");
    } else {
        if (str.charAt(2) != '_') {
            throw new IllegalArgumentException("Invalid locale format: " + str);
        }
        char ch3 = str.charAt(3);
        if (ch3 == '_') {
            return new Locale(str.substring(0, 2), "", str.substring(4));
        }
        char ch4 = str.charAt(4);
        if (ch3 < 'A' || ch3 > 'Z' || ch4 < 'A' || ch4 > 'Z') {
            throw new IllegalArgumentException("Invalid locale format: " + str);
        }
        if (len == 5) {
            return new Locale(str.substring(0, 2), str.substring(3, 5));
        } else {
            if (str.charAt(5) != '_') {
                throw new IllegalArgumentException("Invalid locale format: " + str);
            }
            return new Locale(str.substring(0, 2), str.substring(3, 5), str.substring(6));
        }
    }
}

LocaleUtils.toLocale() rejects strings with only language+variant

LocaleUtils.toLocale() throws an exception on strings containing a language and a variant but no country code. For example : fr__POSIX
This string can be produced with the JDK by instanciating a Locale with an empty string for the country : new Locale("fr", "", "POSIX").toString(). According to the javadoc for the Locale class a variant is allowed with just a language code or just a country code.
Commons Configuration handles this case in its PropertyConverter.toLocale() method. I'd like to replace our implementation by the one provided by LocaleUtils, but our tests fail due to this case.


public void stop() {
    if(this.runningState != STATE_RUNNING && this.runningState != STATE_SUSPENDED) {
        throw new IllegalStateException("Stopwatch is not running. ");
    }
        stopTime = System.currentTimeMillis();
    this.runningState = STATE_STOPPED;
}

public void stop() {
    if(this.runningState != STATE_RUNNING && this.runningState != STATE_SUSPENDED) {
        throw new IllegalStateException("Stopwatch is not running. ");
    }
    if(this.runningState == STATE_RUNNING) {
        stopTime = System.currentTimeMillis();
    }
    this.runningState = STATE_STOPPED;
}

StopWatch: suspend() acts as split(), if followed by stop()

In my opinion, it is a bug that suspend() acts as split(), if followed by stop(); see below:
        StopWatch sw = new StopWatch();
        sw.start();
        Thread.sleep(1000);
        sw.suspend();
        // Time 1 (ok)
        System.out.println(sw.getTime());
        Thread.sleep(2000);
        // Time 1 (again, ok)
        System.out.println(sw.getTime());
        sw.resume();
        Thread.sleep(3000);
        sw.suspend();
        // Time 2 (ok)
        System.out.println(sw.getTime());
        Thread.sleep(4000);
        // Time 2 (again, ok)
        System.out.println(sw.getTime());
        Thread.sleep(5000);
        sw.stop();
        // Time 2 (should be, but is Time 3 => NOT ok)
        System.out.println(sw.getTime());
suspend/resume is like a pause, where time counter doesn't continue. So a following stop()-call shouldn't increase the time counter, should it?


public static boolean isAvailableLocale(Locale locale) {
    return cAvailableLocaleSet.contains(locale);
}

public static boolean isAvailableLocale(Locale locale) {
    return availableLocaleList().contains(locale);
}

FindBugs pointed out:
   UwF: Field not initialized in constructor: org.apache.commons.lang.LocaleUtils.cAvailableLocaleSet
cAvailableSet is used directly once in the source - and if availableLocaleSet() hasn't been called it will cause a NullPointerException.


public StrBuilder appendFixedWidthPadRight(Object obj, int width, char padChar) {
    if (width > 0) {
        ensureCapacity(size + width);
        String str = (obj == null ? getNullText() : obj.toString());
        int strLen = str.length();
        if (strLen >= width) {
            str.getChars(0, strLen, buffer, size);
        } else {
            int padLen = width - strLen;
            str.getChars(0, strLen, buffer, size);
            for (int i = 0; i < padLen; i++) {
                buffer[size + strLen + i] = padChar;
            }
        }
        size += width;
    }
    return this;
}

public StrBuilder appendFixedWidthPadRight(Object obj, int width, char padChar) {
    if (width > 0) {
        ensureCapacity(size + width);
        String str = (obj == null ? getNullText() : obj.toString());
        int strLen = str.length();
        if (strLen >= width) {
            str.getChars(0, width, buffer, size);
        } else {
            int padLen = width - strLen;
            str.getChars(0, strLen, buffer, size);
            for (int i = 0; i < padLen; i++) {
                buffer[size + strLen + i] = padChar;
            }
        }
        size += width;
    }
    return this;
}

Bug in method appendFixedWidthPadRight of class StrBuilder causes an ArrayIndexOutOfBoundsException

There's a bug in method appendFixedWidthPadRight of class StrBuilder:
public StrBuilder appendFixedWidthPadRight(Object obj, int width, char padChar) {
        if (width > 0) {
            ensureCapacity(size + width);
            String str = (obj == null ? getNullText() : obj.toString());
            int strLen = str.length();
            if (strLen >= width)
{
 ==>            str.getChars(0, strLen, buffer, size);   <==== BUG: it should be str.getChars(0, width, buffer, size);
            }
 else {
                int padLen = width - strLen;
                str.getChars(0, strLen, buffer, size);
                for (int i = 0; i < padLen; i++)
{
                    buffer[size + strLen + i] = padChar;
                }
            }
            size += width;
        }
        return this;
    }
This is causing an ArrayIndexOutOfBoundsException, so this method is unusable when strLen > width.
It's counterpart method appendFixedWidthPadLeft seems to be ok.


public final void translate(CharSequence input, Writer out) throws IOException {
    if (out == null) {
        throw new IllegalArgumentException("The Writer must not be null");
    }
    if (input == null) {
        return;
    }
    int pos = 0;
    int len = input.length();
    while (pos < len) {
        int consumed = translate(input, pos, out);
        if (consumed == 0) {
            char[] c = Character.toChars(Character.codePointAt(input, pos));
            out.write(c);
            pos+= c.length;
            continue;
        }
        // contract with translators is that they have to understand codepoints
        // and they just took care of a surrogate pair
        for (int pt = 0; pt < consumed; pt++) {
            pos += Character.charCount(Character.codePointAt(input, pos));
        }
    }
}

public final void translate(CharSequence input, Writer out) throws IOException {
    if (out == null) {
        throw new IllegalArgumentException("The Writer must not be null");
    }
    if (input == null) {
        return;
    }
    int pos = 0;
    int len = input.length();
    while (pos < len) {
        int consumed = translate(input, pos, out);
        if (consumed == 0) {
            char[] c = Character.toChars(Character.codePointAt(input, pos));
            out.write(c);
            pos+= c.length;
            continue;
        }
        // contract with translators is that they have to understand codepoints
        // and they just took care of a surrogate pair
        for (int pt = 0; pt < consumed; pt++) {
            pos += Character.charCount(Character.codePointAt(input, pt));
        }
    }
}

I found that there is bad surrogate pair handling in the CharSequenceTranslator
This is a simple test case for this problem.
\uD83D\uDE30 is a surrogate pair.

@Test
public void testEscapeSurrogatePairs() throws Exception {
    assertEquals("\uD83D\uDE30", StringEscapeUtils.escapeCsv("\uD83D\uDE30"));
}

You'll get the exception as shown below.

java.lang.StringIndexOutOfBoundsException: String index out of range: 2
	at java.lang.String.charAt(String.java:658)
	at java.lang.Character.codePointAt(Character.java:4668)
	at org.apache.commons.lang3.text.translate.CharSequenceTranslator.translate(CharSequenceTranslator.java:95)
	at org.apache.commons.lang3.text.translate.CharSequenceTranslator.translate(CharSequenceTranslator.java:59)
	at org.apache.commons.lang3.StringEscapeUtils.escapeCsv(StringEscapeUtils.java:556)

Patch attached, the method affected:

public final void translate(CharSequence input, Writer out) throws IOException


private void init() {
    thisYear= Calendar.getInstance(timeZone, locale).get(Calendar.YEAR);

    nameValues= new ConcurrentHashMap<Integer, KeyValue[]>();

    StringBuilder regex= new StringBuilder();
    List<Strategy> collector = new ArrayList<Strategy>();

    Matcher patternMatcher= formatPattern.matcher(pattern);
    if(!patternMatcher.lookingAt()) {
        throw new IllegalArgumentException("Invalid pattern");
    }

    currentFormatField= patternMatcher.group();
    Strategy currentStrategy= getStrategy(currentFormatField);
    for(;;) {
        patternMatcher.region(patternMatcher.end(), patternMatcher.regionEnd());
        if(!patternMatcher.lookingAt()) {
            nextStrategy = null;
            break;
        }
        String nextFormatField= patternMatcher.group();
        nextStrategy = getStrategy(nextFormatField);
        if(currentStrategy.addRegex(this, regex)) {
            collector.add(currentStrategy);
        }
        currentFormatField= nextFormatField;
        currentStrategy= nextStrategy;
    }
    if(currentStrategy.addRegex(this, regex)) {
        collector.add(currentStrategy);
    }
    currentFormatField= null;
    strategies= collector.toArray(new Strategy[collector.size()]);
    parsePattern= Pattern.compile(regex.toString());
}

private void init() {
    thisYear= Calendar.getInstance(timeZone, locale).get(Calendar.YEAR);

    nameValues= new ConcurrentHashMap<Integer, KeyValue[]>();

    StringBuilder regex= new StringBuilder();
    List<Strategy> collector = new ArrayList<Strategy>();

    Matcher patternMatcher= formatPattern.matcher(pattern);
    if(!patternMatcher.lookingAt()) {
        throw new IllegalArgumentException("Invalid pattern");
    }

    currentFormatField= patternMatcher.group();
    Strategy currentStrategy= getStrategy(currentFormatField);
    for(;;) {
        patternMatcher.region(patternMatcher.end(), patternMatcher.regionEnd());
        if(!patternMatcher.lookingAt()) {
            nextStrategy = null;
            break;
        }
        String nextFormatField= patternMatcher.group();
        nextStrategy = getStrategy(nextFormatField);
        if(currentStrategy.addRegex(this, regex)) {
            collector.add(currentStrategy);
        }
        currentFormatField= nextFormatField;
        currentStrategy= nextStrategy;
    }
    if (patternMatcher.regionStart() != patternMatcher.regionEnd()) {
        throw new IllegalArgumentException("Failed to parse \""+pattern+"\" ; gave up at index "+patternMatcher.regionStart());
    }
    if(currentStrategy.addRegex(this, regex)) {
        collector.add(currentStrategy);
    }
    currentFormatField= null;
    strategies= collector.toArray(new Strategy[collector.size()]);
    parsePattern= Pattern.compile(regex.toString());
}

FastDateParser does not handle unterminated quotes correctly

FDP does not handled unterminated quotes the same way as SimpleDateFormat
For example:
Format: 'd'd'
Date: d3
This should fail to parse the format and date but it actually works.
The format is parsed as:
Pattern: d(\p
{IsNd}
++)


public void atan2(final double[] y, final int yOffset,
                  final double[] x, final int xOffset,
                  final double[] result, final int resultOffset) {

    // compute r = sqrt(x^2+y^2)
    double[] tmp1 = new double[getSize()];
    multiply(x, xOffset, x, xOffset, tmp1, 0);      // x^2
    double[] tmp2 = new double[getSize()];
    multiply(y, yOffset, y, yOffset, tmp2, 0);      // y^2
    add(tmp1, 0, tmp2, 0, tmp2, 0);                 // x^2 + y^2
    rootN(tmp2, 0, 2, tmp1, 0);                     // r = sqrt(x^2 + y^2)

    if (x[xOffset] >= 0) {

        // compute atan2(y, x) = 2 atan(y / (r + x))
        add(tmp1, 0, x, xOffset, tmp2, 0);          // r + x
        divide(y, yOffset, tmp2, 0, tmp1, 0);       // y /(r + x)
        atan(tmp1, 0, tmp2, 0);                     // atan(y / (r + x))
        for (int i = 0; i < tmp2.length; ++i) {
            result[resultOffset + i] = 2 * tmp2[i]; // 2 * atan(y / (r + x))
        }

    } else {

        // compute atan2(y, x) = +/- pi - 2 atan(y / (r - x))
        subtract(tmp1, 0, x, xOffset, tmp2, 0);     // r - x
        divide(y, yOffset, tmp2, 0, tmp1, 0);       // y /(r - x)
        atan(tmp1, 0, tmp2, 0);                     // atan(y / (r - x))
        result[resultOffset] =
                ((tmp2[0] <= 0) ? -FastMath.PI : FastMath.PI) - 2 * tmp2[0]; // +/-pi - 2 * atan(y / (r - x))
        for (int i = 1; i < tmp2.length; ++i) {
            result[resultOffset + i] = -2 * tmp2[i]; // +/-pi - 2 * atan(y / (r - x))
        }

    }

    // fix value to take special cases (+0/+0, +0/-0, -0/+0, -0/-0, +/-infinity) correctly

}

public void atan2(final double[] y, final int yOffset,
                  final double[] x, final int xOffset,
                  final double[] result, final int resultOffset) {

    // compute r = sqrt(x^2+y^2)
    double[] tmp1 = new double[getSize()];
    multiply(x, xOffset, x, xOffset, tmp1, 0);      // x^2
    double[] tmp2 = new double[getSize()];
    multiply(y, yOffset, y, yOffset, tmp2, 0);      // y^2
    add(tmp1, 0, tmp2, 0, tmp2, 0);                 // x^2 + y^2
    rootN(tmp2, 0, 2, tmp1, 0);                     // r = sqrt(x^2 + y^2)

    if (x[xOffset] >= 0) {

        // compute atan2(y, x) = 2 atan(y / (r + x))
        add(tmp1, 0, x, xOffset, tmp2, 0);          // r + x
        divide(y, yOffset, tmp2, 0, tmp1, 0);       // y /(r + x)
        atan(tmp1, 0, tmp2, 0);                     // atan(y / (r + x))
        for (int i = 0; i < tmp2.length; ++i) {
            result[resultOffset + i] = 2 * tmp2[i]; // 2 * atan(y / (r + x))
        }

    } else {

        // compute atan2(y, x) = +/- pi - 2 atan(y / (r - x))
        subtract(tmp1, 0, x, xOffset, tmp2, 0);     // r - x
        divide(y, yOffset, tmp2, 0, tmp1, 0);       // y /(r - x)
        atan(tmp1, 0, tmp2, 0);                     // atan(y / (r - x))
        result[resultOffset] =
                ((tmp2[0] <= 0) ? -FastMath.PI : FastMath.PI) - 2 * tmp2[0]; // +/-pi - 2 * atan(y / (r - x))
        for (int i = 1; i < tmp2.length; ++i) {
            result[resultOffset + i] = -2 * tmp2[i]; // +/-pi - 2 * atan(y / (r - x))
        }

    }

    // fix value to take special cases (+0/+0, +0/-0, -0/+0, -0/-0, +/-infinity) correctly
    result[resultOffset] = FastMath.atan2(y[yOffset], x[xOffset]);

}

DerivativeStructure.atan2(y,x) does not handle special cases properly

The four special cases +/-0 for both x and y should give the same values as Math.atan2 and FastMath.atan2. However, they give NaN for the value in all cases.


public Complex parse(String source, ParsePosition pos) {
    int initialIndex = pos.getIndex();

    // parse whitespace
    parseAndIgnoreWhitespace(source, pos);

    // parse real
    Number re = parseNumber(source, getRealFormat(), pos);
    if (re == null) {
        // invalid real number
        // set index back to initial, error index should already be set
        // character examined.
        pos.setIndex(initialIndex);
        return null;
    }

    // parse sign
    int startIndex = pos.getIndex();
    char c = parseNextCharacter(source, pos);
    int sign = 0;
    switch (c) {
    case 0 :
        // no sign
        // return real only complex number
        return new Complex(re.doubleValue(), 0.0);
    case '-' :
        sign = -1;
        break;
    case '+' :
        sign = 1;
        break;
    default :
        // invalid sign
        // set index back to initial, error index should be the last
        // character examined.
        pos.setIndex(initialIndex);
        pos.setErrorIndex(startIndex);
        return null;
    }

    // parse whitespace
    parseAndIgnoreWhitespace(source, pos);

    // parse imaginary
    Number im = parseNumber(source, getRealFormat(), pos);
    if (im == null) {
        // invalid imaginary number
        // set index back to initial, error index should already be set
        // character examined.
        pos.setIndex(initialIndex);
        return null;
    }

    // parse imaginary character
    int n = getImaginaryCharacter().length();
    startIndex = pos.getIndex();
    int endIndex = startIndex + n;
    if (
        source.substring(startIndex, endIndex).compareTo(
        getImaginaryCharacter()) != 0) {
        // set index back to initial, error index should be the start index
        // character examined.
        pos.setIndex(initialIndex);
        pos.setErrorIndex(startIndex);
        return null;
    }
    pos.setIndex(endIndex);

    return new Complex(re.doubleValue(), im.doubleValue() * sign);
}

public Complex parse(String source, ParsePosition pos) {
    int initialIndex = pos.getIndex();

    // parse whitespace
    parseAndIgnoreWhitespace(source, pos);

    // parse real
    Number re = parseNumber(source, getRealFormat(), pos);
    if (re == null) {
        // invalid real number
        // set index back to initial, error index should already be set
        // character examined.
        pos.setIndex(initialIndex);
        return null;
    }

    // parse sign
    int startIndex = pos.getIndex();
    char c = parseNextCharacter(source, pos);
    int sign = 0;
    switch (c) {
    case 0 :
        // no sign
        // return real only complex number
        return new Complex(re.doubleValue(), 0.0);
    case '-' :
        sign = -1;
        break;
    case '+' :
        sign = 1;
        break;
    default :
        // invalid sign
        // set index back to initial, error index should be the last
        // character examined.
        pos.setIndex(initialIndex);
        pos.setErrorIndex(startIndex);
        return null;
    }

    // parse whitespace
    parseAndIgnoreWhitespace(source, pos);

    // parse imaginary
    Number im = parseNumber(source, getRealFormat(), pos);
    if (im == null) {
        // invalid imaginary number
        // set index back to initial, error index should already be set
        // character examined.
        pos.setIndex(initialIndex);
        return null;
    }

    // parse imaginary character
    int n = getImaginaryCharacter().length();
    startIndex = pos.getIndex();
    int endIndex = startIndex + n;
    if ((startIndex >= source.length()) ||
        (endIndex > source.length()) ||
        source.substring(startIndex, endIndex).compareTo(
        getImaginaryCharacter()) != 0) {
        // set index back to initial, error index should be the start index
        // character examined.
        pos.setIndex(initialIndex);
        pos.setErrorIndex(startIndex);
        return null;
    }
    pos.setIndex(endIndex);

    return new Complex(re.doubleValue(), im.doubleValue() * sign);
}

java.lang.StringIndexOutOfBoundsException in ComplexFormat.parse(String source, ParsePosition pos)

The parse(String source, ParsePosition pos) method in the ComplexFormat class does not check whether the imaginary character is set or not which produces StringIndexOutOfBoundsException in the substring method :
(line 375 of ComplexFormat)
...
        // parse imaginary character
        int n = getImaginaryCharacter().length();
        startIndex = pos.getIndex();
        int endIndex = startIndex + n;
        if (source.substring(startIndex, endIndex).compareTo(
            getImaginaryCharacter()) != 0) {
...
I encoutered this exception typing in a JTextFied with ComplexFormat set to look up an AbstractFormatter.
If only the user types the imaginary part of the complex number first, he gets this exception.
Solution: Before setting to n length of the imaginary character, check if the source contains it. My proposal:
...
        int n = 0;
        if (source.contains(getImaginaryCharacter()))
        n = getImaginaryCharacter().length();
...
F.S.


public double chiSquare(double[] expected, long[] observed)
    throws IllegalArgumentException {
    if ((expected.length < 2) || (expected.length != observed.length)) {
        throw new IllegalArgumentException(
                "observed, expected array lengths incorrect");
    }
    if (!isPositive(expected) || !isNonNegative(observed)) {
        throw new IllegalArgumentException(
            "observed counts must be non-negative and expected counts must be postive");
    }
    double sumSq = 0.0d;
    double dev = 0.0d;
    for (int i = 0; i < observed.length; i++) {
            dev = ((double) observed[i] - expected[i]);
            sumSq += dev * dev / expected[i];
    }
    return sumSq;
}

public double chiSquare(double[] expected, long[] observed)
    throws IllegalArgumentException {
    if ((expected.length < 2) || (expected.length != observed.length)) {
        throw new IllegalArgumentException(
                "observed, expected array lengths incorrect");
    }
    if (!isPositive(expected) || !isNonNegative(observed)) {
        throw new IllegalArgumentException(
            "observed counts must be non-negative and expected counts must be postive");
    }
    double sumExpected = 0d;
    double sumObserved = 0d;
    for (int i = 0; i < observed.length; i++) {
        sumExpected += expected[i];
        sumObserved += observed[i];
    }
    double ratio = 1.0d;
    boolean rescale = false;
    if (Math.abs(sumExpected - sumObserved) > 10E-6) {
        ratio = sumObserved / sumExpected;
        rescale = true;
    }
    double sumSq = 0.0d;
    double dev = 0.0d;
    for (int i = 0; i < observed.length; i++) {
        if (rescale) {
            dev = ((double) observed[i] - ratio * expected[i]);
            sumSq += dev * dev / (ratio * expected[i]);
        } else {
            dev = ((double) observed[i] - expected[i]);
            sumSq += dev * dev / expected[i];
        }
    }
    return sumSq;
}

chiSquare(double[] expected, long[] observed) is returning incorrect test statistic

ChiSquareTestImpl is returning incorrect chi-squared value. An implicit assumption of public double chiSquare(double[] expected, long[] observed) is that the sum of expected and observed are equal. That is, in the code:
for (int i = 0; i < observed.length; i++)
{
            dev = ((double) observed[i] - expected[i]);
            sumSq += dev * dev / expected[i];
        }
this calculation is only correct if sum(observed)==sum(expected). When they are not equal then one must rescale the expected value by sum(observed) / sum(expected) so that they are.
Ironically, it is an example in the unit test ChiSquareTestTest that highlights the error:
long[] observed1 =
{ 500, 623, 72, 70, 31 }
;
        double[] expected1 =
{ 485, 541, 82, 61, 37 }
;
        assertEquals( "chi-square test statistic", 16.4131070362, testStatistic.chiSquare(expected1, observed1), 1E-10);
        assertEquals("chi-square p-value", 0.002512096, testStatistic.chiSquareTest(expected1, observed1), 1E-9);
16.413 is not correct because the expected values do not make sense, they should be: 521.19403 581.37313  88.11940  65.55224  39.76119 so that the sum of expected equals 1296 which is the sum of observed.
Here is some R code (r-project.org) which proves it:
> o1
[1] 500 623  72  70  31
> e1
[1] 485 541  82  61  37
> chisq.test(o1,p=e1,rescale.p=TRUE)
        Chi-squared test for given probabilities
data:  o1
X-squared = 9.0233, df = 4, p-value = 0.06052
> chisq.test(o1,p=e1,rescale.p=TRUE)$observed
[1] 500 623  72  70  31
> chisq.test(o1,p=e1,rescale.p=TRUE)$expected
[1] 521.19403 581.37313  88.11940  65.55224  39.76119


public double cumulativeProbability(double x) throws MathException {
        return 0.5 * (1.0 + Erf.erf((x - mean) /
                (standardDeviation * Math.sqrt(2.0))));
}

public double cumulativeProbability(double x) throws MathException {
    try {
        return 0.5 * (1.0 + Erf.erf((x - mean) /
                (standardDeviation * Math.sqrt(2.0))));
    } catch (MaxIterationsExceededException ex) {
        if (x < (mean - 20 * standardDeviation)) { // JDK 1.5 blows at 38
            return 0.0d;
        } else if (x > (mean + 20 * standardDeviation)) {
            return 1.0d;
        } else {
            throw ex;
        }
    }
}

NormalDistributionImpl::cumulativeProbability(double x) throws ConvergenceException
if x deviates too much from the mean. For example, when x=+/-100, mean=0, sd=1.
Of course the value of the CDF is hard to evaluate in these cases,
but effectively it should be either zero or one.


public double density(final double[] vals) throws DimensionMismatchException {
    final int dim = getDimension();
    if (vals.length != dim) {
        throw new DimensionMismatchException(vals.length, dim);
    }

    return FastMath.pow(2 * FastMath.PI, -dim / 2) *
        FastMath.pow(covarianceMatrixDeterminant, -0.5) *
        getExponentTerm(vals);
}

public double density(final double[] vals) throws DimensionMismatchException {
    final int dim = getDimension();
    if (vals.length != dim) {
        throw new DimensionMismatchException(vals.length, dim);
    }

    return FastMath.pow(2 * FastMath.PI, -0.5 * dim) *
        FastMath.pow(covarianceMatrixDeterminant, -0.5) *
        getExponentTerm(vals);
}

MultivariateNormalDistribution.density(double[]) returns wrong value when the dimension is odd

To reproduce:

Assert.assertEquals(0.398942280401433, new MultivariateNormalDistribution(new double[]{0}, new double[][]{{1}}).density(new double[]{0}), 1e-15);


private RealMatrix squareRoot(RealMatrix m) {
        final EigenDecomposition dec = new EigenDecomposition(m);
        return dec.getSquareRoot();
}

private RealMatrix squareRoot(RealMatrix m) {
    if (m instanceof DiagonalMatrix) {
        final int dim = m.getRowDimension();
        final RealMatrix sqrtM = new DiagonalMatrix(dim);
        for (int i = 0; i < dim; i++) {
           sqrtM.setEntry(i, i, FastMath.sqrt(m.getEntry(i, i)));
        }
        return sqrtM;
    } else {
        final EigenDecomposition dec = new EigenDecomposition(m);
        return dec.getSquareRoot();
    }
}

new multivariate vector optimizers cannot be used with large number of weights

When using the Weigth class to pass a large number of weights to multivariate vector optimizers, an nxn full matrix is created (and copied) when a n elements vector is used. This exhausts memory when n is large.
This happens for example when using curve fitters (even simple curve fitters like polynomial ones for low degree) with large number of points. I encountered this with curve fitting on 41200 points, which created a matrix with 1.7 billion elements.


public Dfp multiply(final int x) {
        return multiplyFast(x);
}

public Dfp multiply(final int x) {
    if (x >= 0 && x < RADIX) {
        return multiplyFast(x);
    } else {
        return multiply(newInstance(x));
    }
}

Dfp Dfp.multiply(int x) does not comply with the general contract FieldElement.multiply(int n)

In class org.apache.commons.math3.Dfp,  the method multiply(int n) is limited to 0 <= n <= 9999. This is not consistent with the general contract of FieldElement.multiply(int n), where there should be no limitation on the values of n.


private void checkParameters() {
    final double[] init = getStartPoint();
    final double[] lB = getLowerBound();
    final double[] uB = getUpperBound();

    // Checks whether there is at least one finite bound value.
    boolean hasFiniteBounds = false;
    for (int i = 0; i < lB.length; i++) {
        if (!Double.isInfinite(lB[i]) ||
            !Double.isInfinite(uB[i])) {
            hasFiniteBounds = true;
            break;
        }
    }
    // Checks whether there is at least one infinite bound value.
    boolean hasInfiniteBounds = false;
    if (hasFiniteBounds) {
        for (int i = 0; i < lB.length; i++) {
            if (Double.isInfinite(lB[i]) ||
                Double.isInfinite(uB[i])) {
                hasInfiniteBounds = true;
                break;
            }
        }

        if (hasInfiniteBounds) {
            // If there is at least one finite bound, none can be infinite,
            // because mixed cases are not supported by the current code.
            throw new MathUnsupportedOperationException();
        } else {
            // Convert API to internal handling of boundaries.
            boundaries = new double[2][];
            boundaries[0] = lB;
            boundaries[1] = uB;

            // Abort early if the normalization will overflow (cf. "encode" method).
        }
    } else {
        // Convert API to internal handling of boundaries.
        boundaries = null;
    }

    if (inputSigma != null) {
        if (inputSigma.length != init.length) {
            throw new DimensionMismatchException(inputSigma.length, init.length);
        }
        for (int i = 0; i < init.length; i++) {
            if (inputSigma[i] < 0) {
                throw new NotPositiveException(inputSigma[i]);
            }
            if (boundaries != null) {
                if (inputSigma[i] > boundaries[1][i] - boundaries[0][i]) {
                    throw new OutOfRangeException(inputSigma[i], 0, boundaries[1][i] - boundaries[0][i]);
                }
            }
        }
    }
}

private void checkParameters() {
    final double[] init = getStartPoint();
    final double[] lB = getLowerBound();
    final double[] uB = getUpperBound();

    // Checks whether there is at least one finite bound value.
    boolean hasFiniteBounds = false;
    for (int i = 0; i < lB.length; i++) {
        if (!Double.isInfinite(lB[i]) ||
            !Double.isInfinite(uB[i])) {
            hasFiniteBounds = true;
            break;
        }
    }
    // Checks whether there is at least one infinite bound value.
    boolean hasInfiniteBounds = false;
    if (hasFiniteBounds) {
        for (int i = 0; i < lB.length; i++) {
            if (Double.isInfinite(lB[i]) ||
                Double.isInfinite(uB[i])) {
                hasInfiniteBounds = true;
                break;
            }
        }

        if (hasInfiniteBounds) {
            // If there is at least one finite bound, none can be infinite,
            // because mixed cases are not supported by the current code.
            throw new MathUnsupportedOperationException();
        } else {
            // Convert API to internal handling of boundaries.
            boundaries = new double[2][];
            boundaries[0] = lB;
            boundaries[1] = uB;

            // Abort early if the normalization will overflow (cf. "encode" method).
            for (int i = 0; i < lB.length; i++) {
                if (Double.isInfinite(boundaries[1][i] - boundaries[0][i])) {
                    final double max = Double.MAX_VALUE + boundaries[0][i];
                    final NumberIsTooLargeException e
                        = new NumberIsTooLargeException(boundaries[1][i],
                                                        max,
                                                        true);
                    e.getContext().addMessage(LocalizedFormats.OVERFLOW);
                    e.getContext().addMessage(LocalizedFormats.INDEX, i);

                    throw e;
                }
            }
        }
    } else {
        // Convert API to internal handling of boundaries.
        boundaries = null;
    }

    if (inputSigma != null) {
        if (inputSigma.length != init.length) {
            throw new DimensionMismatchException(inputSigma.length, init.length);
        }
        for (int i = 0; i < init.length; i++) {
            if (inputSigma[i] < 0) {
                throw new NotPositiveException(inputSigma[i]);
            }
            if (boundaries != null) {
                if (inputSigma[i] > boundaries[1][i] - boundaries[0][i]) {
                    throw new OutOfRangeException(inputSigma[i], 0, boundaries[1][i] - boundaries[0][i]);
                }
            }
        }
    }
}

Wide bounds to CMAESOptimizer result in NaN parameters passed to fitness function

If you give large values as lower/upper bounds (for example -Double.MAX_VALUE as a lower bound), the optimizer can call the fitness function with parameters set to NaN.  My guess is this is due to FitnessFunction.encode/decode generating NaN when normalizing/denormalizing parameters.  For example, if the difference between the lower and upper bound is greater than Double.MAX_VALUE, encode could divide infinity by infinity.


public double getNumericalMean() {
    return (double) (getSampleSize() * getNumberOfSuccesses()) / (double) getPopulationSize();
}

public double getNumericalMean() {
    return getSampleSize() * (getNumberOfSuccesses() / (double) getPopulationSize());
}

HypergeometricDistribution.sample suffers from integer overflow

Hi, I have an application which broke when ported from commons math 2.2 to 3.2. It looks like the HypergeometricDistribution.sample() method doesn't work as well as it used to with large integer values – the example code below should return a sample between 0 and 50, but usually returns -50.

import org.apache.commons.math3.distribution.HypergeometricDistribution;

public class Foo {
  public static void main(String[] args) {
    HypergeometricDistribution a = new HypergeometricDistribution(
        43130568, 42976365, 50);
    System.out.printf("%d %d%n", a.getSupportLowerBound(), a.getSupportUpperBound()); // Prints "0 50"
    System.out.printf("%d%n",a.sample());                                             // Prints "-50"
  }
}

In the debugger, I traced it as far as an integer overflow in HypergeometricDistribution.getNumericalMean() – instead of doing

return (double) (getSampleSize() * getNumberOfSuccesses()) / (double) getPopulationSize();

it could do:

return getSampleSize() * ((double) getNumberOfSuccesses() / (double) getPopulationSize());

This seemed to fix it, based on a quick test.


public double[] repairAndDecode(final double[] x) {
    return
        decode(x);
}

public double[] repairAndDecode(final double[] x) {
    return boundaries != null && isRepairMode ?
        decode(repair(x)) :
        decode(x);
}

The CMAESOptimizer can exceed the bounds passed to optimize.  Looking at the generationLoop in doOptimize(), it does a bounds check by calling isFeasible() but if checkFeasableCount is zero (the default) then isFeasible() is never even called.  Also, even with non-zero checkFeasableCount it may give up before finding an in-bounds offspring and go forward with an out-of-bounds offspring.  This is against svn revision 1387637.  I can provide an example program where the optimizer ends up with a fit outside the prescribed bounds if that would help.


private void guessAOmega() {
    // initialize the sums for the linear model between the two integrals
    double sx2 = 0;
    double sy2 = 0;
    double sxy = 0;
    double sxz = 0;
    double syz = 0;

    double currentX = observations[0].getX();
    double currentY = observations[0].getY();
    double f2Integral = 0;
    double fPrime2Integral = 0;
    final double startX = currentX;
    for (int i = 1; i < observations.length; ++i) {
        // one step forward
        final double previousX = currentX;
        final double previousY = currentY;
        currentX = observations[i].getX();
        currentY = observations[i].getY();

        // update the integrals of f2 and f'2
        // considering a linear model for f (and therefore constant f')
        final double dx = currentX - previousX;
        final double dy = currentY - previousY;
        final double f2StepIntegral =
            dx * (previousY * previousY + previousY * currentY + currentY * currentY) / 3;
        final double fPrime2StepIntegral = dy * dy / dx;

        final double x = currentX - startX;
        f2Integral += f2StepIntegral;
        fPrime2Integral += fPrime2StepIntegral;

        sx2 += x * x;
        sy2 += f2Integral * f2Integral;
        sxy += x * f2Integral;
        sxz += x * fPrime2Integral;
        syz += f2Integral * fPrime2Integral;
    }

    // compute the amplitude and pulsation coefficients
    double c1 = sy2 * sxz - sxy * syz;
    double c2 = sxy * sxz - sx2 * syz;
    double c3 = sx2 * sy2 - sxy * sxy;
    if ((c1 / c2 < 0) || (c2 / c3 < 0)) {
        final int last = observations.length - 1;
        // Range of the observations, assuming that the
        // observations are sorted.
        final double xRange = observations[last].getX() - observations[0].getX();
        if (xRange == 0) {
            throw new ZeroException();
        }
        omega = 2 * Math.PI / xRange;

        double yMin = Double.POSITIVE_INFINITY;
        double yMax = Double.NEGATIVE_INFINITY;
        for (int i = 1; i < observations.length; ++i) {
            final double y = observations[i].getY();
            if (y < yMin) {
                yMin = y;
            }
            if (y > yMax) {
                yMax = y;
            }
        }
        a = 0.5 * (yMax - yMin);
    } else {
            // In some ill-conditioned cases (cf. MATH-844), the guesser
            // procedure cannot produce sensible results.

        a = FastMath.sqrt(c1 / c2);
        omega = FastMath.sqrt(c2 / c3);
    }
}

private void guessAOmega() {
    // initialize the sums for the linear model between the two integrals
    double sx2 = 0;
    double sy2 = 0;
    double sxy = 0;
    double sxz = 0;
    double syz = 0;

    double currentX = observations[0].getX();
    double currentY = observations[0].getY();
    double f2Integral = 0;
    double fPrime2Integral = 0;
    final double startX = currentX;
    for (int i = 1; i < observations.length; ++i) {
        // one step forward
        final double previousX = currentX;
        final double previousY = currentY;
        currentX = observations[i].getX();
        currentY = observations[i].getY();

        // update the integrals of f2 and f'2
        // considering a linear model for f (and therefore constant f')
        final double dx = currentX - previousX;
        final double dy = currentY - previousY;
        final double f2StepIntegral =
            dx * (previousY * previousY + previousY * currentY + currentY * currentY) / 3;
        final double fPrime2StepIntegral = dy * dy / dx;

        final double x = currentX - startX;
        f2Integral += f2StepIntegral;
        fPrime2Integral += fPrime2StepIntegral;

        sx2 += x * x;
        sy2 += f2Integral * f2Integral;
        sxy += x * f2Integral;
        sxz += x * fPrime2Integral;
        syz += f2Integral * fPrime2Integral;
    }

    // compute the amplitude and pulsation coefficients
    double c1 = sy2 * sxz - sxy * syz;
    double c2 = sxy * sxz - sx2 * syz;
    double c3 = sx2 * sy2 - sxy * sxy;
    if ((c1 / c2 < 0) || (c2 / c3 < 0)) {
        final int last = observations.length - 1;
        // Range of the observations, assuming that the
        // observations are sorted.
        final double xRange = observations[last].getX() - observations[0].getX();
        if (xRange == 0) {
            throw new ZeroException();
        }
        omega = 2 * Math.PI / xRange;

        double yMin = Double.POSITIVE_INFINITY;
        double yMax = Double.NEGATIVE_INFINITY;
        for (int i = 1; i < observations.length; ++i) {
            final double y = observations[i].getY();
            if (y < yMin) {
                yMin = y;
            }
            if (y > yMax) {
                yMax = y;
            }
        }
        a = 0.5 * (yMax - yMin);
    } else {
        if (c2 == 0) {
            // In some ill-conditioned cases (cf. MATH-844), the guesser
            // procedure cannot produce sensible results.
            throw new MathIllegalStateException(LocalizedFormats.ZERO_DENOMINATOR);
        }

        a = FastMath.sqrt(c1 / c2);
        omega = FastMath.sqrt(c2 / c3);
    }
}

"HarmonicFitter.ParameterGuesser" sometimes fails to return sensible values

The inner class "ParameterGuesser" in "HarmonicFitter" (package "o.a.c.m.optimization.fitting") fails to compute a usable guess for the "amplitude" parameter.


public double percentageValue() {
    return multiply(100).doubleValue();
}

public double percentageValue() {
    return 100 * doubleValue();
}

The percentageValue() method of the Fraction class works by first multiplying the Fraction by 100, then converting the Fraction to a double. This causes overflows when the numerator is greater than Integer.MAX_VALUE/100, even when the value of the fraction is far below this value.
The patch changes the method to first convert to a double value, and then multiply this value by 100 - the result should be the same, but with less overflows. An addition to the test for the method that covers this bug is also included.


public static double linearCombination(final double[] a, final double[] b)
    throws DimensionMismatchException {
    final int len = a.length;
    if (len != b.length) {
        throw new DimensionMismatchException(len, b.length);
    }

        // Revert to scalar multiplication.

    final double[] prodHigh = new double[len];
    double prodLowSum = 0;

    for (int i = 0; i < len; i++) {
        final double ai = a[i];
        final double ca = SPLIT_FACTOR * ai;
        final double aHigh = ca - (ca - ai);
        final double aLow = ai - aHigh;

        final double bi = b[i];
        final double cb = SPLIT_FACTOR * bi;
        final double bHigh = cb - (cb - bi);
        final double bLow = bi - bHigh;
        prodHigh[i] = ai * bi;
        final double prodLow = aLow * bLow - (((prodHigh[i] -
                                                aHigh * bHigh) -
                                               aLow * bHigh) -
                                              aHigh * bLow);
        prodLowSum += prodLow;
    }

    final double prodHighCur = prodHigh[0];
    double prodHighNext = prodHigh[1];
    double sHighPrev = prodHighCur + prodHighNext;
    double sPrime = sHighPrev - prodHighNext;
    double sLowSum = (prodHighNext - (sHighPrev - sPrime)) + (prodHighCur - sPrime);

    final int lenMinusOne = len - 1;
    for (int i = 1; i < lenMinusOne; i++) {
        prodHighNext = prodHigh[i + 1];
        final double sHighCur = sHighPrev + prodHighNext;
        sPrime = sHighCur - prodHighNext;
        sLowSum += (prodHighNext - (sHighCur - sPrime)) + (sHighPrev - sPrime);
        sHighPrev = sHighCur;
    }

    double result = sHighPrev + (prodLowSum + sLowSum);

    if (Double.isNaN(result)) {
        // either we have split infinite numbers or some coefficients were NaNs,
        // just rely on the naive implementation and let IEEE754 handle this
        result = 0;
        for (int i = 0; i < len; ++i) {
            result += a[i] * b[i];
        }
    }

    return result;
}

public static double linearCombination(final double[] a, final double[] b)
    throws DimensionMismatchException {
    final int len = a.length;
    if (len != b.length) {
        throw new DimensionMismatchException(len, b.length);
    }

    if (len == 1) {
        // Revert to scalar multiplication.
        return a[0] * b[0];
    }

    final double[] prodHigh = new double[len];
    double prodLowSum = 0;

    for (int i = 0; i < len; i++) {
        final double ai = a[i];
        final double ca = SPLIT_FACTOR * ai;
        final double aHigh = ca - (ca - ai);
        final double aLow = ai - aHigh;

        final double bi = b[i];
        final double cb = SPLIT_FACTOR * bi;
        final double bHigh = cb - (cb - bi);
        final double bLow = bi - bHigh;
        prodHigh[i] = ai * bi;
        final double prodLow = aLow * bLow - (((prodHigh[i] -
                                                aHigh * bHigh) -
                                               aLow * bHigh) -
                                              aHigh * bLow);
        prodLowSum += prodLow;
    }

    final double prodHighCur = prodHigh[0];
    double prodHighNext = prodHigh[1];
    double sHighPrev = prodHighCur + prodHighNext;
    double sPrime = sHighPrev - prodHighNext;
    double sLowSum = (prodHighNext - (sHighPrev - sPrime)) + (prodHighCur - sPrime);

    final int lenMinusOne = len - 1;
    for (int i = 1; i < lenMinusOne; i++) {
        prodHighNext = prodHigh[i + 1];
        final double sHighCur = sHighPrev + prodHighNext;
        sPrime = sHighCur - prodHighNext;
        sLowSum += (prodHighNext - (sHighCur - sPrime)) + (sHighPrev - sPrime);
        sHighPrev = sHighCur;
    }

    double result = sHighPrev + (prodLowSum + sLowSum);

    if (Double.isNaN(result)) {
        // either we have split infinite numbers or some coefficients were NaNs,
        // just rely on the naive implementation and let IEEE754 handle this
        result = 0;
        for (int i = 0; i < len; ++i) {
            result += a[i] * b[i];
        }
    }

    return result;
}

When MathArrays.linearCombination is passed arguments with length 1, it throws an ArrayOutOfBoundsException. This is caused by this line:
double prodHighNext = prodHigh[1];
linearCombination should check the length of the arguments and fall back to simple multiplication if length == 1.


private double calculateAsymptoticPValue(final double Umin,
                                         final int n1,
                                         final int n2)
    throws ConvergenceException, MaxCountExceededException {

    final int n1n2prod = n1 * n2;

    // http://en.wikipedia.org/wiki/Mann%E2%80%93Whitney_U#Normal_approximation
    final double EU = n1n2prod / 2.0;
    final double VarU = n1n2prod * (n1 + n2 + 1) / 12.0;

    final double z = (Umin - EU) / FastMath.sqrt(VarU);

    final NormalDistribution standardNormal = new NormalDistribution(0, 1);

    return 2 * standardNormal.cumulativeProbability(z);
}

private double calculateAsymptoticPValue(final double Umin,
                                         final int n1,
                                         final int n2)
    throws ConvergenceException, MaxCountExceededException {

    final double n1n2prod = n1 * n2;

    // http://en.wikipedia.org/wiki/Mann%E2%80%93Whitney_U#Normal_approximation
    final double EU = n1n2prod / 2.0;
    final double VarU = n1n2prod * (n1 + n2 + 1) / 12.0;

    final double z = (Umin - EU) / FastMath.sqrt(VarU);

    final NormalDistribution standardNormal = new NormalDistribution(0, 1);

    return 2 * standardNormal.cumulativeProbability(z);
}

Mann-Whitney U Test Suffers From Integer Overflow With Large Data Sets

When performing a Mann-Whitney U Test on large data sets (the attached test uses two 1500 element sets), intermediate integer values used in calculateAsymptoticPValue can overflow, leading to invalid results, such as p-values of NaN, or incorrect calculations.
Attached is a patch, including a test, and a fix, which modifies the affected code to use doubles


protected void computeGeometricalProperties() {

    final Vector2D[][] v = getVertices();

    if (v.length == 0) {
        final BSPTree<Euclidean2D> tree = getTree(false);
        if ((Boolean) tree.getAttribute()) {
            // the instance covers the whole space
            setSize(Double.POSITIVE_INFINITY);
            setBarycenter(Vector2D.NaN);
        } else {
            setSize(0);
            setBarycenter(new Vector2D(0, 0));
        }
    } else if (v[0][0] == null) {
        // there is at least one open-loop: the polygon is infinite
        setSize(Double.POSITIVE_INFINITY);
        setBarycenter(Vector2D.NaN);
    } else {
        // all loops are closed, we compute some integrals around the shape

        double sum  = 0;
        double sumX = 0;
        double sumY = 0;

        for (Vector2D[] loop : v) {
            double x1 = loop[loop.length - 1].getX();
            double y1 = loop[loop.length - 1].getY();
            for (final Vector2D point : loop) {
                final double x0 = x1;
                final double y0 = y1;
                x1 = point.getX();
                y1 = point.getY();
                final double factor = x0 * y1 - y0 * x1;
                sum  += factor;
                sumX += factor * (x0 + x1);
                sumY += factor * (y0 + y1);
            }
        }

        if (sum < 0) {
            // the polygon as a finite outside surrounded by an infinite inside
            setSize(Double.POSITIVE_INFINITY);
            setBarycenter(Vector2D.NaN);
        } else {
            setSize(sum / 2);
            setBarycenter(new Vector2D(sumX / (3 * sum), sumY / (3 * sum)));
        }

    }

}

protected void computeGeometricalProperties() {

    final Vector2D[][] v = getVertices();

    if (v.length == 0) {
        final BSPTree<Euclidean2D> tree = getTree(false);
        if (tree.getCut() == null && (Boolean) tree.getAttribute()) {
            // the instance covers the whole space
            setSize(Double.POSITIVE_INFINITY);
            setBarycenter(Vector2D.NaN);
        } else {
            setSize(0);
            setBarycenter(new Vector2D(0, 0));
        }
    } else if (v[0][0] == null) {
        // there is at least one open-loop: the polygon is infinite
        setSize(Double.POSITIVE_INFINITY);
        setBarycenter(Vector2D.NaN);
    } else {
        // all loops are closed, we compute some integrals around the shape

        double sum  = 0;
        double sumX = 0;
        double sumY = 0;

        for (Vector2D[] loop : v) {
            double x1 = loop[loop.length - 1].getX();
            double y1 = loop[loop.length - 1].getY();
            for (final Vector2D point : loop) {
                final double x0 = x1;
                final double y0 = y1;
                x1 = point.getX();
                y1 = point.getY();
                final double factor = x0 * y1 - y0 * x1;
                sum  += factor;
                sumX += factor * (x0 + x1);
                sumY += factor * (y0 + y1);
            }
        }

        if (sum < 0) {
            // the polygon as a finite outside surrounded by an infinite inside
            setSize(Double.POSITIVE_INFINITY);
            setBarycenter(Vector2D.NaN);
        } else {
            setSize(sum / 2);
            setBarycenter(new Vector2D(sumX / (3 * sum), sumY / (3 * sum)));
        }

    }

}

BSPTree class and recovery of a Euclidean 3D BRep

New to the work here. Thanks for your efforts on this code.
I create a BSPTree from a BoundaryRep (Brep) my test Brep is a cube as represented by a float array containing 8 3D points in(x,y,z) order and an array of indices (12 triplets for the 12 faces of the cube). I construct a BSPMesh() as shown in the code below. I can construct the PolyhedronsSet() but have problems extracting the faces from the BSPTree to reconstruct the BRep. The attached code (BSPMesh2.java) shows that a small change to 1 of the vertex positions causes/corrects the problem.
Any ideas?


protected void dropPhase1Objective() {
    if (getNumObjectiveFunctions() == 1) {
        return;
    }

    List<Integer> columnsToDrop = new ArrayList<Integer>();
    columnsToDrop.add(0);

    // positive cost non-artificial variables
    for (int i = getNumObjectiveFunctions(); i < getArtificialVariableOffset(); i++) {
        final double entry = tableau.getEntry(0, i);
        if (Precision.compareTo(entry, 0d, maxUlps) > 0) {
            columnsToDrop.add(i);
        }
    }

    // non-basic artificial variables
    for (int i = 0; i < getNumArtificialVariables(); i++) {
      int col = i + getArtificialVariableOffset();
      if (getBasicRow(col) == null) {
        columnsToDrop.add(col);
      }
    }

    double[][] matrix = new double[getHeight() - 1][getWidth() - columnsToDrop.size()];
    for (int i = 1; i < getHeight(); i++) {
      int col = 0;
      for (int j = 0; j < getWidth(); j++) {
        if (!columnsToDrop.contains(j)) {
          matrix[i - 1][col++] = tableau.getEntry(i, j);
        }
      }
    }

    for (int i = columnsToDrop.size() - 1; i >= 0; i--) {
      columnLabels.remove((int) columnsToDrop.get(i));
    }

    this.tableau = new Array2DRowRealMatrix(matrix);
    this.numArtificialVariables = 0;
}

protected void dropPhase1Objective() {
    if (getNumObjectiveFunctions() == 1) {
        return;
    }

    List<Integer> columnsToDrop = new ArrayList<Integer>();
    columnsToDrop.add(0);

    // positive cost non-artificial variables
    for (int i = getNumObjectiveFunctions(); i < getArtificialVariableOffset(); i++) {
        final double entry = tableau.getEntry(0, i);
        if (Precision.compareTo(entry, 0d, epsilon) > 0) {
            columnsToDrop.add(i);
        }
    }

    // non-basic artificial variables
    for (int i = 0; i < getNumArtificialVariables(); i++) {
      int col = i + getArtificialVariableOffset();
      if (getBasicRow(col) == null) {
        columnsToDrop.add(col);
      }
    }

    double[][] matrix = new double[getHeight() - 1][getWidth() - columnsToDrop.size()];
    for (int i = 1; i < getHeight(); i++) {
      int col = 0;
      for (int j = 0; j < getWidth(); j++) {
        if (!columnsToDrop.contains(j)) {
          matrix[i - 1][col++] = tableau.getEntry(i, j);
        }
      }
    }

    for (int i = columnsToDrop.size() - 1; i >= 0; i--) {
      columnLabels.remove((int) columnsToDrop.get(i));
    }

    this.tableau = new Array2DRowRealMatrix(matrix);
    this.numArtificialVariables = 0;
}

Methode SimplexSolver.optimeze(...) gives bad results with commons-math3-3.0
in a simple test problem. It works well in commons-math-2.2.


public Iterator<Chromosome> iterator() {
    return chromosomes.iterator();
}

public Iterator<Chromosome> iterator() {
    return getChromosomes().iterator();
}

ListPopulation Iterator allows you to remove chromosomes from the population.

Calling the iterator method of ListPopulation returns an iterator of the protected modifiable list. Before returning the iterator we should wrap it in an unmodifiable list.