private void replaceAssignmentExpression(Var v, Reference ref,
                                         Map<String, String> varmap) {
  // Compute all of the assignments necessary
  List<Node> nodes = Lists.newArrayList();
  Node val = ref.getAssignedValue();
  blacklistVarReferencesInTree(val, v.scope);
  Preconditions.checkState(val.getType() == Token.OBJECTLIT);
  Set<String> all = Sets.newLinkedHashSet(varmap.keySet());
  for (Node key = val.getFirstChild(); key != null;
       key = key.getNext()) {
    String var = key.getString();
    Node value = key.removeFirstChild();
    // TODO(user): Copy type information.
    nodes.add(
      new Node(Token.ASSIGN,
               Node.newString(Token.NAME, varmap.get(var)), value));
    all.remove(var);
  }

  // TODO(user): Better source information.
  for (String var : all) {
    nodes.add(
      new Node(Token.ASSIGN,
               Node.newString(Token.NAME, varmap.get(var)),
               NodeUtil.newUndefinedNode(null)));
  }

  Node replacement;
    // All assignments evaluate to true, so make sure that the
    // expr statement evaluates to true in case it matters.
    nodes.add(new Node(Token.TRUE));

    // Join these using COMMA.  A COMMA node must have 2 children, so we
    // create a tree. In the tree the first child be the COMMA to match
    // the parser, otherwise tree equality tests fail.
    nodes = Lists.reverse(nodes);
    replacement = new Node(Token.COMMA);
    Node cur = replacement;
    int i;
    for (i = 0; i < nodes.size() - 2; i++) {
      cur.addChildToFront(nodes.get(i));
      Node t = new Node(Token.COMMA);
      cur.addChildToFront(t);
      cur = t;
    }
    cur.addChildToFront(nodes.get(i));
    cur.addChildToFront(nodes.get(i + 1));

  Node replace = ref.getParent();
  replacement.copyInformationFromForTree(replace);

  if (replace.getType() == Token.VAR) {
    replace.getParent().replaceChild(
        replace, NodeUtil.newExpr(replacement));
  } else {
    replace.getParent().replaceChild(replace, replacement);
  }
}

private void replaceAssignmentExpression(Var v, Reference ref,
                                         Map<String, String> varmap) {
  // Compute all of the assignments necessary
  List<Node> nodes = Lists.newArrayList();
  Node val = ref.getAssignedValue();
  blacklistVarReferencesInTree(val, v.scope);
  Preconditions.checkState(val.getType() == Token.OBJECTLIT);
  Set<String> all = Sets.newLinkedHashSet(varmap.keySet());
  for (Node key = val.getFirstChild(); key != null;
       key = key.getNext()) {
    String var = key.getString();
    Node value = key.removeFirstChild();
    // TODO(user): Copy type information.
    nodes.add(
      new Node(Token.ASSIGN,
               Node.newString(Token.NAME, varmap.get(var)), value));
    all.remove(var);
  }

  // TODO(user): Better source information.
  for (String var : all) {
    nodes.add(
      new Node(Token.ASSIGN,
               Node.newString(Token.NAME, varmap.get(var)),
               NodeUtil.newUndefinedNode(null)));
  }

  Node replacement;
  if (nodes.isEmpty()) {
    replacement = new Node(Token.TRUE);
  } else {
    // All assignments evaluate to true, so make sure that the
    // expr statement evaluates to true in case it matters.
    nodes.add(new Node(Token.TRUE));

    // Join these using COMMA.  A COMMA node must have 2 children, so we
    // create a tree. In the tree the first child be the COMMA to match
    // the parser, otherwise tree equality tests fail.
    nodes = Lists.reverse(nodes);
    replacement = new Node(Token.COMMA);
    Node cur = replacement;
    int i;
    for (i = 0; i < nodes.size() - 2; i++) {
      cur.addChildToFront(nodes.get(i));
      Node t = new Node(Token.COMMA);
      cur.addChildToFront(t);
      cur = t;
    }
    cur.addChildToFront(nodes.get(i));
    cur.addChildToFront(nodes.get(i + 1));
  }

  Node replace = ref.getParent();
  replacement.copyInformationFromForTree(replace);

  if (replace.getType() == Token.VAR) {
    replace.getParent().replaceChild(
        replace, NodeUtil.newExpr(replacement));
  } else {
    replace.getParent().replaceChild(replace, replacement);
  }
}

compiler-20110811 crashes with index(1) must be less than size(1)

What steps will reproduce the problem?
Run compiler on https://raw.github.com/scottschiller/SoundManager2/master/script/soundmanager2-nodebug.js

You can copy this into the Appspot closure compiler to see the error:
// ==ClosureCompiler==
// @output_file_name default.js
// @compilation_level SIMPLE_OPTIMIZATIONS
// @code_url https://raw.github.com/scottschiller/SoundManager2/master/script/soundmanager2-nodebug.js
// ==/ClosureCompiler==

I've attached a dump of the error from appspot.

(This is the popular SoundManager library for HTML5 audio)

What is the expected output? What do you see instead?
Got crash...

What version of the product are you using? On what operating system?
Latest (compiler-20110811). We were previously using the June build, and had no problems

Please provide any additional information below.


private static String extractClassNameIfGoog(Node node, Node parent,
    String functionName){
  String className = null;
  if (NodeUtil.isExprCall(parent)) {
    Node callee = node.getFirstChild();
    if (callee != null && callee.getType() == Token.GETPROP) {
      String qualifiedName = callee.getQualifiedName();
      if (functionName.equals(qualifiedName)) {
        Node target = callee.getNext();
        if (target != null) {
          className = target.getString();
        }
      }
    }
  }
  return className;
}

private static String extractClassNameIfGoog(Node node, Node parent,
    String functionName){
  String className = null;
  if (NodeUtil.isExprCall(parent)) {
    Node callee = node.getFirstChild();
    if (callee != null && callee.getType() == Token.GETPROP) {
      String qualifiedName = callee.getQualifiedName();
      if (functionName.equals(qualifiedName)) {
        Node target = callee.getNext();
        if (target != null && target.getType() == Token.STRING) {
          className = target.getString();
        }
      }
    }
  }
  return className;
}

compiler crashes when  goog.provide used with non string

What steps will reproduce the problem?
1. insert  goog.provide(some.function);
2. compile.
3.
What is the expected output? What do you see instead?

This should give an error diagnostic. What it gives is:

java.lang.RuntimeException: java.lang.RuntimeException: INTERNAL COMPILER ERROR.
Please email js-compiler@google.com with this stack trace.
GETPROP 17 [originalname: Spike] [source_file: file.js] is not a string node
 Node(CALL): file.js:17:12
goog.provide(mine.Spike);
...
[stack traces...]

I think this is the current build as of the day of this report.


private void computeGenKill(Node n, BitSet gen, BitSet kill,
    boolean conditional) {

  switch (n.getType()) {
    case Token.SCRIPT:
    case Token.BLOCK:
    case Token.FUNCTION:
      return;

    case Token.WHILE:
    case Token.DO:
    case Token.IF:
      computeGenKill(NodeUtil.getConditionExpression(n), gen, kill,
          conditional);
      return;

    case Token.FOR:
      if (!NodeUtil.isForIn(n)) {
        computeGenKill(NodeUtil.getConditionExpression(n), gen, kill,
            conditional);
      } else {
        // for(x in y) {...}
        Node lhs = n.getFirstChild();
        Node rhs = lhs.getNext();
        if (NodeUtil.isVar(lhs)) {
          // for(var x in y) {...}
          lhs = lhs.getLastChild();
        }
          addToSetIfLocal(lhs, kill);
          addToSetIfLocal(lhs, gen);
        computeGenKill(rhs, gen, kill, conditional);
      }
      return;

    case Token.VAR:
      for (Node c = n.getFirstChild(); c != null; c = c.getNext()) {
        if (c.hasChildren()) {
          computeGenKill(c.getFirstChild(), gen, kill, conditional);
          if (!conditional) {
            addToSetIfLocal(c, kill);
          }
        }
      }
      return;

    case Token.AND:
    case Token.OR:
      computeGenKill(n.getFirstChild(), gen, kill, conditional);
      // May short circuit.
      computeGenKill(n.getLastChild(), gen, kill, true);
      return;

    case Token.HOOK:
      computeGenKill(n.getFirstChild(), gen, kill, conditional);
      // Assume both sides are conditional.
      computeGenKill(n.getFirstChild().getNext(), gen, kill, true);
      computeGenKill(n.getLastChild(), gen, kill, true);
      return;

    case Token.NAME:
      if (isArgumentsName(n)) {
        markAllParametersEscaped();
      } else {
        addToSetIfLocal(n, gen);
      }
      return;

    default:
      if (NodeUtil.isAssignmentOp(n) && NodeUtil.isName(n.getFirstChild())) {
        Node lhs = n.getFirstChild();
        if (!conditional) {
          addToSetIfLocal(lhs, kill);
        }
        if (!NodeUtil.isAssign(n)) {
          // assignments such as a += 1 reads a.
          addToSetIfLocal(lhs, gen);
        }
        computeGenKill(lhs.getNext(), gen, kill, conditional);
      } else {
        for (Node c = n.getFirstChild(); c != null; c = c.getNext()) {
          computeGenKill(c, gen, kill, conditional);
        }
      }
      return;
  }
}

private void computeGenKill(Node n, BitSet gen, BitSet kill,
    boolean conditional) {

  switch (n.getType()) {
    case Token.SCRIPT:
    case Token.BLOCK:
    case Token.FUNCTION:
      return;

    case Token.WHILE:
    case Token.DO:
    case Token.IF:
      computeGenKill(NodeUtil.getConditionExpression(n), gen, kill,
          conditional);
      return;

    case Token.FOR:
      if (!NodeUtil.isForIn(n)) {
        computeGenKill(NodeUtil.getConditionExpression(n), gen, kill,
            conditional);
      } else {
        // for(x in y) {...}
        Node lhs = n.getFirstChild();
        Node rhs = lhs.getNext();
        if (NodeUtil.isVar(lhs)) {
          // for(var x in y) {...}
          lhs = lhs.getLastChild();
        }
        if (NodeUtil.isName(lhs)) {
          addToSetIfLocal(lhs, kill);
          addToSetIfLocal(lhs, gen);
        } else {
          computeGenKill(lhs, gen, kill, conditional);
        }
        computeGenKill(rhs, gen, kill, conditional);
      }
      return;

    case Token.VAR:
      for (Node c = n.getFirstChild(); c != null; c = c.getNext()) {
        if (c.hasChildren()) {
          computeGenKill(c.getFirstChild(), gen, kill, conditional);
          if (!conditional) {
            addToSetIfLocal(c, kill);
          }
        }
      }
      return;

    case Token.AND:
    case Token.OR:
      computeGenKill(n.getFirstChild(), gen, kill, conditional);
      // May short circuit.
      computeGenKill(n.getLastChild(), gen, kill, true);
      return;

    case Token.HOOK:
      computeGenKill(n.getFirstChild(), gen, kill, conditional);
      // Assume both sides are conditional.
      computeGenKill(n.getFirstChild().getNext(), gen, kill, true);
      computeGenKill(n.getLastChild(), gen, kill, true);
      return;

    case Token.NAME:
      if (isArgumentsName(n)) {
        markAllParametersEscaped();
      } else {
        addToSetIfLocal(n, gen);
      }
      return;

    default:
      if (NodeUtil.isAssignmentOp(n) && NodeUtil.isName(n.getFirstChild())) {
        Node lhs = n.getFirstChild();
        if (!conditional) {
          addToSetIfLocal(lhs, kill);
        }
        if (!NodeUtil.isAssign(n)) {
          // assignments such as a += 1 reads a.
          addToSetIfLocal(lhs, gen);
        }
        computeGenKill(lhs.getNext(), gen, kill, conditional);
      } else {
        for (Node c = n.getFirstChild(); c != null; c = c.getNext()) {
          computeGenKill(c, gen, kill, conditional);
        }
      }
      return;
  }
}

What steps will reproduce the problem?
1. open http://closure-compiler.appspot.com/
2. input js code:
  function keys(obj) {
    var a = [], i = 0;
    for (a[i++] in obj)
      ;
    return a;
  }
3. press [compile] button.

What is the expected output? What do you see instead?
Except OK. See java error.

What version of the product are you using? On what operating system?
Online CC version.


static boolean functionCallHasSideEffects(
    Node callNode, @Nullable AbstractCompiler compiler) {
  if (callNode.getType() != Token.CALL) {
    throw new IllegalStateException(
        "Expected CALL node, got " + Token.name(callNode.getType()));
  }

  if (callNode.isNoSideEffectsCall()) {
    return false;
  }

  Node nameNode = callNode.getFirstChild();

  // Built-in functions with no side effects.
  if (nameNode.getType() == Token.NAME) {
    String name = nameNode.getString();
    if (BUILTIN_FUNCTIONS_WITHOUT_SIDEEFFECTS.contains(name)) {
      return false;
    }
  } else if (nameNode.getType() == Token.GETPROP) {
    if (callNode.hasOneChild()
        && OBJECT_METHODS_WITHOUT_SIDEEFFECTS.contains(
              nameNode.getLastChild().getString())) {
      return false;
    }

    if (callNode.isOnlyModifiesThisCall()
        && evaluatesToLocalValue(nameNode.getFirstChild())) {
      return false;
    }

    // Functions in the "Math" namespace have no side effects.

    if (compiler != null && !compiler.hasRegExpGlobalReferences()) {
      if (nameNode.getFirstChild().getType() == Token.REGEXP
          && REGEXP_METHODS.contains(nameNode.getLastChild().getString())) {
        return false;
      } else if (nameNode.getFirstChild().getType() == Token.STRING
          && STRING_REGEXP_METHODS.contains(
              nameNode.getLastChild().getString())) {
        Node param = nameNode.getNext();
        if (param != null &&
            (param.getType() == Token.STRING
                || param.getType() == Token.REGEXP))
        return false;
      }
    }
  }

  return true;
}

static boolean functionCallHasSideEffects(
    Node callNode, @Nullable AbstractCompiler compiler) {
  if (callNode.getType() != Token.CALL) {
    throw new IllegalStateException(
        "Expected CALL node, got " + Token.name(callNode.getType()));
  }

  if (callNode.isNoSideEffectsCall()) {
    return false;
  }

  Node nameNode = callNode.getFirstChild();

  // Built-in functions with no side effects.
  if (nameNode.getType() == Token.NAME) {
    String name = nameNode.getString();
    if (BUILTIN_FUNCTIONS_WITHOUT_SIDEEFFECTS.contains(name)) {
      return false;
    }
  } else if (nameNode.getType() == Token.GETPROP) {
    if (callNode.hasOneChild()
        && OBJECT_METHODS_WITHOUT_SIDEEFFECTS.contains(
              nameNode.getLastChild().getString())) {
      return false;
    }

    if (callNode.isOnlyModifiesThisCall()
        && evaluatesToLocalValue(nameNode.getFirstChild())) {
      return false;
    }

    // Functions in the "Math" namespace have no side effects.
    if (nameNode.getFirstChild().getType() == Token.NAME) {
      String namespaceName = nameNode.getFirstChild().getString();
      if (namespaceName.equals("Math")) {
        return false;
      }
    }

    if (compiler != null && !compiler.hasRegExpGlobalReferences()) {
      if (nameNode.getFirstChild().getType() == Token.REGEXP
          && REGEXP_METHODS.contains(nameNode.getLastChild().getString())) {
        return false;
      } else if (nameNode.getFirstChild().getType() == Token.STRING
          && STRING_REGEXP_METHODS.contains(
              nameNode.getLastChild().getString())) {
        Node param = nameNode.getNext();
        if (param != null &&
            (param.getType() == Token.STRING
                || param.getType() == Token.REGEXP))
        return false;
      }
    }
  }

  return true;
}

What steps will reproduce the problem?
1. Try the following code, in Simple mode
Math.blah = function(test) { test.a = 5; };
var test = new Object();
Math.blah(test);
2. The output is
Math.blah=function(a){a.a=5};var test={};

What is the expected output? What do you see instead?
Note that Math.blah(test) was removed. It should not be. It issues a warning: JSC_USELESS_CODE: Suspicious code. This code lacks side-effects. Is there a bug? at line 4 character 9

What version of the product are you using? On what operating system?
Tested on Google hosted Closure service.

Please provide any additional information below.
Closure seems to be protective about Math in particular, and doesn't like people messing around with her? So, when I try the following code:-
var n = {};
n.blah = function(test) { test.a = 5; };
var test = new Object();
n.blah(test);

It works. When I replace n by Math, then again, Closure kicks out blah. I need that poor fellow. Please talk some sense into it.


private String format(JSError error, boolean warning) {
  // extract source excerpt
  SourceExcerptProvider source = getSource();
  String sourceExcerpt = source == null ? null :
      excerpt.get(
          source, error.sourceName, error.lineNumber, excerptFormatter);

  // formatting the message
  StringBuilder b = new StringBuilder();
  if (error.sourceName != null) {
    b.append(error.sourceName);
    if (error.lineNumber > 0) {
      b.append(':');
      b.append(error.lineNumber);
    }
    b.append(": ");
  }

  b.append(getLevelName(warning ? CheckLevel.WARNING : CheckLevel.ERROR));
  b.append(" - ");

  b.append(error.description);
  b.append('\n');
  if (sourceExcerpt != null) {
    b.append(sourceExcerpt);
    b.append('\n');
    int charno = error.getCharno();

    // padding equal to the excerpt and arrow at the end
    // charno == sourceExpert.length() means something is missing
    // at the end of the line
    if (excerpt.equals(LINE)
        && 0 <= charno && charno < sourceExcerpt.length()) {
      for (int i = 0; i < charno; i++) {
        char c = sourceExcerpt.charAt(i);
        if (Character.isWhitespace(c)) {
          b.append(c);
        } else {
          b.append(' ');
        }
      }
      b.append("^\n");
    }
  }
  return b.toString();
}

private String format(JSError error, boolean warning) {
  // extract source excerpt
  SourceExcerptProvider source = getSource();
  String sourceExcerpt = source == null ? null :
      excerpt.get(
          source, error.sourceName, error.lineNumber, excerptFormatter);

  // formatting the message
  StringBuilder b = new StringBuilder();
  if (error.sourceName != null) {
    b.append(error.sourceName);
    if (error.lineNumber > 0) {
      b.append(':');
      b.append(error.lineNumber);
    }
    b.append(": ");
  }

  b.append(getLevelName(warning ? CheckLevel.WARNING : CheckLevel.ERROR));
  b.append(" - ");

  b.append(error.description);
  b.append('\n');
  if (sourceExcerpt != null) {
    b.append(sourceExcerpt);
    b.append('\n');
    int charno = error.getCharno();

    // padding equal to the excerpt and arrow at the end
    // charno == sourceExpert.length() means something is missing
    // at the end of the line
    if (excerpt.equals(LINE)
        && 0 <= charno && charno <= sourceExcerpt.length()) {
      for (int i = 0; i < charno; i++) {
        char c = sourceExcerpt.charAt(i);
        if (Character.isWhitespace(c)) {
          b.append(c);
        } else {
          b.append(' ');
        }
      }
      b.append("^\n");
    }
  }
  return b.toString();
}

Column-indicating caret is sometimes not in error output

For some reason, the caret doesn't always show up in the output when there are errors.

When test.js looks like this:

>alert(1;

, the output is this:

>java -jar compiler.jar --js test.js
test.js:1: ERROR - Parse error. missing ) after argument list

1 error(s), 0 warning(s)

However, when test.js looks like this (notice the line break after the semicolon):

>alert(1;
>

, the output is this:

>java -jar compiler.jar --js test.js
test.js:1: ERROR - Parse error. missing ) after argument list
alert(1;
        ^

1 error(s), 0 warning(s)

That's the simplest reproduction of the problem that I could come up with, but I just encountered the problem in a file with ~100 LOC in it. This is the first time I believe I've run into the problem, but when it happens, my error parser fails and it becomes a pain to track down the raw output to find the actual problem.

Tested against r1171, committed 6/10 08:06. The problem is present going back to at least r1000, so this isn't a new issue.


private boolean isPrototypePropertyAssign(Node assign) {
  Node n = assign.getFirstChild();
  if (n != null && NodeUtil.isVarOrSimpleAssignLhs(n, assign)
      && n.getType() == Token.GETPROP
      ) {
    // We want to exclude the assignment itself from the usage list
    boolean isChainedProperty =
        n.getFirstChild().getType() == Token.GETPROP;

    if (isChainedProperty) {
      Node child = n.getFirstChild().getFirstChild().getNext();

      if (child.getType() == Token.STRING &&
          child.getString().equals("prototype")) {
        return true;
      }
    }
  }

  return false;
}

private boolean isPrototypePropertyAssign(Node assign) {
  Node n = assign.getFirstChild();
  if (n != null && NodeUtil.isVarOrSimpleAssignLhs(n, assign)
      && n.getType() == Token.GETPROP
      && assign.getParent().getType() == Token.EXPR_RESULT) {
    // We want to exclude the assignment itself from the usage list
    boolean isChainedProperty =
        n.getFirstChild().getType() == Token.GETPROP;

    if (isChainedProperty) {
      Node child = n.getFirstChild().getFirstChild().getNext();

      if (child.getType() == Token.STRING &&
          child.getString().equals("prototype")) {
        return true;
      }
    }
  }

  return false;
}

Advanced compilations renames a function and then deletes it, leaving a reference to a renamed but non-existent function

If we provide the below code to advanced:

function A() {
this._x = 1;
}

A.prototype['func1'] = // done to save public reference to func1
A.prototype.func1 = function() {
  this._x = 2;
  this.func2();
}

A.prototype.func2 = function() {
  this._x = 3;
  this.func3();
}

window['A'] = A;

We get the output:

function a() {
  this.a = 1
}
a.prototype.func1 = a.prototype.b = function() {
  this.a = 2;
  this.c() // Problem!
};
window.A = a;

So the compiler emits no errors, and renames 'func2' to 'c' but ends up throwing away the definition of that function!

The problem arises when I use:

A.prototype['func1'] = // done to save public reference to func1
A.prototype.func1 = function() {
...
}

The ['func1'] line is apparently enough to save the reference correctly, but also has the side effect of causing the function innards to do the wrong thing.

I can of course instead write it as:

A.prototype['func1'] = A.prototype.func1;
A.prototype.func1 = function() {
  this._x = 2;
  this.func2();
}

In which case Advanced will compile correctly and the results will also be valid.

function a() {
  this.a = 1
}
a.prototype.func1 = a.prototype.b;
a.prototype.b = function() {
  this.a = 2;
  this.a = 3 // func2, correctly minified
};
window.A = a;

For now I can just use the expected way of declaring that func1 export, but since the compiler returns with no errors or warnings and creates a function with no definition, it seems worth reporting.


private void visitCall(NodeTraversal t, Node n) {
  Node child = n.getFirstChild();
  JSType childType = getJSType(child).restrictByNotNullOrUndefined();

  if (!childType.canBeCalled()) {
    report(t, n, NOT_CALLABLE, childType.toString());
    ensureTyped(t, n);
    return;
  }

  // A couple of types can be called as if they were functions.
  // If it is a function type, then validate parameters.
  if (childType instanceof FunctionType) {
    FunctionType functionType = (FunctionType) childType;

    boolean isExtern = false;
    JSDocInfo functionJSDocInfo = functionType.getJSDocInfo();
    if(functionJSDocInfo != null) {
      String sourceName = functionJSDocInfo.getSourceName();
      CompilerInput functionSource = compiler.getInput(sourceName);
      isExtern = functionSource.isExtern();
    }

    // Non-native constructors should not be called directly
    // unless they specify a return type and are defined
    // in an extern.
    if (functionType.isConstructor() &&
        !functionType.isNativeObjectType() &&
        (functionType.getReturnType().isUnknownType() ||
         functionType.getReturnType().isVoidType() ||
         !isExtern)) {
      report(t, n, CONSTRUCTOR_NOT_CALLABLE, childType.toString());
    }

    // Functions with explcit 'this' types must be called in a GETPROP
    // or GETELEM.

    visitParameterList(t, n, functionType);
    ensureTyped(t, n, functionType.getReturnType());
  } else {
    ensureTyped(t, n);
  }

  // TODO: Add something to check for calls of RegExp objects, which is not
  // supported by IE.  Either say something about the return type or warn
  // about the non-portability of the call or both.
}

private void visitCall(NodeTraversal t, Node n) {
  Node child = n.getFirstChild();
  JSType childType = getJSType(child).restrictByNotNullOrUndefined();

  if (!childType.canBeCalled()) {
    report(t, n, NOT_CALLABLE, childType.toString());
    ensureTyped(t, n);
    return;
  }

  // A couple of types can be called as if they were functions.
  // If it is a function type, then validate parameters.
  if (childType instanceof FunctionType) {
    FunctionType functionType = (FunctionType) childType;

    boolean isExtern = false;
    JSDocInfo functionJSDocInfo = functionType.getJSDocInfo();
    if(functionJSDocInfo != null) {
      String sourceName = functionJSDocInfo.getSourceName();
      CompilerInput functionSource = compiler.getInput(sourceName);
      isExtern = functionSource.isExtern();
    }

    // Non-native constructors should not be called directly
    // unless they specify a return type and are defined
    // in an extern.
    if (functionType.isConstructor() &&
        !functionType.isNativeObjectType() &&
        (functionType.getReturnType().isUnknownType() ||
         functionType.getReturnType().isVoidType() ||
         !isExtern)) {
      report(t, n, CONSTRUCTOR_NOT_CALLABLE, childType.toString());
    }

    // Functions with explcit 'this' types must be called in a GETPROP
    // or GETELEM.
    if (functionType.isOrdinaryFunction() &&
        !functionType.getTypeOfThis().isUnknownType() &&
        !functionType.getTypeOfThis().isNativeObjectType() &&
        !(child.getType() == Token.GETELEM ||
          child.getType() == Token.GETPROP)) {
      report(t, n, EXPECTED_THIS_TYPE, functionType.toString());
    }

    visitParameterList(t, n, functionType);
    ensureTyped(t, n, functionType.getReturnType());
  } else {
    ensureTyped(t, n);
  }

  // TODO: Add something to check for calls of RegExp objects, which is not
  // supported by IE.  Either say something about the return type or warn
  // about the non-portability of the call or both.
}

Compiler should warn/error when instance methods are operated on

What steps will reproduce the problem?
1. Compile and run the following code:
  goog.require('goog.graphics.Path');
  function demo() {
    var path = new goog.graphics.Path();
    var points = [[1,1], [2,2]];
    for (var i = 0; i < points.length; i++) {
      (i == 0 ? path.moveTo : path.lineTo)(points[i][0], points[i][1]);
    }
  }
  goog.exportSymbol('demo', demo);

What is the expected output? What do you see instead?
I expect it to either work or produce a warning.  In this case, the latter since there's an error in the javascript - when calling path.moveTo(x, y), "this" is set correctly to the path element in the moveTo function.  But when the function is operated on, as in (i == 0 ? path.moveTo : path.lineTo)(x, y), it's no longer an instance method invocation, so "this" reverts to the window object.  In this case, an error results because moveTo references a field in Path that is now "undefined".  Better would be to issue a warning/error that an instance method is being converted to a normal function (perhaps only if it references this).

What version of the product are you using? On what operating system?
Unknown (it's built into my build tools) - I presume this issue is present in all builds.  Running on ubuntu.

Please provide any additional information below.


public JSType caseObjectType(ObjectType type) {
  if (value.equals("function")) {
    JSType ctorType = getNativeType(U2U_CONSTRUCTOR_TYPE);
    return resultEqualsValue && ctorType.isSubtype(type) ? ctorType : null;
      // Objects are restricted to "Function", subtypes are left
      // Only filter out subtypes of "function"
  }
  return matchesExpectation("object") ? type : null;
}

public JSType caseObjectType(ObjectType type) {
  if (value.equals("function")) {
    JSType ctorType = getNativeType(U2U_CONSTRUCTOR_TYPE);
    if (resultEqualsValue) {
      // Objects are restricted to "Function", subtypes are left
      return ctorType.getGreatestSubtype(type);
    } else {
      // Only filter out subtypes of "function"
      return type.isSubtype(ctorType) ? null : type;
    }
  }
  return matchesExpectation("object") ? type : null;
}

Bad type inference with goog.isFunction and friends

experimental/johnlenz/typeerror/test.js:16: WARNING - Property length
never defined on Number
      var i = object.length;

This is the reduced test case:

/**
 * @param {*} object Any object.
 * @return {boolean}
 */
test.isMatched = function(object) {
  if (goog.isDef(object)) {
    if (goog.isFunction(object)) {
      // return object();
    } else if (goog.isBoolean(object)) {
      // return object;
    } else if (goog.isString(object)) {
      // return test.isDef(object);
    } else if (goog.isArray(object)) {
      var i = object.length;
    }
  }
  return false;
};


private void declareArguments(Node functionNode) {
  Node astParameters = functionNode.getFirstChild().getNext();
  Node body = astParameters.getNext();
  FunctionType functionType = (FunctionType) functionNode.getJSType();
  if (functionType != null) {
    Node jsDocParameters = functionType.getParametersNode();
    if (jsDocParameters != null) {
      Node jsDocParameter = jsDocParameters.getFirstChild();
      for (Node astParameter : astParameters.children()) {
        if (jsDocParameter != null) {
          defineSlot(astParameter, functionNode,
              jsDocParameter.getJSType(), true);
          jsDocParameter = jsDocParameter.getNext();
        } else {
          defineSlot(astParameter, functionNode, null, true);
        }
      }
    }
  }
} // end declareArguments

private void declareArguments(Node functionNode) {
  Node astParameters = functionNode.getFirstChild().getNext();
  Node body = astParameters.getNext();
  FunctionType functionType = (FunctionType) functionNode.getJSType();
  if (functionType != null) {
    Node jsDocParameters = functionType.getParametersNode();
    if (jsDocParameters != null) {
      Node jsDocParameter = jsDocParameters.getFirstChild();
      for (Node astParameter : astParameters.children()) {
        if (jsDocParameter != null) {
          defineSlot(astParameter, functionNode,
              jsDocParameter.getJSType(), false);
          jsDocParameter = jsDocParameter.getNext();
        } else {
          defineSlot(astParameter, functionNode, null, true);
        }
      }
    }
  }
} // end declareArguments

unexpected typed coverage of less than 100%

What steps will reproduce the problem?
1. Create JavaScript file:
/*global window*/
/*jslint sub: true*/
/**
 * @constructor
 * @param {!Element} element
 */
function Example(element) {
    /**
     * @param {!string} ns
     * @param {!string} name
     * @return {undefined}
     */
    this.appendElement = function appendElement(ns, name) {
        var e = element.ownerDocument.createElementNS(ns, name);
        element.appendChild(e);
    };
}
window["Example"] = Example;
2. compile it:
java -jar compiler.jar --jscomp_error checkTypes --summary_detail_level 3 --js v.js --js_output_file compiled.js
3. observe the outcome:
0 error(s), 0 warning(s), 73.7% typed

What is the expected output? What do you see instead?
This was expected:
0 error(s), 0 warning(s), 100% typed

What version of the product are you using? On what operating system?
Closure Compiler Version: 964, Built on: 2011/04/05 14:31 on GNU/Linux.

Please provide any additional information below.


Node processFunctionNode(FunctionNode functionNode) {
  Name name = functionNode.getFunctionName();
  Boolean isUnnamedFunction = false;
  if (name == null) {
    name = new Name();
    name.setIdentifier("");
    isUnnamedFunction = true;
  }
  Node node = newNode(Token.FUNCTION);
  Node newName = transform(name);
  if (isUnnamedFunction) {
    // Old Rhino tagged the empty name node with the line number of the
    // declaration.
    newName.setLineno(functionNode.getLineno());
    // TODO(bowdidge) Mark line number of paren correctly.
    // Same problem as below - the left paren might not be on the
    // same line as the function keyword.
    int lpColumn = functionNode.getAbsolutePosition() +
        functionNode.getLp();
    newName.setCharno(position2charno(lpColumn));
  }

  node.addChildToBack(newName);
  Node lp = newNode(Token.LP);
  // The left paren's complicated because it's not represented by an
  // AstNode, so there's nothing that has the actual line number that it
  // appeared on.  We know the paren has to appear on the same line as the
  // function name (or else a semicolon will be inserted.)  If there's no
  // function name, assume the paren was on the same line as the function.
  // TODO(bowdidge): Mark line number of paren correctly.
  Name fnName = functionNode.getFunctionName();
  if (fnName != null) {
    lp.setLineno(fnName.getLineno());
  } else {
    lp.setLineno(functionNode.getLineno());
  }
  int lparenCharno = functionNode.getLp() +
      functionNode.getAbsolutePosition();

  lp.setCharno(position2charno(lparenCharno));
  for (AstNode param : functionNode.getParams()) {
    lp.addChildToBack(transform(param));
  }
  node.addChildToBack(lp);

  Node bodyNode = transform(functionNode.getBody());
  parseDirectives(bodyNode);
  node.addChildToBack(bodyNode);
 return node;
}

Node processFunctionNode(FunctionNode functionNode) {
  Name name = functionNode.getFunctionName();
  Boolean isUnnamedFunction = false;
  if (name == null) {
    int functionType = functionNode.getFunctionType();
    if (functionType != FunctionNode.FUNCTION_EXPRESSION) {
      errorReporter.error(
        "unnamed function statement",
        sourceName,
        functionNode.getLineno(), "", 0);
    }
    name = new Name();
    name.setIdentifier("");
    isUnnamedFunction = true;
  }
  Node node = newNode(Token.FUNCTION);
  Node newName = transform(name);
  if (isUnnamedFunction) {
    // Old Rhino tagged the empty name node with the line number of the
    // declaration.
    newName.setLineno(functionNode.getLineno());
    // TODO(bowdidge) Mark line number of paren correctly.
    // Same problem as below - the left paren might not be on the
    // same line as the function keyword.
    int lpColumn = functionNode.getAbsolutePosition() +
        functionNode.getLp();
    newName.setCharno(position2charno(lpColumn));
  }

  node.addChildToBack(newName);
  Node lp = newNode(Token.LP);
  // The left paren's complicated because it's not represented by an
  // AstNode, so there's nothing that has the actual line number that it
  // appeared on.  We know the paren has to appear on the same line as the
  // function name (or else a semicolon will be inserted.)  If there's no
  // function name, assume the paren was on the same line as the function.
  // TODO(bowdidge): Mark line number of paren correctly.
  Name fnName = functionNode.getFunctionName();
  if (fnName != null) {
    lp.setLineno(fnName.getLineno());
  } else {
    lp.setLineno(functionNode.getLineno());
  }
  int lparenCharno = functionNode.getLp() +
      functionNode.getAbsolutePosition();

  lp.setCharno(position2charno(lparenCharno));
  for (AstNode param : functionNode.getParams()) {
    lp.addChildToBack(transform(param));
  }
  node.addChildToBack(lp);

  Node bodyNode = transform(functionNode.getBody());
  parseDirectives(bodyNode);
  node.addChildToBack(bodyNode);
 return node;
}

An unnamed function statement statements should generate a parse error, but it does not, for example:

function () {};

Note: Unnamed function expression are legal:

(function(){});


public final boolean isEmptyType() {
  return isNoType() || isNoObjectType() || isNoResolvedType();
}

public final boolean isEmptyType() {
  return isNoType() || isNoObjectType() || isNoResolvedType() ||
      (registry.getNativeFunctionType(
           JSTypeNative.LEAST_FUNCTION_TYPE) == this);
}

.indexOf fails to produce missing property warning

The following code compiled with VERBOSE warnings or with the missingProperties check enabled fails to produce a warning or error:

var s = new String("hello");
alert(s.toLowerCase.indexOf("l"));

However, other string functions do properly produce the warning:

var s = new String("hello");
alert(s.toLowerCase.substr(0, 1));


public int parseArguments(Parameters params) throws CmdLineException {
  String param = params.getParameter(0);

  if (param == null) {
    setter.addValue(true);
    return 0;
  } else {
    String lowerParam = param.toLowerCase();
    if (TRUES.contains(lowerParam)) {
      setter.addValue(true);
    } else if (FALSES.contains(lowerParam)) {
      setter.addValue(false);
    } else {
      setter.addValue(true);
      return 0;
    }
    return 1;
  }
}

public int parseArguments(Parameters params) throws CmdLineException {
  String param = null;
  try {
    param = params.getParameter(0);
  } catch (CmdLineException e) {}

  if (param == null) {
    setter.addValue(true);
    return 0;
  } else {
    String lowerParam = param.toLowerCase();
    if (TRUES.contains(lowerParam)) {
      setter.addValue(true);
    } else if (FALSES.contains(lowerParam)) {
      setter.addValue(false);
    } else {
      setter.addValue(true);
      return 0;
    }
    return 1;
  }
}

What steps will reproduce the problem?
1. Download sources of latest (r698) command-line version of closure compiler.
2. Build (with ant from command line).
3. Run compiler (java -jar compiler.jar --version).

What is the expected output?
Closure Compiler (http://code.google.com/closure/compiler)
Version: 698
Built on: 2011/01/17 12:16

What do you see instead?
Опция "--version" требует операнд
(Option "--version" requires operand)
and full list of options with description.

What version of the product are you using? On what operating system?
Latest source of command-line compiler from SVN (r698). OS Linux Mint 7, Sun Java 1.6.0_22.

Please provide any additional information below.
When running compiler with
java -jar compiler.jar --version ?
it shows error message, then version info, then full list of options.


static boolean evaluatesToLocalValue(Node value, Predicate<Node> locals) {
  switch (value.getType()) {
    case Token.ASSIGN:
      // A result that is aliased by a non-local name, is the effectively the
      // same as returning a non-local name, but this doesn't matter if the
      // value is immutable.
      return NodeUtil.isImmutableValue(value.getLastChild())
          || (locals.apply(value)
              && evaluatesToLocalValue(value.getLastChild(), locals));
    case Token.COMMA:
      return evaluatesToLocalValue(value.getLastChild(), locals);
    case Token.AND:
    case Token.OR:
      return evaluatesToLocalValue(value.getFirstChild(), locals)
         && evaluatesToLocalValue(value.getLastChild(), locals);
    case Token.HOOK:
      return evaluatesToLocalValue(value.getFirstChild().getNext(), locals)
         && evaluatesToLocalValue(value.getLastChild(), locals);
    case Token.INC:
    case Token.DEC:
      if (value.getBooleanProp(Node.INCRDECR_PROP)) {
        return evaluatesToLocalValue(value.getFirstChild(), locals);
      } else {
        return true;
      }
    case Token.THIS:
      return locals.apply(value);
    case Token.NAME:
      return isImmutableValue(value) || locals.apply(value);
    case Token.GETELEM:
    case Token.GETPROP:
      // There is no information about the locality of object properties.
      return locals.apply(value);
    case Token.CALL:
      return callHasLocalResult(value)
          || isToStringMethodCall(value)
          || locals.apply(value);
    case Token.NEW:
      // TODO(nicksantos): This needs to be changed so that it
      // returns true iff we're sure the value was never aliased from inside
      // the constructor (similar to callHasLocalResult)
      return true;
    case Token.FUNCTION:
    case Token.REGEXP:
    case Token.ARRAYLIT:
    case Token.OBJECTLIT:
      // Literals objects with non-literal children are allowed.
      return true;
    case Token.IN:
      // TODO(johnlenz): should IN operator be included in #isSimpleOperator?
      return true;
    default:
      // Other op force a local value:
      //  x = '' + g (x is now an local string)
      //  x -= g (x is now an local number)
      if (isAssignmentOp(value)
          || isSimpleOperator(value)
          || isImmutableValue(value)) {
        return true;
      }

      throw new IllegalStateException(
          "Unexpected expression node" + value +
          "\n parent:" + value.getParent());
  }
}

static boolean evaluatesToLocalValue(Node value, Predicate<Node> locals) {
  switch (value.getType()) {
    case Token.ASSIGN:
      // A result that is aliased by a non-local name, is the effectively the
      // same as returning a non-local name, but this doesn't matter if the
      // value is immutable.
      return NodeUtil.isImmutableValue(value.getLastChild())
          || (locals.apply(value)
              && evaluatesToLocalValue(value.getLastChild(), locals));
    case Token.COMMA:
      return evaluatesToLocalValue(value.getLastChild(), locals);
    case Token.AND:
    case Token.OR:
      return evaluatesToLocalValue(value.getFirstChild(), locals)
         && evaluatesToLocalValue(value.getLastChild(), locals);
    case Token.HOOK:
      return evaluatesToLocalValue(value.getFirstChild().getNext(), locals)
         && evaluatesToLocalValue(value.getLastChild(), locals);
    case Token.INC:
    case Token.DEC:
      if (value.getBooleanProp(Node.INCRDECR_PROP)) {
        return evaluatesToLocalValue(value.getFirstChild(), locals);
      } else {
        return true;
      }
    case Token.THIS:
      return locals.apply(value);
    case Token.NAME:
      return isImmutableValue(value) || locals.apply(value);
    case Token.GETELEM:
    case Token.GETPROP:
      // There is no information about the locality of object properties.
      return locals.apply(value);
    case Token.CALL:
      return callHasLocalResult(value)
          || isToStringMethodCall(value)
          || locals.apply(value);
    case Token.NEW:
      // TODO(nicksantos): This needs to be changed so that it
      // returns true iff we're sure the value was never aliased from inside
      // the constructor (similar to callHasLocalResult)
      return false;
    case Token.FUNCTION:
    case Token.REGEXP:
    case Token.ARRAYLIT:
    case Token.OBJECTLIT:
      // Literals objects with non-literal children are allowed.
      return true;
    case Token.IN:
      // TODO(johnlenz): should IN operator be included in #isSimpleOperator?
      return true;
    default:
      // Other op force a local value:
      //  x = '' + g (x is now an local string)
      //  x -= g (x is now an local number)
      if (isAssignmentOp(value)
          || isSimpleOperator(value)
          || isImmutableValue(value)) {
        return true;
      }

      throw new IllegalStateException(
          "Unexpected expression node" + value +
          "\n parent:" + value.getParent());
  }
}

side-effects analysis incorrectly removing function calls with side effects

Sample Code:
---
/** @constructor */
function Foo() {
  var self = this;
  window.setTimeout(function() {
    window.location = self.location;
  }, 0);
}

Foo.prototype.setLocation = function(loc) {
  this.location = loc;
};

(new Foo()).setLocation('http://www.google.com/');
---

The setLocation call will get removed in advanced mode.


private boolean isFoldableExpressBlock(Node n) {
  if (n.getType() == Token.BLOCK) {
    if (n.hasOneChild()) {
      Node maybeExpr = n.getFirstChild();
        // IE has a bug where event handlers behave differently when
        // their return value is used vs. when their return value is in
        // an EXPR_RESULT. It's pretty freaking weird. See:
        // http://code.google.com/p/closure-compiler/issues/detail?id=291
        // We try to detect this case, and not fold EXPR_RESULTs
        // into other expressions.

          // We only have to worry about methods with an implicit 'this'
          // param, or this doesn't happen.

        return NodeUtil.isExpressionNode(maybeExpr);
    }
  }

  return false;
}

private boolean isFoldableExpressBlock(Node n) {
  if (n.getType() == Token.BLOCK) {
    if (n.hasOneChild()) {
      Node maybeExpr = n.getFirstChild();
      if (maybeExpr.getType() == Token.EXPR_RESULT) {
        // IE has a bug where event handlers behave differently when
        // their return value is used vs. when their return value is in
        // an EXPR_RESULT. It's pretty freaking weird. See:
        // http://code.google.com/p/closure-compiler/issues/detail?id=291
        // We try to detect this case, and not fold EXPR_RESULTs
        // into other expressions.
        if (maybeExpr.getFirstChild().getType() == Token.CALL) {
          Node calledFn = maybeExpr.getFirstChild().getFirstChild();

          // We only have to worry about methods with an implicit 'this'
          // param, or this doesn't happen.
          if (calledFn.getType() == Token.GETELEM) {
            return false;
          } else if (calledFn.getType() == Token.GETPROP &&
                     calledFn.getLastChild().getString().startsWith("on")) {
            return false;
          }
        }

        return true;
      }
      return false;
    }
  }

  return false;
}

IE8 error: Object doesn't support this action

What steps will reproduce the problem?
1. Use script with fragment like
    if (e.onchange) {
      e.onchange({
        _extendedByPrototype: Prototype.emptyFunction,
        target: e
      });
    }
2. Compile with Compiler (command-line, latest version)
3. Use in IE8

What is the expected output?
Script:
if(b.onchange){b.onchange({_extendedByPrototype:Prototype.emptyFunction,target
:b})}

What do you see instead?
Script:
b.onchange&&b.onchange({_extendedByPrototype:Prototype.emptyFunction,target
:b})
IE8:
Error message "Object doesn't support this action"

What version of the product are you using? On what operating system?
Version: 20100917 (revision 440)
Built on: 2010/09/17 17:55


private VariableLiveness isVariableReadBeforeKill(
    Node n, String variable) {
  if (NodeUtil.isName(n) && variable.equals(n.getString())) {
    if (NodeUtil.isLhs(n, n.getParent())) {
      // The expression to which the assignment is made is evaluated before
      // the RHS is evaluated (normal left to right evaluation) but the KILL
      // occurs after the RHS is evaluated.
      return VariableLiveness.KILL;
    } else {
      return VariableLiveness.READ;
    }
  }

  // Expressions are evaluated left-right, depth first.
  for (Node child = n.getFirstChild();
      child != null; child = child.getNext()) {
    if (!ControlFlowGraph.isEnteringNewCfgNode(child)) { // Not a FUNCTION
      VariableLiveness state = isVariableReadBeforeKill(child, variable);
      if (state != VariableLiveness.MAYBE_LIVE) {
        return state;
      }
    }
  }
  return VariableLiveness.MAYBE_LIVE;
}

private VariableLiveness isVariableReadBeforeKill(
    Node n, String variable) {
  if (NodeUtil.isName(n) && variable.equals(n.getString())) {
    if (NodeUtil.isLhs(n, n.getParent())) {
      Preconditions.checkState(n.getParent().getType() == Token.ASSIGN);
      // The expression to which the assignment is made is evaluated before
      // the RHS is evaluated (normal left to right evaluation) but the KILL
      // occurs after the RHS is evaluated.
      Node rhs = n.getNext();
      VariableLiveness state = isVariableReadBeforeKill(rhs, variable);
      if (state == VariableLiveness.READ) {
        return state;
      }
      return VariableLiveness.KILL;
    } else {
      return VariableLiveness.READ;
    }
  }

  // Expressions are evaluated left-right, depth first.
  for (Node child = n.getFirstChild();
      child != null; child = child.getNext()) {
    if (!ControlFlowGraph.isEnteringNewCfgNode(child)) { // Not a FUNCTION
      VariableLiveness state = isVariableReadBeforeKill(child, variable);
      if (state != VariableLiveness.MAYBE_LIVE) {
        return state;
      }
    }
  }
  return VariableLiveness.MAYBE_LIVE;
}

Incorrect assignment removal from expression in simple mode.

function closureCompilerTest(someNode) {
   var nodeId;
   return ((nodeId=someNode.id) && (nodeId=parseInt(nodeId.substr(1))) && nodeId>0);
}

COMPILES TO:

function closureCompilerTest(b){var a;return b.id&&(a=parseInt(a.substr(1)))&&a>0};

"nodeId=someNode.id" is replaced with simply "b.id" which is incorrect as the value of "nodeId" is used.


public boolean shouldTraverse(NodeTraversal t, Node n, Node parent) {

  if (n.getType() == Token.FUNCTION) {
    // Don't traverse functions that are constructors or have the @this
    // or @override annotation.
    JSDocInfo jsDoc = getFunctionJsDocInfo(n);
    if (jsDoc != null &&
        (jsDoc.isConstructor() ||
         jsDoc.isInterface() ||
         jsDoc.hasThisType() ||
         jsDoc.isOverride())) {
      return false;
    }

    // Don't traverse functions unless they would normally
    // be able to have a @this annotation associated with them. e.g.,
    // var a = function() { }; // or
    // function a() {} // or
    // a.x = function() {}; // or
    // var a = {x: function() {}};
    int pType = parent.getType();
    if (!(pType == Token.BLOCK ||
          pType == Token.SCRIPT ||
          pType == Token.NAME ||
          pType == Token.ASSIGN ||

          // object literal keys
          pType == Token.STRING ||
          pType == Token.NUMBER)) {
      return false;
    }

    // Don't traverse functions that are getting lent to a prototype.
  }

  if (parent != null && parent.getType() == Token.ASSIGN) {
    Node lhs = parent.getFirstChild();
    Node rhs = lhs.getNext();

    if (n == lhs) {
      // Always traverse the left side of the assignment. To handle
      // nested assignments properly (e.g., (a = this).property = c;),
      // assignLhsChild should not be overridden.
      if (assignLhsChild == null) {
        assignLhsChild = lhs;
      }
    } else {
      // Only traverse the right side if it's not an assignment to a prototype
      // property or subproperty.
      if (NodeUtil.isGet(lhs)) {
        if (lhs.getType() == Token.GETPROP &&
            lhs.getLastChild().getString().equals("prototype")) {
          return false;
        }
        Node llhs = lhs.getFirstChild();
        if (llhs.getType() == Token.GETPROP &&
            llhs.getLastChild().getString().equals("prototype")) {
          return false;
        }
      }
    }
  }

  return true;
}

public boolean shouldTraverse(NodeTraversal t, Node n, Node parent) {

  if (n.getType() == Token.FUNCTION) {
    // Don't traverse functions that are constructors or have the @this
    // or @override annotation.
    JSDocInfo jsDoc = getFunctionJsDocInfo(n);
    if (jsDoc != null &&
        (jsDoc.isConstructor() ||
         jsDoc.isInterface() ||
         jsDoc.hasThisType() ||
         jsDoc.isOverride())) {
      return false;
    }

    // Don't traverse functions unless they would normally
    // be able to have a @this annotation associated with them. e.g.,
    // var a = function() { }; // or
    // function a() {} // or
    // a.x = function() {}; // or
    // var a = {x: function() {}};
    int pType = parent.getType();
    if (!(pType == Token.BLOCK ||
          pType == Token.SCRIPT ||
          pType == Token.NAME ||
          pType == Token.ASSIGN ||

          // object literal keys
          pType == Token.STRING ||
          pType == Token.NUMBER)) {
      return false;
    }

    // Don't traverse functions that are getting lent to a prototype.
    Node gramps = parent.getParent();
    if (NodeUtil.isObjectLitKey(parent, gramps)) {
      JSDocInfo maybeLends = gramps.getJSDocInfo();
      if (maybeLends != null &&
          maybeLends.getLendsName() != null &&
          maybeLends.getLendsName().endsWith(".prototype")) {
        return false;
      }
    }
  }

  if (parent != null && parent.getType() == Token.ASSIGN) {
    Node lhs = parent.getFirstChild();
    Node rhs = lhs.getNext();

    if (n == lhs) {
      // Always traverse the left side of the assignment. To handle
      // nested assignments properly (e.g., (a = this).property = c;),
      // assignLhsChild should not be overridden.
      if (assignLhsChild == null) {
        assignLhsChild = lhs;
      }
    } else {
      // Only traverse the right side if it's not an assignment to a prototype
      // property or subproperty.
      if (NodeUtil.isGet(lhs)) {
        if (lhs.getType() == Token.GETPROP &&
            lhs.getLastChild().getString().equals("prototype")) {
          return false;
        }
        Node llhs = lhs.getFirstChild();
        if (llhs.getType() == Token.GETPROP &&
            llhs.getLastChild().getString().equals("prototype")) {
          return false;
        }
      }
    }
  }

  return true;
}

Some javascript toolkits (dojo, base, etc.) have a special way of declaring (what java calls) classes, for example in dojo:

dojo.declare("MyClass", [superClass1, superClass2], {
    foo: function(){ ... }
    bar: function(){ ... }
});

JSDoc (or at least JSDoc toolkit) supports this via annotations:

/**
 * @name MyClass
 * @class
 * @extends superClass1
 * @extends superClass2
 */
dojo.declare("MyClass", [superClass1, superClass2], /** @lends
MyClass.prototype */ {
    foo: function(){ ... }
    bar: function(){ ... }
});

The @lends keyword in particular is useful since it tells JSDoc that foo and bar are part of MyClass's prototype.   But closure compiler isn't picking up on that, thus I get a bunch of errors about "dangerous use of this" inside of foo() and bar().

So, can @lends support be added to the closure compiler?

The workaround is to use @this on every method, but not sure if that is sufficient to make advanced mode compilation work correctly.


void replace() {
  if (firstNode == null) {
    // Don't touch the base case ('goog').
    replacementNode = candidateDefinition;
    return;
  }

  // Handle the case where there is a duplicate definition for an explicitly
  // provided symbol.
  if (candidateDefinition != null && explicitNode != null) {
    explicitNode.detachFromParent();
    compiler.reportCodeChange();

    // Does this need a VAR keyword?
    replacementNode = candidateDefinition;
    if (NodeUtil.isExpressionNode(candidateDefinition)) {
      candidateDefinition.putBooleanProp(Node.IS_NAMESPACE, true);
      Node assignNode = candidateDefinition.getFirstChild();
      Node nameNode = assignNode.getFirstChild();
      if (nameNode.getType() == Token.NAME) {
        // Need to convert this assign to a var declaration.
        Node valueNode = nameNode.getNext();
        assignNode.removeChild(nameNode);
        assignNode.removeChild(valueNode);
        nameNode.addChildToFront(valueNode);
        Node varNode = new Node(Token.VAR, nameNode);
        varNode.copyInformationFrom(candidateDefinition);
        candidateDefinition.getParent().replaceChild(
            candidateDefinition, varNode);
        nameNode.setJSDocInfo(assignNode.getJSDocInfo());
        compiler.reportCodeChange();
        replacementNode = varNode;
      }
    }
  } else {
    // Handle the case where there's not a duplicate definition.
    replacementNode = createDeclarationNode();
    if (firstModule == minimumModule) {
      firstNode.getParent().addChildBefore(replacementNode, firstNode);
    } else {
      // In this case, the name was implicitly provided by two independent
      // modules. We need to move this code up to a common module.
      int indexOfDot = namespace.indexOf('.');
      if (indexOfDot == -1) {
        // Any old place is fine.
        compiler.getNodeForCodeInsertion(minimumModule)
            .addChildToBack(replacementNode);
      } else {
        // Add it after the parent namespace.
        ProvidedName parentName =
            providedNames.get(namespace.substring(0, indexOfDot));
        Preconditions.checkNotNull(parentName);
        Preconditions.checkNotNull(parentName.replacementNode);
        parentName.replacementNode.getParent().addChildAfter(
            replacementNode, parentName.replacementNode);
      }
    }
    if (explicitNode != null) {
      explicitNode.detachFromParent();
    }
    compiler.reportCodeChange();
  }
}

void replace() {
  if (firstNode == null) {
    // Don't touch the base case ('goog').
    replacementNode = candidateDefinition;
    return;
  }

  // Handle the case where there is a duplicate definition for an explicitly
  // provided symbol.
  if (candidateDefinition != null && explicitNode != null) {
    explicitNode.detachFromParent();
    compiler.reportCodeChange();

    // Does this need a VAR keyword?
    replacementNode = candidateDefinition;
    if (NodeUtil.isExpressionNode(candidateDefinition)) {
      candidateDefinition.putBooleanProp(Node.IS_NAMESPACE, true);
      Node assignNode = candidateDefinition.getFirstChild();
      Node nameNode = assignNode.getFirstChild();
      if (nameNode.getType() == Token.NAME) {
        // Need to convert this assign to a var declaration.
        Node valueNode = nameNode.getNext();
        assignNode.removeChild(nameNode);
        assignNode.removeChild(valueNode);
        nameNode.addChildToFront(valueNode);
        Node varNode = new Node(Token.VAR, nameNode);
        varNode.copyInformationFrom(candidateDefinition);
        candidateDefinition.getParent().replaceChild(
            candidateDefinition, varNode);
        nameNode.setJSDocInfo(assignNode.getJSDocInfo());
        compiler.reportCodeChange();
        replacementNode = varNode;
      }
    }
  } else {
    // Handle the case where there's not a duplicate definition.
    replacementNode = createDeclarationNode();
    if (firstModule == minimumModule) {
      firstNode.getParent().addChildBefore(replacementNode, firstNode);
    } else {
      // In this case, the name was implicitly provided by two independent
      // modules. We need to move this code up to a common module.
      int indexOfDot = namespace.lastIndexOf('.');
      if (indexOfDot == -1) {
        // Any old place is fine.
        compiler.getNodeForCodeInsertion(minimumModule)
            .addChildToBack(replacementNode);
      } else {
        // Add it after the parent namespace.
        ProvidedName parentName =
            providedNames.get(namespace.substring(0, indexOfDot));
        Preconditions.checkNotNull(parentName);
        Preconditions.checkNotNull(parentName.replacementNode);
        parentName.replacementNode.getParent().addChildAfter(
            replacementNode, parentName.replacementNode);
      }
    }
    if (explicitNode != null) {
      explicitNode.detachFromParent();
    }
    compiler.reportCodeChange();
  }
}

bug with implicit namespaces across modules

If there are three modules, the latter two of which depend on the root module:

// Module A
goog.provide('apps');

// Module B
goog.provide('apps.foo.bar.B');

// Module C
goog.provide('apps.foo.bar.C');

and this is compiled in SIMPLE_OPTIMIZATIONS mode, the following code will be produced:

// Module A
var apps={};apps.foo.bar={};apps.foo={};

// Module B
apps.foo.bar.B={};

// Module C
apps.foo.bar.C={};

This will result in a runtime error in Module A because apps.foo.bar is assigned before apps.foo.

The patch for the fix (with regression test) is available at:
http://codereview.appspot.com/2416041


static boolean isValidDefineValue(Node val, Set<String> defines) {
  switch (val.getType()) {
    case Token.STRING:
    case Token.NUMBER:
    case Token.TRUE:
    case Token.FALSE:
      return true;

    // Binary operators are only valid if both children are valid.
    case Token.BITAND:
    case Token.BITNOT:
    case Token.BITOR:
    case Token.BITXOR:

    // Uniary operators are valid if the child is valid.
    case Token.NOT:
    case Token.NEG:
      return isValidDefineValue(val.getFirstChild(), defines);

    // Names are valid if and only if they are defines themselves.
    case Token.NAME:
    case Token.GETPROP:
      if (val.isQualifiedName()) {
        return defines.contains(val.getQualifiedName());
      }
  }
  return false;
}

static boolean isValidDefineValue(Node val, Set<String> defines) {
  switch (val.getType()) {
    case Token.STRING:
    case Token.NUMBER:
    case Token.TRUE:
    case Token.FALSE:
      return true;

    // Binary operators are only valid if both children are valid.
    case Token.ADD:
    case Token.BITAND:
    case Token.BITNOT:
    case Token.BITOR:
    case Token.BITXOR:
    case Token.DIV:
    case Token.EQ:
    case Token.GE:
    case Token.GT:
    case Token.LE:
    case Token.LSH:
    case Token.LT:
    case Token.MOD:
    case Token.MUL:
    case Token.NE:
    case Token.RSH:
    case Token.SHEQ:
    case Token.SHNE:
    case Token.SUB:
    case Token.URSH:
      return isValidDefineValue(val.getFirstChild(), defines)
          && isValidDefineValue(val.getLastChild(), defines);

    // Uniary operators are valid if the child is valid.
    case Token.NOT:
    case Token.NEG:
    case Token.POS:
      return isValidDefineValue(val.getFirstChild(), defines);

    // Names are valid if and only if they are defines themselves.
    case Token.NAME:
    case Token.GETPROP:
      if (val.isQualifiedName()) {
        return defines.contains(val.getQualifiedName());
      }
  }
  return false;
}

closure-compiler @define annotation does not allow line to be split on 80 characters.

What steps will reproduce the problem?
1.  Create a JavaScript file with the followiing:
/** @define {string} */
var CONSTANT = "some very long string name that I want to wrap " +
               "and so break using a + since I don't want to " +
               "introduce a newline into the string."
2.  Run closure-compiler on the .js file.
3.  See it generate an error on the '+'.

What is the expected output? What do you see instead?
It should work, since the line is assigning a constant value to the var.

Please provide any additional information below.
Removing the '+' and making the string all one line does work correctly.


private void visitParameterList(NodeTraversal t, Node call,
    FunctionType functionType) {
  Iterator<Node> arguments = call.children().iterator();
  arguments.next(); // skip the function name

  Iterator<Node> parameters = functionType.getParameters().iterator();
  int ordinal = 0;
  Node parameter = null;
  Node argument = null;
  while (arguments.hasNext() &&
         parameters.hasNext()) {
    // If there are no parameters left in the list, then the while loop
    // above implies that this must be a var_args function.
      parameter = parameters.next();
    argument = arguments.next();
    ordinal++;

    validator.expectArgumentMatchesParameter(t, argument,
        getJSType(argument), getJSType(parameter), call, ordinal);
  }

  int numArgs = call.getChildCount() - 1;
  int minArgs = functionType.getMinArguments();
  int maxArgs = functionType.getMaxArguments();
  if (minArgs > numArgs || maxArgs < numArgs) {
    report(t, call, WRONG_ARGUMENT_COUNT,
            validator.getReadableJSTypeName(call.getFirstChild(), false),
            String.valueOf(numArgs), String.valueOf(minArgs),
            maxArgs != Integer.MAX_VALUE ?
            " and no more than " + maxArgs + " argument(s)" : "");
  }
}

private void visitParameterList(NodeTraversal t, Node call,
    FunctionType functionType) {
  Iterator<Node> arguments = call.children().iterator();
  arguments.next(); // skip the function name

  Iterator<Node> parameters = functionType.getParameters().iterator();
  int ordinal = 0;
  Node parameter = null;
  Node argument = null;
  while (arguments.hasNext() &&
         (parameters.hasNext() ||
          parameter != null && parameter.isVarArgs())) {
    // If there are no parameters left in the list, then the while loop
    // above implies that this must be a var_args function.
    if (parameters.hasNext()) {
      parameter = parameters.next();
    }
    argument = arguments.next();
    ordinal++;

    validator.expectArgumentMatchesParameter(t, argument,
        getJSType(argument), getJSType(parameter), call, ordinal);
  }

  int numArgs = call.getChildCount() - 1;
  int minArgs = functionType.getMinArguments();
  int maxArgs = functionType.getMaxArguments();
  if (minArgs > numArgs || maxArgs < numArgs) {
    report(t, call, WRONG_ARGUMENT_COUNT,
            validator.getReadableJSTypeName(call.getFirstChild(), false),
            String.valueOf(numArgs), String.valueOf(minArgs),
            maxArgs != Integer.MAX_VALUE ?
            " and no more than " + maxArgs + " argument(s)" : "");
  }
}

Missing type-checks for var_args notation

What steps will reproduce the problem?
1. Compile this:
//-------------------------------------
// ==ClosureCompiler==
// @compilation_level SIMPLE_OPTIMIZATIONS
// @warning_level VERBOSE
// @output_file_name default.js
// @formatting pretty_print
// ==/ClosureCompiler==

/**
* @param {...string} var_args
*/
function foo(var_args) {
    return arguments.length;
}

foo('hello'); // no warning - ok
foo(123); // warning - ok
foo('hello', 123); // no warning! error.
//-------------------------------------

What is the expected output? What do you see instead?
Should get a type-mismatch warning for the second parameter in the third foo() call.

What version of the product are you using? On what operating system?
Both online compiler and the 20100616 release.

Please provide any additional information below.
Seems like the type-checker treats 'var_args' as a single param and thus fails to type check the subsequent parameters.

// Fredrik


private Node tryFoldShift(Node n, Node left, Node right) {
  if (left.getType() == Token.NUMBER &&
      right.getType() == Token.NUMBER) {

    double result;
    double lval = left.getDouble();
    double rval = right.getDouble();

    // check ranges.  We do not do anything that would clip the double to
    // a 32-bit range, since the user likely does not intend that.
    if (!(lval >= Integer.MIN_VALUE && lval <= Integer.MAX_VALUE)) {
      error(BITWISE_OPERAND_OUT_OF_RANGE, left);
      return n;
    }

    // only the lower 5 bits are used when shifting, so don't do anything
    // if the shift amount is outside [0,32)
    if (!(rval >= 0 && rval < 32)) {
      error(SHIFT_AMOUNT_OUT_OF_BOUNDS, right);
      return n;
    }

    // Convert the numbers to ints
    int lvalInt = (int) lval;
    if (lvalInt != lval) {
      error(FRACTIONAL_BITWISE_OPERAND, left);
      return n;
    }

    int rvalInt = (int) rval;
    if (rvalInt != rval) {
      error(FRACTIONAL_BITWISE_OPERAND, right);
      return n;
    }

    switch (n.getType()) {
      case Token.LSH:
        result = lvalInt << rvalInt;
        break;
      case Token.RSH:
        result = lvalInt >> rvalInt;
        break;
      case Token.URSH:
        // JavaScript handles zero shifts on signed numbers differently than
        // Java as an Java int can not represent the unsigned 32-bit number
        // where JavaScript can so use a long here.
        result = lvalInt >>> rvalInt;
        break;
      default:
        throw new AssertionError("Unknown shift operator: " +
            Node.tokenToName(n.getType()));
    }

    Node newNumber = Node.newNumber(result);
    n.getParent().replaceChild(n, newNumber);
    reportCodeChange();

    return newNumber;
  }

  return n;
}

private Node tryFoldShift(Node n, Node left, Node right) {
  if (left.getType() == Token.NUMBER &&
      right.getType() == Token.NUMBER) {

    double result;
    double lval = left.getDouble();
    double rval = right.getDouble();

    // check ranges.  We do not do anything that would clip the double to
    // a 32-bit range, since the user likely does not intend that.
    if (!(lval >= Integer.MIN_VALUE && lval <= Integer.MAX_VALUE)) {
      error(BITWISE_OPERAND_OUT_OF_RANGE, left);
      return n;
    }

    // only the lower 5 bits are used when shifting, so don't do anything
    // if the shift amount is outside [0,32)
    if (!(rval >= 0 && rval < 32)) {
      error(SHIFT_AMOUNT_OUT_OF_BOUNDS, right);
      return n;
    }

    // Convert the numbers to ints
    int lvalInt = (int) lval;
    if (lvalInt != lval) {
      error(FRACTIONAL_BITWISE_OPERAND, left);
      return n;
    }

    int rvalInt = (int) rval;
    if (rvalInt != rval) {
      error(FRACTIONAL_BITWISE_OPERAND, right);
      return n;
    }

    switch (n.getType()) {
      case Token.LSH:
        result = lvalInt << rvalInt;
        break;
      case Token.RSH:
        result = lvalInt >> rvalInt;
        break;
      case Token.URSH:
        // JavaScript handles zero shifts on signed numbers differently than
        // Java as an Java int can not represent the unsigned 32-bit number
        // where JavaScript can so use a long here.
        long lvalLong = lvalInt & 0xffffffffL;
        result = lvalLong >>> rvalInt;
        break;
      default:
        throw new AssertionError("Unknown shift operator: " +
            Node.tokenToName(n.getType()));
    }

    Node newNumber = Node.newNumber(result);
    n.getParent().replaceChild(n, newNumber);
    reportCodeChange();

    return newNumber;
  }

  return n;
}

Unsigned Shift Right (>>>) bug operating on negative numbers

What steps will reproduce the problem?
i = -1 >>> 0 ;

What is the expected output? What do you see instead?
Expected: i = -1 >>> 0 ;  // or // i = 4294967295 ;
Instead: i = -1 ;

What version of the product are you using? On what operating system?
The UI version as of 7/18/2001 (http://closure-compiler.appspot.com/home)

Please provide any additional information below.
-1 >>> 0 == 4294967295 == Math.pow( 2, 32 ) - 1
Test in any browser and/or See ECMA-262-5 11.7.3


public boolean shouldTraverse(NodeTraversal t, Node n, Node parent) {

  if (n.getType() == Token.FUNCTION) {
    // Don't traverse functions that are constructors or have the @this
    // or @override annotation.
    JSDocInfo jsDoc = getFunctionJsDocInfo(n);
    if (jsDoc != null &&
        (jsDoc.isConstructor() ||
         jsDoc.hasThisType() ||
         jsDoc.isOverride())) {
      return false;
    }

    // Don't traverse functions unless they would normally
    // be able to have a @this annotation associated with them. e.g.,
    // var a = function() { }; // or
    // function a() {} // or
    // a.x = function() {};
    int pType = parent.getType();
    if (!(pType == Token.BLOCK ||
          pType == Token.SCRIPT ||
          pType == Token.NAME ||
          pType == Token.ASSIGN)) {
      return false;
    }
  }

  if (parent != null && parent.getType() == Token.ASSIGN) {
    Node lhs = parent.getFirstChild();
    Node rhs = lhs.getNext();

    if (n == lhs) {
      // Always traverse the left side of the assignment. To handle
      // nested assignments properly (e.g., (a = this).property = c;),
      // assignLhsChild should not be overridden.
      if (assignLhsChild == null) {
        assignLhsChild = lhs;
      }
    } else {
      // Only traverse the right side if it's not an assignment to a prototype
      // property or subproperty.
        if (lhs.getType() == Token.GETPROP &&
            lhs.getLastChild().getString().equals("prototype")) {
          return false;
        }
        if (lhs.getQualifiedName() != null && lhs.getQualifiedName().contains(".prototype.")) {
          return false;
        }
    }
  }

  return true;
}

public boolean shouldTraverse(NodeTraversal t, Node n, Node parent) {

  if (n.getType() == Token.FUNCTION) {
    // Don't traverse functions that are constructors or have the @this
    // or @override annotation.
    JSDocInfo jsDoc = getFunctionJsDocInfo(n);
    if (jsDoc != null &&
        (jsDoc.isConstructor() ||
         jsDoc.isInterface() ||
         jsDoc.hasThisType() ||
         jsDoc.isOverride())) {
      return false;
    }

    // Don't traverse functions unless they would normally
    // be able to have a @this annotation associated with them. e.g.,
    // var a = function() { }; // or
    // function a() {} // or
    // a.x = function() {};
    int pType = parent.getType();
    if (!(pType == Token.BLOCK ||
          pType == Token.SCRIPT ||
          pType == Token.NAME ||
          pType == Token.ASSIGN)) {
      return false;
    }
  }

  if (parent != null && parent.getType() == Token.ASSIGN) {
    Node lhs = parent.getFirstChild();
    Node rhs = lhs.getNext();

    if (n == lhs) {
      // Always traverse the left side of the assignment. To handle
      // nested assignments properly (e.g., (a = this).property = c;),
      // assignLhsChild should not be overridden.
      if (assignLhsChild == null) {
        assignLhsChild = lhs;
      }
    } else {
      // Only traverse the right side if it's not an assignment to a prototype
      // property or subproperty.
      if (NodeUtil.isGet(lhs)) {
        if (lhs.getType() == Token.GETPROP &&
            lhs.getLastChild().getString().equals("prototype")) {
          return false;
        }
        Node llhs = lhs.getFirstChild();
        if (llhs.getType() == Token.GETPROP &&
            llhs.getLastChild().getString().equals("prototype")) {
          return false;
        }
      }
    }
  }

  return true;
}

Prototypes declared with quotes produce a JSC_USED_GLOBAL_THIS warning.

Compiling the following code (in advanced optimizations with VERBOSE
warning levels):

/** @constructor */
function MyClass() {}
MyClass.prototype["MyMethod"] = function(a) {
  this.a = a;
}
window["MyClass"] = MyClass;

Results in the following warning: "dangerous use of the global this
object." This notation is convenient to declare a prototype that is purely
used for export purposes. The warning can be suppressed by using an @this
notation.

Given the following externs:

/**@interface */
function MyParent() {}
/** @param {*} a */
MyParent.prototype.MyMethod = function(a) {}

And the following code:

/**
* @constructor
* @implements {MyParent}
*/
function MyClass() {}
MyClass.prototype["MyMethod"] = function(a) {
  this.a2 = a;
}
window["MyClass"] = MyClass;

The compiler also produces the waring: "property MyMethod on interface
MyParent is not implemented by type MyClass".


private char getMappingCode(final String str, final int index) {
    // map() throws IllegalArgumentException
    final char mappedChar = this.map(str.charAt(index));
    // HW rule check
    if (index > 1 && mappedChar != '0') {
        final char hwChar = str.charAt(index - 1);
        if ('H' == hwChar || 'W' == hwChar) {
            final char preHWChar = str.charAt(index - 2);
            final char firstCode = this.map(preHWChar);
            if (firstCode == mappedChar || 'H' == preHWChar || 'W' == preHWChar) {
                return 0;
            }
        }
    }
    return mappedChar;
}

private char getMappingCode(final String str, final int index) {
    // map() throws IllegalArgumentException
    final char mappedChar = this.map(str.charAt(index));
    // HW rule check
    if (index > 1 && mappedChar != '0') {
        for (int i=index-1 ; i>=0 ; i--) {
            final char prevChar = str.charAt(i);
            if (this.map(prevChar)==mappedChar) {
                return 0;
            }
            if ('H'!=prevChar && 'W'!=prevChar) {
                break;
            }
        }
    }
    return mappedChar;
}

The Soundex algorithm says that if two characters that map to the same code are separated by H or W, the second one is not encoded.
However, in the implementation (in Soundex.getMappingCode() line 191), a character that is preceded by two characters that are either H or W, is not encoded, regardless of what the last consonant was.
Source: http://en.wikipedia.org/wiki/Soundex#American_Soundex


public static String newStringIso8859_1(final byte[] bytes) {
    return new String(bytes, Charsets.ISO_8859_1);
}

public static String newStringIso8859_1(final byte[] bytes) {
    return newString(bytes, Charsets.ISO_8859_1);
}

StringUtils.newStringxxx(null) should return null, not NPE

Method calls such as StringUtils.newStringIso8859_1(null) should return null, not NPE.
It looks like this capability was lost with the fix for CODEC-136, i.e.
http://svn.apache.org/viewvc?rev=1306366&view=rev
Several methods were changed from

return StringUtils.newString(bytes, CharEncoding.xxx);
to
return new String(bytes, Charsets.xxx);

The new code should have been:

return newString(bytes, Charsets.xxx);

The newString method handles null input.
There were no tests for null input so the change in behaviour was missed.


public static boolean equals(final CharSequence cs1, final CharSequence cs2) {
    if (cs1 == cs2) {
        return true;
    }
    if (cs1 == null || cs2 == null) {
        return false;
    }
    if (cs1 instanceof String && cs2 instanceof String) {
        return cs1.equals(cs2);
    }
    return CharSequenceUtils.regionMatches(cs1, false, 0, cs2, 0, Math.max(cs1.length(), cs2.length()));
}

public static boolean equals(final CharSequence cs1, final CharSequence cs2) {
    if (cs1 == cs2) {
        return true;
    }
    if (cs1 == null || cs2 == null) {
        return false;
    }
    if (cs1 instanceof String && cs2 instanceof String) {
        return cs1.equals(cs2);
    }
    return cs1.length() == cs2.length() && CharSequenceUtils.regionMatches(cs1, false, 0, cs2, 0, cs1.length());
}

StringUtils.equals(CharSequence cs1, CharSequence cs2) can fail with String Index OBE

StringUtils.equals(CharSequence cs1, CharSequence cs2) fails with String Index OBE if the two sequences are different lengths.


void encode(byte[] in, int inPos, int inAvail) {
    if (eof) {
        return;
    }
    // inAvail < 0 is how we're informed of EOF in the underlying data we're
    // encoding.
    if (inAvail < 0) {
        eof = true;
        if (buf == null || buf.length - pos < encodeSize) {
            resizeBuf();
        }
        switch (modulus) {
            case 1:
                buf[pos++] = encodeTable[(x >> 2) & MASK_6BITS];
                buf[pos++] = encodeTable[(x << 4) & MASK_6BITS];
                // URL-SAFE skips the padding to further reduce size.
                if (encodeTable == STANDARD_ENCODE_TABLE) {
                    buf[pos++] = PAD;
                    buf[pos++] = PAD;
                }
                break;

            case 2:
                buf[pos++] = encodeTable[(x >> 10) & MASK_6BITS];
                buf[pos++] = encodeTable[(x >> 4) & MASK_6BITS];
                buf[pos++] = encodeTable[(x << 2) & MASK_6BITS];
                // URL-SAFE skips the padding to further reduce size.
                if (encodeTable == STANDARD_ENCODE_TABLE) {
                    buf[pos++] = PAD;
                }
                break;
        }
        if (lineLength > 0) {
            System.arraycopy(lineSeparator, 0, buf, pos, lineSeparator.length);
            pos += lineSeparator.length;
        }
    } else {
        for (int i = 0; i < inAvail; i++) {
            if (buf == null || buf.length - pos < encodeSize) {
                resizeBuf();
            }
            modulus = (++modulus) % 3;
            int b = in[inPos++];
            if (b < 0) { b += 256; }
            x = (x << 8) + b;
            if (0 == modulus) {
                buf[pos++] = encodeTable[(x >> 18) & MASK_6BITS];
                buf[pos++] = encodeTable[(x >> 12) & MASK_6BITS];
                buf[pos++] = encodeTable[(x >> 6) & MASK_6BITS];
                buf[pos++] = encodeTable[x & MASK_6BITS];
                currentLinePos += 4;
                if (lineLength > 0 && lineLength <= currentLinePos) {
                    System.arraycopy(lineSeparator, 0, buf, pos, lineSeparator.length);
                    pos += lineSeparator.length;
                    currentLinePos = 0;
                }
            }
        }
    }
}

void encode(byte[] in, int inPos, int inAvail) {
    if (eof) {
        return;
    }
    // inAvail < 0 is how we're informed of EOF in the underlying data we're
    // encoding.
    if (inAvail < 0) {
        eof = true;
        if (buf == null || buf.length - pos < encodeSize) {
            resizeBuf();
        }
        switch (modulus) {
            case 1:
                buf[pos++] = encodeTable[(x >> 2) & MASK_6BITS];
                buf[pos++] = encodeTable[(x << 4) & MASK_6BITS];
                // URL-SAFE skips the padding to further reduce size.
                if (encodeTable == STANDARD_ENCODE_TABLE) {
                    buf[pos++] = PAD;
                    buf[pos++] = PAD;
                }
                break;

            case 2:
                buf[pos++] = encodeTable[(x >> 10) & MASK_6BITS];
                buf[pos++] = encodeTable[(x >> 4) & MASK_6BITS];
                buf[pos++] = encodeTable[(x << 2) & MASK_6BITS];
                // URL-SAFE skips the padding to further reduce size.
                if (encodeTable == STANDARD_ENCODE_TABLE) {
                    buf[pos++] = PAD;
                }
                break;
        }
        if (lineLength > 0 && pos > 0) {
            System.arraycopy(lineSeparator, 0, buf, pos, lineSeparator.length);
            pos += lineSeparator.length;
        }
    } else {
        for (int i = 0; i < inAvail; i++) {
            if (buf == null || buf.length - pos < encodeSize) {
                resizeBuf();
            }
            modulus = (++modulus) % 3;
            int b = in[inPos++];
            if (b < 0) { b += 256; }
            x = (x << 8) + b;
            if (0 == modulus) {
                buf[pos++] = encodeTable[(x >> 18) & MASK_6BITS];
                buf[pos++] = encodeTable[(x >> 12) & MASK_6BITS];
                buf[pos++] = encodeTable[(x >> 6) & MASK_6BITS];
                buf[pos++] = encodeTable[x & MASK_6BITS];
                currentLinePos += 4;
                if (lineLength > 0 && lineLength <= currentLinePos) {
                    System.arraycopy(lineSeparator, 0, buf, pos, lineSeparator.length);
                    pos += lineSeparator.length;
                    currentLinePos = 0;
                }
            }
        }
    }
}

Base64 bug with empty input (new byte[0])

Base64.encode(new byte[0]) doesn't return an empty byte array back!  It returns CRLF.


public int read(byte b[], int offset, int len) throws IOException {
    if (b == null) {
        throw new NullPointerException();
    } else if (offset < 0 || len < 0) {
        throw new IndexOutOfBoundsException();
    } else if (offset > b.length || offset + len > b.length) {
        throw new IndexOutOfBoundsException();
    } else if (len == 0) {
        return 0;
    } else {
        /*
         Rationale for while-loop on (readLen == 0):
         -----
         Base64.readResults() usually returns > 0 or EOF (-1).  In the
         rare case where it returns 0, we just keep trying.

         This is essentially an undocumented contract for InputStream
         implementors that want their code to work properly with
         java.io.InputStreamReader, since the latter hates it when
         InputStream.read(byte[]) returns a zero.  Unfortunately our
         readResults() call must return 0 if a large amount of the data
         being decoded was non-base64, so this while-loop enables proper
         interop with InputStreamReader for that scenario.
         -----
         This is a fix for CODEC-101
        */
            if (!base64.hasData()) {
                byte[] buf = new byte[doEncode ? 4096 : 8192];
                int c = in.read(buf);
                // A little optimization to avoid System.arraycopy()
                // when possible.
                if (c > 0 && b.length == len) {
                    base64.setInitialBuffer(b, offset, len);
                }
                if (doEncode) {
                    base64.encode(buf, 0, c);
                } else {
                    base64.decode(buf, 0, c);
                }
            }
        return base64.readResults(b, offset, len);
    }
}

public int read(byte b[], int offset, int len) throws IOException {
    if (b == null) {
        throw new NullPointerException();
    } else if (offset < 0 || len < 0) {
        throw new IndexOutOfBoundsException();
    } else if (offset > b.length || offset + len > b.length) {
        throw new IndexOutOfBoundsException();
    } else if (len == 0) {
        return 0;
    } else {
        int readLen = 0;
        /*
         Rationale for while-loop on (readLen == 0):
         -----
         Base64.readResults() usually returns > 0 or EOF (-1).  In the
         rare case where it returns 0, we just keep trying.

         This is essentially an undocumented contract for InputStream
         implementors that want their code to work properly with
         java.io.InputStreamReader, since the latter hates it when
         InputStream.read(byte[]) returns a zero.  Unfortunately our
         readResults() call must return 0 if a large amount of the data
         being decoded was non-base64, so this while-loop enables proper
         interop with InputStreamReader for that scenario.
         -----
         This is a fix for CODEC-101
        */
        while (readLen == 0) {
            if (!base64.hasData()) {
                byte[] buf = new byte[doEncode ? 4096 : 8192];
                int c = in.read(buf);
                // A little optimization to avoid System.arraycopy()
                // when possible.
                if (c > 0 && b.length == len) {
                    base64.setInitialBuffer(b, offset, len);
                }
                if (doEncode) {
                    base64.encode(buf, 0, c);
                } else {
                    base64.decode(buf, 0, c);
                }
            }
            readLen = base64.readResults(b, offset, len);
        }
        return readLen;
    }
}

Using new InputStreamReader(new Base64InputStream(in, true)) sometimes fails with "java.io.IOException: Underlying input stream returned zero bytes".
This is been tracked down that Base64InputStream#read(byte[]) incorrectly returns 0 at end of any stream which is multiple of 3 bytes long.


Base64.encodeBase64String() shouldn't chunk.
Change this:

public static String encodeBase64String(byte[] binaryData) {
    return StringUtils.newStringUtf8(encodeBase64(binaryData, true));
}

To this:

public static String encodeBase64String(byte[] binaryData) {
    return StringUtils.newStringUtf8(encodeBase64(binaryData, false));
}

This will fix the following tests ggregory added a few minutes ago:
        //assertEquals("Zg==", Base64.encodeBase64String(StringUtils.getBytesUtf8("f")));
        //assertEquals("Zm8=", Base64.encodeBase64String(StringUtils.getBytesUtf8("fo")));
        //assertEquals("Zm9v", Base64.encodeBase64String(StringUtils.getBytesUtf8("foo")));
        //assertEquals("Zm9vYg==", Base64.encodeBase64String(StringUtils.getBytesUtf8("foob")));
        //assertEquals("Zm9vYmE=", Base64.encodeBase64String(StringUtils.getBytesUtf8("fooba")));
        //assertEquals("Zm9vYmFy", Base64.encodeBase64String(StringUtils.getBytesUtf8("foobar")));


public static byte[] encodeBase64(byte[] binaryData, boolean isChunked, boolean urlSafe, int maxResultSize) {
    if (binaryData == null || binaryData.length == 0) {
        return binaryData;
    }

    long len = getEncodeLength(binaryData, MIME_CHUNK_SIZE, CHUNK_SEPARATOR);
    if (len > maxResultSize) {
        throw new IllegalArgumentException("Input array too big, the output array would be bigger (" +
            len +
            ") than the specified maxium size of " +
            maxResultSize);
    }

    Base64 b64 = isChunked ? new Base64(urlSafe) : new Base64(0, CHUNK_SEPARATOR, urlSafe);
    return b64.encode(binaryData);
}

public static byte[] encodeBase64(byte[] binaryData, boolean isChunked, boolean urlSafe, int maxResultSize) {
    if (binaryData == null || binaryData.length == 0) {
        return binaryData;
    }

    long len = getEncodeLength(binaryData, isChunked ? MIME_CHUNK_SIZE : 0, CHUNK_SEPARATOR);
    if (len > maxResultSize) {
        throw new IllegalArgumentException("Input array too big, the output array would be bigger (" +
            len +
            ") than the specified maxium size of " +
            maxResultSize);
    }

    Base64 b64 = isChunked ? new Base64(urlSafe) : new Base64(0, CHUNK_SEPARATOR, urlSafe);
    return b64.encode(binaryData);
}

Base64.encodeBase64(byte[] binaryData, boolean isChunked, boolean urlSafe, int maxResultSize) throws IAE for valid maxResultSize if isChunked is false

If isChunked is false, Base64.encodeBase64(byte[] binaryData, boolean isChunked, boolean urlSafe, int maxResultSize) throws IAE for valid maxResultSize.
Test case and fix will be applied shortly.


private Object readResolve() {
    calculateHashCode(keys);
    return this;
}

MultiKey subclassing has deserialization problem since COLLECTIONS-266: either declare protected readResolve() or MultiKey must be final

MultiKey from collections 4 provides a transient hashCode and a private readResolve to resolve COLLECTIONS-266: Issue with MultiKey when serialized/deserialized via RMI.
Unfortunately the solution does not work in case of subclassing: readResolve in MultiKey should be declared protected readResolve() to be called during deserialization of the subclass. Otherwise MultiKey must be final to avoid such subclassing.
Testcase:
MultiKeySerializationTest.java
package de.ivu.test.common.collections4;

import static org.junit.Assert.assertEquals;

import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.ObjectInputStream;
import java.io.ObjectOutputStream;

import org.apache.commons.collections4.keyvalue.MultiKey;
import org.junit.Test;

public class MultiKeySerializationTest {

    @Test
    @SuppressWarnings("unchecked")
    public void testReadResolveEqualHashCode()
            throws IOException, ClassNotFoundException {
        class MultiKey2<A, B>
                extends MultiKey {

            private static final long serialVersionUID = 1928896152249821416L;

            public MultiKey2(A key1, B key2) {
                super(key1, key2);
            }

            public A getFirst() {
                return (A) getKey(0);
            }

            public B getSecond() {
                return (B) getKey(1);
            }

            // FIXME: MultiKey should either declare protected readResolve() or must be final.
        }
        MultiKey2<String, String> one = new MultiKey2<>("bla", "blub");
        System.out.println(one.hashCode());
        ByteArrayOutputStream byteOut = new ByteArrayOutputStream();
        ObjectOutputStream out = new ObjectOutputStream(byteOut);
        out.writeObject(one);
        out.close();
        byte[] serialized = byteOut.toByteArray();
        ByteArrayInputStream byteIn = new ByteArrayInputStream(serialized);
        ObjectInputStream in = new ObjectInputStream(byteIn);
        MultiKey2<String, String> two = (MultiKey2<String, String>) in.readObject();
        System.out.println(two.hashCode());
        assertEquals("hashCode must be equal - please check for protected readResolve in MultiKey*", one.hashCode(),
            two.hashCode());
    }
}

Fix:
MultiKey.java
@@ -274,7 +274,7 @@
      * only stable for the same process).
      * @return the instance with recalculated hash code
      */
-    private Object readResolve() {
+    protected Object readResolve() {
         calculateHashCode(keys);
         return this;
     }


public void close() throws IOException {
    if (!this.closed) {
        super.close();
        this.closed = true;
    }
}

public void close() throws IOException {
    if (!this.closed) {
        this.finish();
        super.close();
        this.closed = true;
    }
}

When unpacking an CPIO archive (made with the compress classes or even made with OSX cpio comandline tool) an EOF exception is thrown.
Here is the testcode:
        final File input = getFile("cmdcreated.cpio");
        final InputStream in = new FileInputStream(input);
        CpioArchiveInputStream cin = new CpioArchiveInputStream(in);
        CpioArchiveEntry entry = null;
        while ((entry = (CpioArchiveEntry) cin.getNextCPIOEntry()) != null)
{
            File target = new File(dir, entry.getName());
            final OutputStream out = new FileOutputStream(target);
            IOUtils.copy(in, out);
            out.close();
        }

        cin.close();
Stacktrace is here:
java.io.EOFException
	at org.apache.commons.compress.archivers.cpio.CpioArchiveInputStream.readFully(CpioArchiveInputStream.java:293)
	at org.apache.commons.compress.archivers.cpio.CpioArchiveInputStream.getNextCPIOEntry(CpioArchiveInputStream.java:168)
	at org.apache.commons.compress.archivers.cpio.CpioArchiveInputStreamTest.testCpioUnpack(CpioArchiveInputStreamTest.java:26)
	...
This happens with the first read access to the archive. It occured while my try to improve the testcases.


private void resolveLocalFileHeaderData(Map<ZipArchiveEntry, NameAndComment>
                                        entriesWithoutUTF8Flag)
    throws IOException {
    // changing the name of a ZipArchiveEntry is going to change
    // the hashcode - see COMPRESS-164
    // Map needs to be reconstructed in order to keep central
    // directory order
    for (ZipArchiveEntry ze : entries.keySet()) {
        OffsetEntry offsetEntry = entries.get(ze);
        long offset = offsetEntry.headerOffset;
        archive.seek(offset + LFH_OFFSET_FOR_FILENAME_LENGTH);
        byte[] b = new byte[SHORT];
        archive.readFully(b);
        int fileNameLen = ZipShort.getValue(b);
        archive.readFully(b);
        int extraFieldLen = ZipShort.getValue(b);
        int lenToSkip = fileNameLen;
        while (lenToSkip > 0) {
            int skipped = archive.skipBytes(lenToSkip);
            if (skipped <= 0) {
                throw new RuntimeException("failed to skip file name in"
                                           + " local file header");
            }
            lenToSkip -= skipped;
        }
        byte[] localExtraData = new byte[extraFieldLen];
        archive.readFully(localExtraData);
        ze.setExtra(localExtraData);
        offsetEntry.dataOffset = offset + LFH_OFFSET_FOR_FILENAME_LENGTH
            + SHORT + SHORT + fileNameLen + extraFieldLen;

        if (entriesWithoutUTF8Flag.containsKey(ze)) {
            String orig = ze.getName();
            NameAndComment nc = entriesWithoutUTF8Flag.get(ze);
            ZipUtil.setNameAndCommentFromExtraFields(ze, nc.name,
                                                     nc.comment);
            if (!orig.equals(ze.getName())) {
                nameMap.remove(orig);
                nameMap.put(ze.getName(), ze);
            }
        }
    }
}

private void resolveLocalFileHeaderData(Map<ZipArchiveEntry, NameAndComment>
                                        entriesWithoutUTF8Flag)
    throws IOException {
    // changing the name of a ZipArchiveEntry is going to change
    // the hashcode - see COMPRESS-164
    // Map needs to be reconstructed in order to keep central
    // directory order
    Map<ZipArchiveEntry, OffsetEntry> origMap =
        new LinkedHashMap<ZipArchiveEntry, OffsetEntry>(entries);
    entries.clear();
    for (ZipArchiveEntry ze : origMap.keySet()) {
        OffsetEntry offsetEntry = origMap.get(ze);
        long offset = offsetEntry.headerOffset;
        archive.seek(offset + LFH_OFFSET_FOR_FILENAME_LENGTH);
        byte[] b = new byte[SHORT];
        archive.readFully(b);
        int fileNameLen = ZipShort.getValue(b);
        archive.readFully(b);
        int extraFieldLen = ZipShort.getValue(b);
        int lenToSkip = fileNameLen;
        while (lenToSkip > 0) {
            int skipped = archive.skipBytes(lenToSkip);
            if (skipped <= 0) {
                throw new RuntimeException("failed to skip file name in"
                                           + " local file header");
            }
            lenToSkip -= skipped;
        }
        byte[] localExtraData = new byte[extraFieldLen];
        archive.readFully(localExtraData);
        ze.setExtra(localExtraData);
        offsetEntry.dataOffset = offset + LFH_OFFSET_FOR_FILENAME_LENGTH
            + SHORT + SHORT + fileNameLen + extraFieldLen;

        if (entriesWithoutUTF8Flag.containsKey(ze)) {
            String orig = ze.getName();
            NameAndComment nc = entriesWithoutUTF8Flag.get(ze);
            ZipUtil.setNameAndCommentFromExtraFields(ze, nc.name,
                                                     nc.comment);
            if (!orig.equals(ze.getName())) {
                nameMap.remove(orig);
                nameMap.put(ze.getName(), ze);
            }
        }
        entries.put(ze, offsetEntry);
    }
}

Cannot Read Winzip Archives With Unicode Extra Fields

I have a zip file created with WinZip containing Unicode extra fields. Upon attempting to extract it with org.apache.commons.compress.archivers.zip.ZipFile, ZipFile.getInputStream() returns null for ZipArchiveEntries previously retrieved with ZipFile.getEntry() or even ZipFile.getEntries(). See UTF8ZipFilesTest.patch in the attachments for a test case exposing the bug. The original test case stopped short of trying to read the entries, that's why this wasn't flagged up before.
The problem lies in the fact that inside ZipFile.java entries are stored in a HashMap. However, at one point after populating the HashMap, the unicode extra fields are read, which leads to a change of the ZipArchiveEntry name, and therefore a change of its hash code. Because of this, subsequent gets on the HashMap fail to retrieve the original values.
ZipFile.patch contains an (admittedly simple-minded) fix for this problem by reconstructing the entries HashMap after the Unicode extra fields have been parsed. The purpose of this patch is mainly to show that the problem is indeed what I think, rather than providing a well-designed solution.
The patches have been tested against revision 1210416.