docs/api/cpp/tool-use-antlr.md

How to Generate a Parser using ANTLR4

Overview

antlr4rust is a Rust library that provides a Rust runtime for using parsers generated by ANTLR4.

Parsers are generated from ANTLR .g4 grammar files by the ANTLR4 tool.

LiteRT-LM uses ANTLR parsers to parse tool call expressions generated by LLMs.

Instructions

These are instructions for generating the Rust source files for the lexer and parser from ANTLR .g4 files. It's assumed you've already written .g4 files that define the grammar for the language you want to parse.

In the example code, the lexer is defined in AntlrFcLexer.g4 and parser is defined in AntlrFcParser.g4.

1. Build the ANTLR4 tool using Maven or download it from https://github.com/antlr4rust/antlr4/releases/.

2. Make a directory for the generated files.

   mkdir generated
   

3. Run the ANTLR4 tool to generate source files for the lexer and parser from the .g4 files:

   java -jar /path/to/antlr4-4.13.3-SNAPSHOT-complete.jar -Dlanguage=Rust AntlrFcLexer.g4 AntlrFcParser.g4 -o generated
   

4. Write a simple Rust source file that declares the generated modules, e.g. create and edit generated/antlr_fc_tool_call_parser.rs:

   //! This crate contains the generated ANTLR4 Rust modules for parsing FC tool calls.
   pub mod antlrfclexer;
   pub mod antlrfcparser;
   pub mod antlrfcparserlistener;
   

5. Write a rust_library target in generated/BUILD:

   rust_library(
       name = "antlr_fc_tool_call_parser",
       srcs = [
           "antlr_fc_tool_call_parser.rs",
           "antlrfclexer.rs",
           "antlrfcparser.rs",
           "antlrfcparserlistener.rs",
       ],
       deps = [
           "//third_party/rust/antlr4rust/v0_5:antlr4rust",
       ],
   )