Creating a new ESL language compiler and verifying it

Fastpath Logic, Inc., over a three year period, built a world class ESL compiler. The input language is a C++ syntax that is used to define a chip design and verification infrastructure. The language went through several iterations, including a transition from a TCL like syntax, to the final C++ syntax. The language is used to describe those non-algorithmic components in a design and in the verification components. The language ties together the chip design and verification components. The language is split into different categories for the specification of the components that are generated in the reference model, test benches, and RTL design.

Software Pipeline
The development of a compiler requires an architecture that is effectively a software pipeline. The first stages are the lexer, parser, and tree walker (LPW). The next stage is an object model which is populated by traversing the abstract syntax tree and calling object model constructors and build methods. In addition a scope tree is used in the LPW to track the current scope. Scoping is also tracked in the object model. Languages which have classes have functions which can be called to augment the information added to the object by the arguments to the constructor. The object model is modified by a synthesis engine which converts the high level C++ description into the detailed design and verification infrastructure objects. These objects are then traversed by several visitors to generate the reference model, testbench, and RTL infrastructure. We will now describe method used to test the compiler.

Compiler Phases and Debug Messages
The compiler is divided into phases. When the compiler is compiled with the -Ddebug flag set, each phase in the compiler has a begin and end print statement that is printed out, along with any error or warning messages that occur during that compilation phase. The result is a log file that contains a set of begin/end messages and any errors or warnings in the individual phases.

Syntax for reporting compiler phase warnings and errors.
BEGIN phase_name
any warnings or errors that occur in this phase
END phase_name

Generating Tests Automatically
The object model needs to check that that arguments to the class functions are correct. This requires a detailed specification that lays out the objects that can be passed in to other objects based on the inheritance hierarchy. For example if an identifier class is inherited from an expression class in the inheritance hierarchy then the identifier may be used in most cases that an expression can be used in other class functions that take an expression argument. However, a boolean expression may not be allowed as an argument for some functions that take expression arguments. Tables must be built that describe the allowed arguments for each function. The test team can then construct automatic test generators using a language such as Perl. The tests are marked as valid or invalid in the test name (e.g. signal_bitrange_invalid.csl). The test generators are broken down along language syntax categories. Each test generator creates a set of tests in a directory. Running all test generators creates a directory hierarchy of tests broken down by language syntax category.

Regressing the Compiler
A script is written to compile each test and save the test and test results in a directory. The regression script creates a directory hierarchy based on the input directory hierarchy.

Hierarchical Regression Report
The regression script creates a hierarchical HTML report with the test results broken down along category showing the status of each of the compiler phases. Each category's results are shown on a row with the result of each compiler phase shown in the cells in the row showing the number of tests that passed that phase.

Checking for memory leaks and major errors
The regression script has an option to run valgrind on the test to check for memory leaks. If valgrind is a regression script option then the results of running valgrind is shown on each category row. The regression script inspects each test's log for major errors such as asserts and segmentation faults. A count of major error is shown on each category row. The total number of tests passing and failing are shown in the category row. The percentage of passing tests is shown in each row. The test generators generate directories of tests that contain either all valid or invalid tests. If the category test directory is valid then all tests are expected to pass. If all cells in the row should be green if the category passed. If the category test directory is invalid then all tests are expected to fail and all cells in the row should be red if the category passed.

Color coded output
At the bottom of the top level report the results of each column are displayed. We will now describe the color coding scheme used for valid tests. Invalid tests use the opposite color listed below. Cells are color coded so that the viewer can quickly see if all tests have passed and where the trouble spots are located. If 100% of the valid tests in the category pass, the cell has a green background. If the phase has warnings and no errors the cell has a yellow background. If the phase has 1 or more errors the cell background is red. In addition the result of compiling the different output languages for the category is shown. The automatically generated C++ reference model is compiled with GCC.

The report below shows the results on a local developers machine. They broke local build and the golden regressions are failing. They have to correct the bug(s) prior to checking in the code. The cells should all be green.


The summary of the regression is shown at the bottom of the regression report.



Drilling down into the categories
Clicking on a cell on row in the top level HTML report brings up a new page with the tests from the category. Note that there may be more than one page per category that lists the tests. In some instances hundreds of pages with category test results are generated which is why the top level summary for each category is important. Each of the individual tests are listed on a row. the compiler phase status for the test is shown on each cell in the row in the same way that the overall summary row is organized as described above.

Clicking on a test name brings up a new window with the test source code shown. Clicking on a test category compiler phase cell brings up the test log file as a result of compiling the test. Each compiler phase begin/end and any warnings and/or errors are shown in the log file. The cells are color coded to show the pass/fail status of the compiler phase.


Each category has a list of tests and the results of the phase is color coded.


The summary of the categories tests is shown below.



We used the Perl HTML template library to create the hierarchical web pages.

Check in regressions
Developers are required to run a golden regression which contains tests which must pass prior to check in.

Hourly regressions
Hourly regressions are run using luntbuild on a server to verify that the build is not broken. First the latest source code revision is checked out from the repository. Then the compiler is built. If the build succeeds then the hourly golden regression is run. The summary results from the hourly regression are emailed to the team. The HTML regression report for each regression is available on the build server. Users can view the results via a web browser.

Nightly regressions
Due to computing resource limitations a longer set of tests is run only once a day.
A shorter list of tests are run using valgrind.
Code coverage tools are also run.

Weekly regressions
Due to computing resource limitations a significantly longer set of tests is run only once a week.
A longer list of tests are run using valgrind.
Code coverage tools are also run.

Summary
The HTML reporting mechanism shows managers, developers, and testers a summary of the compiler regression. Regressions may be tracked over time to see the trend in compiler development. The results can be graphed to see if the project is making froward progress.