Running Automated JavaScript Tests

SpiderMonkey uses two separate test suites.

The Test262 test suite is the implementation conformance test suite for ECMA-262, the language specification for JavaScript. All JavaScript engines use Test262 to ensure that they implement JavaScript correctly. Test262 is run using mach jstests.

In addition to Test262, SpiderMonkey also has a large collection of internal tests. These tests are run using mach jit-test.

Both sets of tests can be run from a normal build of the JS shell.

Running Test262 locally

The jstests shell harness is in js/src/tests/jstests.py. It can be invoked using

./mach jstests

Note that mach will generally find the JS shell itself; the –shell argument can be used to specify the location manually. All other flags will be passed along to the harness.

Test262 includes a lot of tests. When working on a specific feature, it is often useful to specify a subset of tests:

./mach jstests TEST_PATH_SUBSTRING [TEST_PATH_SUBSTRING_2 ...]

This runs all tests whose paths contain any TEST_PATH_SUBSTRING. To exclude tests, you can use the --exclude=EXCLUDED option. Other useful options include:

  • -s (--show-cmd): Show the exact command line used to run each test.

  • -o (--show-output): Print each test’s output.

  • --args SHELL_ARGS: Extra arguments to pass to the JS shell.

  • --rr Run tests under the rr record-and-replay debugger.

For a complete list of options, use:

./mach jstests -- -h

The Test262 tests can also be run in the browser, using:

./mach jstestbrowser

To run a specific test, you can use the --filter=PATTERN option, where PATTERN is a RegExp pattern that is tested against file:///{PATH_TO_OBJ_DIR}/dist/test-stage/jsreftest/tests/jsreftest.html?test={RELATIVE_PATH_TO_TEST_FROM_js/src/tests}.

For historical reasons, jstests also includes a SpiderMonkey specific suite of additional language feature tests in js/src/tests/non262.

Running jit-tests locally

The jit-test shell harness is in js/src/jit-test/jit_test.py. It can be invoked using

./mach jit-test

Basic usage of mach jit-test is similar to mach jstests. A subset of tests can be specified:

./mach jit-test TEST_PATH_SUBSTRING [TEST_PATH_SUBSTRING_2 ...]

The --jitflags option allows you to test the JS executable with different flags. The flags are used to control which features are run, such as which JITs are enabled and how quickly they will kick in. The valid options are named bundles like ‘all’, ‘interp’, ‘none’, etc. See the full list in the -h / --help output.

Another helpful option specific to jit-tests is -R <filename> (or --retest <filename>). The first time this is run, it runs the entire test suite and writes a list of tests that failed to the given filename. The next time it is run, it will run only the tests in the given filename, and then will write the list of tests that failed when it is done. Thus, you can keep retesting as you fix bugs, and only the tests that failed the last time will run, speeding things up a bit.

Other useful options include:

  • -s (--show-cmd): Show the exact command line used to run each test.

  • -o (--show-output): Print each test’s output.

  • --args SHELL_ARGS: Extra arguments to pass to the JS shell.

  • --debug-rr Run a test under the rr record-and-replay debugger.

  • --cgc Run a test with frequent compacting GCs (equivalent to SM(cgc))

Adding new jit-tests

Creating new tests for jit-tests is easy. Just add a new JS file in an appropriate directory under js/src/jit-test/tests. (By default, tests should go in test/basic.) The test harness will automatically find the test and run it. The test is considered to pass if the exit code of the JS shell is zero (i.e., JS didn’t crash and there were no JS errors). Use the assertEq function to verify values in your test.

There are some advanced options for tests. See the README (in js/src/jit-tests) for details.

Running jstests on Treeherder

On Treeherder, jstests run in the browser are shown as R(J) (search for jsreftest in mach try fuzzy). SpiderMonkey shell jobs are shown as SM(...); most of them include JS shell runs of both jstests and jit-tests.