Running jstests

Running jstests on a JS shell

The jstests shell harness is js/src/tests/jstests.py. Basic usage is:

jstests.py PATH_TO_JS_SHELL

Or using mach:

./mach jstests

Note that mach will generally find the JS shell itself; the --shell argument can be used to specify the location manually. All other flags will be passed along to the harness.

Developers will often want to use the -d option:

jstests.py -d PATH_TO_JS_SHELL

The -d option skips tests that are marked as randomly failing; random failures are usually just noise when being run for day-to-day developer testing.

Another common use case is to run just one test, or all the tests in just one directory:

jstests.py PATH_TO_JS_SHELL TEST_PATH_SUBSTRING [ TEST_PATH_SUBSTRING_2 ... ]

This runs all the tests whose paths contain TEST_PATH_SUBSTRING. Thus, you can run one test by giving its filename, or any substring that is a substring of that test filename only. You can run all tests in a directory by giving the directory path. You can run three specific tests by giving the three names. Or you can run all but selected tests with the --exclude option. Now that the full set of test262 tests is included in the tree, jstests take a long time to complete. For a smoke test or if you are not changing language-level functionality, you may wish to use

jstests.py PATH_TO_JS_SHELL --exclude=test262

Other options allow you to show the test command lines being run, command output and return codes, run tests named in a given file, exclude tests named in a given file, hide the progress bar, change the timeout, run skipped tests, print output in Tinderbox format, run a test in the debugger, or run tests in valgrind. Run with the -h option to see all the options.

The jstests test suite also includes some tests from web-platform-tests. By default, these are only run when no tests paths are specified; this can be overridden by the --wpt=enabled flag.

Running JSTest in a browser

JSTests also runs on browser, with the following command:

./mach jstestbrowser

To run specific test, you can use --filter=PATTERN command-line argument, where PATTERN is a RegExp pattern that is tested against file:///{PATH_TO_OBJ_DIR}/dist/test-stage/jsreftest/tests/jsreftest.html?test={RELATIVE_PATH_TO_TEST_FROM_js/src/tests} string:

./mach jstestbrowser --filter=PATTERN

Running jstests on Treeherder

When viewing Treeherder after a push to the Mozilla repositories, jstests run in the browser are shown as R(J) meaning "Javascript Reftests". Most of the Spidermonkey shell jobs, labeled SM(...), will include JS shell runs of both jstests and jit-tests.

Running jit-tests

The jit-test suite uses a Python harness that is similar to the jstests shell harness, but with some minor differences. Basic usage is the same:

jit_test.py PATH_TO_JS_SHELL

Developers will usually want to run like this to skip the slow tests and cover the most important options:

jit_test.py --no-slow PATH_TO_JS_SHELL

You can select specific tests to run in the same way as the jstests shell harness. Most of the options in the jstests harness are also available for this one, but a few have different names. Use -h to see all the options.

The --jitflags option allows you to test the JS executable with different flags. The flags are used to control which features are run, such as which JITs are enabled and how quickly they will kick in. The valid options are named bundles like 'all', 'interp', 'none', etc. See the full list in the -h / --help output.

One helpful option specific to jit-tests is -R filename (or --retest filename). The first time this is run, it runs the suite and writes a list of tests that failed to the given filename. The next time it is run, it will run only the tests in the given filename, and then will write the list of tests that failed when it is done. Thus, you can keep retesting as you fix bugs and only the tests that failed the last time will run, speeding things up a bit.

Running jit-tests on Treeherder

When viewing Treeherder after a push to the Mozilla repositories, jit-tests are run by calling make check, and appear under B.

Creating new tests for jit-tests

Creating new tests for jit-tests is easy. Just add a new JS file to the tests/basic directory (or any appropriate directory under 'tests'). The test harness will automatically find the test and run it. The test is considered to pass if the exit code of the JS shell is zero (i.e., JS didn't crash and there were no JS errors). Use the assertEq function to verify values in your test.

There are some advanced options for tests. See the README (in js/src/jit-tests) for details.