IndexedDB Performance Testing

How to run tests on CI:

  • Windows: mach try perf --show-all -q "test-windows10-64-shippable-qr/opt-browsertime-indexeddb"

  • Linux: mach try perf --show-all -q "test-linux1804-64-shippable-qr/opt-browsertime-indexeddb"

  • Mac: mach try perf --show-all -q "test-macosx1015-64-shippable-qr/opt-browsertime-indexeddb"

  • All but 32-bit jobs: mach try perf --chrome --safari --show-all -q 'shippable-browsertime-indexeddb !32'

  • In general:

    • Open test selection interface with mach try perf --show-all

    • Filter out the preferred tests by typing letters which are expected to be part of the test job name string (as in the -q argument above)

    • Note down the string used as a filter for rerunning the job (or rerun it with mach try again --list-tasks and mach try again --index)

How to run tests locally with the profiler?

  • Build the browser with release or release with debug symbols flags (not in debug mode)

  • Use mach raptor --browsertime -t $(test_name) --gecko-profile --post-startup-delay=1000 where test name, such as addMarN is one of the items listed in testing/raptor/raptor/tests/custom/browsertime-indexeddb.ini

  • After the test is complete, the generated profile is opened with the default browser.

  • The generated profile file path is listed also in the command line output.

  • For best symbolication results, it may help to

    • run the same browser build that was used for the tests with ./mach run

    • navigate to “profiler.firefox.com”

    • use the “Load a profile from file” button

How to compare performance to a different browser?

  • The test outputs a time_duration value for all supported browsers

  • Using Chrome as an example,

    • mach raptor --browsertime -t $(test_name) --post-startup-delay=1000 --app=chrome -b "/c/Program Files/Google/Chrome/Application/chrome.exe"

    • where test name, such as addMarN is one of the items listed in testing/raptor/raptor/tests/custom/browsertime-indexeddb.ini

    • browser executable path after the -b argument varies locally

    • in some cases, a test driver argument such as --browsertime-chromedriver may be required

How to add more tests?

  • For the test boilerplate, copy and rename an old test script such as testing/raptor/browsertime/indexeddb_write.js under the testing/raptor/browsertime/ directory

  • Modify the test case script argument of commands.js.run / Selenium’s executeAsyncScript

    • Test parameters can be passed to such script with syntax ${variable_name} where variable_name represents the parameter in the context of executeAsyncScript or commands.js.run.

    • Use quotes to capture a string value, for example "${variable_name}"

    • TIP: Debugging the test case could be simpler by serving it locally without the boilerplate

  • Add [test_name] section to file testing/raptor/raptor/tests/custom/browsertime-indexeddb.ini where test_name must be 10 characters or less in order to be a valid Treeherder test name

    • Under the [test_name] section, specify the test script name as a value of test_script =

    • Under the [test_name] section, specity the test parameters as a sequence of --browsertime.key=value arguments as a value of browsertime_args =

    • Under the [test_name] section, override any other values as needed

  • Add test as a subtest to run for Desktop taskcluster/kinds/test/browsertime-desktop.yml (maybe also for mobile)

  • Add test documentation to testing/raptor/raptor/perfdocs/config.yml

  • Generated files:

    • Run ./mach lint --warnings --outgoing --fix to regenerate the documentation and task files, and warn about omissions

    • Running ./mach lint -l perfdocs --fix . may also be needed

  • Testing:

    • Test the new test by running it with the profiler

    • Test the new test by running it with a different browser

    • Test the new test by triggering it on CI