Developing in mozperftest

Architecture overview

mozperftest implements a mach command that is a thin wrapper on the top of runner.py, which allows us to run the tool without having to go through a mach call. Command arguments are prepared in argparser.py and then made available for the runner.

The runner creates a MachEnvironment instance (see environment.py) and a Metadata instance (see metadata.py). These two objects are shared during the whole test and used to share data across all parts.

The runner then calls MachEnvironment.run, which is in charge of running the test. The MachEnvironment instance runs a sequence of layers.

Layers are classes responsible of one single aspect of a performance test. They are organized in three categories:

  • system: anything that sets up and tears down some resources or services on the system. Existing system layers: android, proxy

  • test: layers that are in charge of running a test to collect metrics. Existing test layers: browsertime and androidlog

  • metrics: all layers that process the metrics to turn them into usable metrics. Existing system layers: perfherder and console

The MachEnvironment instance collects a series of layers for each category and runs them sequentially.

The goal of this organization is to allow adding new performance tests runners that will be based on a specific combination of layers. To avoid messy code, we need to make sure that each layer represents a single aspect of the process and that is completely independent from other layers (besides sharing the data through the common environment.)

For instance, we could use perftest to run a C++ benchmark by implementing a new test layer.

Layer

A layer is a class that inherits from mozperftest.layers.Layer and implements a few methods and class variables.

List of methods and variables:

  • name: name of the layer (class variable, mandatory)

  • activated: boolean to activate by default the layer (class variable, False)

  • user_exception: will trigger the on_exception hook when an exception occurs

  • arguments: dict containing arguments. Each argument is following the argparser standard

  • run(self, medatata): called to execute the layer

  • setup(self): called when the layer is about to be executed

  • teardown(self): called when the layer is exiting

Example:

class EmailSender(Layer):
    """Sends an email with the results
    """
    name = "email"
    activated = False

    arguments = {
        "recipient": {
            "type": str,
            "default": "tarek@mozilla.com",
            "help": "Recipient",
        },
    }

    def setup(self):
        self.server = smtplib.SMTP(smtp_server,port)

    def teardown(self):
        self.server.quit()

    def __call__(self, metadata):
        self.server.send_email(self.get_arg("recipient"), metadata.results())

It can then be added to one of the top functions that are used to create a list of layers for each category:

  • mozperftest.metrics.pick_metrics for the metrics category

  • mozperftest.system.pick_system for the system category

  • mozperftest.test.pick_browser for the test category

And also added in each get_layers function in each of those category. The get_layers functions are invoked when building the argument parser.

In our example, adding the EmailSender layer will add two new options:

  • –email a flag to activate the layer

  • –email-recipient

Important layers

mozperftest can be used to run performance tests against browsers using the browsertime test layer. It leverages the browsertime.js framework and provides a full integration into Mozilla’s build and CI systems.

Browsertime uses the selenium webdriver client to drive the browser, and provides some metrics to measure performance during a user journey.

Coding style

For the coding style, we want to:

  • Follow PEP 257 for docstrings

  • Avoid complexity as much as possible

  • Use modern Python 3 code (for instance pathlib instead of os.path)

  • Avoid dependencies on Mozilla build projects and frameworks as much as possible (mozharness, mozbuild, etc), or make sure they are isolated and documented

Landing patches

Warning

It is mandatory for each patch to have a test. Any change without a test will be rejected.

Before landing a patch for mozperftest, make sure you run perftest-test:

% ./mach perftest-test
=> black [OK]
=> flake8 [OK]
=> remove old coverage data [OK]
=> running tests [OK]
=> coverage
Name                                             Stmts   Miss  Cover   Missing
------------------------------------------------------------------------------------------
mozperftest/metrics/notebook/analyzer.py         29      20     31%    26-36, 39-42, 45-51
...
mozperftest/system/proxy.py                      37      0     100%
------------------------------------------------------------------------------------------
TOTAL                                            1614    240    85%

[OK]

The command will run black, flake8 and also make sure that the test coverage has not regressed.

You can use the -s option to bypass flake8/black to speed up your workflow, but make sure you do a full tests run. You can also pass the name of one single test module.