Overview

The aim of the telemetry-tests-client suite is to verify Firefox collects telemetry probes, aggregates that data, and submits telemetry pings containing the data to a HTTP server. The integration tests try to make no assumptions about the internal workings of Firefox and use automation to mimic user behavior.

The integration test suite for Firefox Client Telemetry runs on CI tier 2 with treeherder symbol tt(c) and is checked in to version control at mozilla-central under toolkit/components/telemetry/tests/marionette/.

It currently features the following two integration tests.

TestMainTabScalars

A test that opens and closes a number of browser tabs, restarts the browser in a new session and then verifies the correctness of scalar data in the resulting main ping.

See test_main_tab_scalars.py

TestSearchCounts

A test that performs a search in a new tab, restarts Firefox in a new session and verifies the correctness of client, session and subsession IDs, as well as scalar and keyed histogram data in the shutdown ping, installs an addon, verifies the environment-change ping, and performs three additional search actions before restarting and verifying the new main ping.

See test_search_counts_across_sessions.py

Running the tests locally

You can run the tests on your local machine using mach:

./mach telemetry-tests-client

Running the tests on try

You can run the tests across all platforms on the try server using mach:

./mach try fuzzy -q "'telemetry-tests-client"

Disabling an individual failing test

The telemetry-tests-client suite is implemented in Python and uses Marionette for browser automation and wptserve for the HTTP ping server. The integration tests are based on Python's unittest testing library and can be disabled by calling self.skipTest("reason") in a test method.

The example below demonstrates how to disable test_main_ping2:

import unittest

from telemetry_harness.testcase import TelemetryTestCase

class TestMainPingExample(TelemetryTestCase):
    """Example tests for the telemetry main ping."""

    def test_main_ping1(self):
        """Example test that we want to run."""

        self.search_in_new_tab("mozilla firefox")

    def test_main_ping2(self):
        """Example test that we want to skip."""

        self.skipTest("demonstrating skipping")

        self.search_in_new_tab("firefox telemetry")

Who to contact for help

• the test cases are owned by Chris Hutten-Czapski (chutten on IRC) from the Firefox Telemetry team (#telemetry on IRC)
• the test harness is owned by Raphael Pierzina (raphael on IRC) from the Firefox Test Engineering team (#fx-test on IRC)

Bugzilla

Bugs can be filed under the Toolkit product for the Telemetry component.