The external-media-tests is a Marionette-based test suite designed to test playback of video elements on arbitrary web pages. Right now, tests using this framework play videos on YouTube and Netflix pages, but the basic tests should be usable for any website which plays back HTML5 video. MediaTestCase uses the Firefox Puppeteer library.
Information about the use of external-media-tests in Mozilla automation can be found on wiki.mozilla.org.
Normally, you get this source by cloning a Firefox repo such as mozilla-central. The path to these tests would be in <mozilla-central>/dom/media/test/external, and these instructions refer to this path as
$PROJECT_HOME. You can also get these tests by downloading one of the
tests.comon.zip files matching a downloaded Firefox build from http://archive.mozilla.org, or using
mozdownload with the
--extension tests.common.zip parameters.
If you have built Firefox using
./mach build from a source tree such as mozilla-central, you can run the following command:
$ ./mach external-media-tests
You can pass any of the test options on this command line. They are listed below.
If you are testing a version of Firefox that you have not built, you must set up a virtualenv to run tests from. You will need a path to the installer or binary of Firefox.
$ virtualenv foo
$ source foo/bin/activate #or `foo\Scripts\activate` on Windows
external-media-tests in development mode. (To get an environment that is closer to what is used in Mozilla's automation jobs, run
pip install -r requirements.txt first.)
$ python setup.py develop ``` Now `external-media-tests` should be a recognized command. Try `external-media-tests --help` to see if it works.
In the examples below,
$FF_PATH is a path to a recent Firefox binary. If you are running from a source build, the commands below should be invoked with:
$ ./mach external-media-tests
If you are running with a virtualenv, you will need to run like this:
$ external-media-tests --binary $FF_PATH
$ external-media-tests --installer $FF_INSTALLER_PATH
$ external-media-tests --installer-url <url to installer package>
The following examples assume that you will use one of these command lines instead of
$EXTERNAL-MEDIA-TESTS. This runs all the tests listed in
You can also run all the tests at a particular path:
$ $EXTERNAL-MEDIA-TESTS some/path/foo
Or you can run the tests that are listed in a manifest file of your choice:
$ $EXTERNAL-MEDIA-TESTS some/other/path/manifest.ini
By default, the URLs listed in
external_media_tests/urls/default.ini are used for the tests, but you can also supply your ini file of URLs:
$ $EXTERNAL-MEDIA-TESTS --urls some/other/path/my_urls.ini
If you are running EME tests in a local build, you must build with the following parameters in your
mozconfig file before you build Firefox:
You must also copy the files
voucher.bin from a production build into your
<objdir>/dist/bin directory. This is not necessary when running production builds.
To run EME tests, you must use a Firefox profile that has the credentials of the service you are using stored in it. With Netflix, this will be created when you log in and save the credentials. You must also use a custom .ini file for URLs to the provider's content and indicate which test to run, as above. For example:
$ $EXTERNAL-MEDIA-TESTS some/path/tests.ini --profile custom_profile_dir --urls some/path/provider-urls.ini
Note, the profile argument needs to specify the directory of a profile, not the name of that profile. For example, the following arguments could be used:
$EXTERNAL-MEDIA-TESTS dom/media/test/external/external_media_tests/playback/eme.ini --profile ~/AppData/Roaming/Mozilla/Firefox/profiles/6usy8n4x.mozilla-debug-netflix/ --urls dom/
What if Firefox crashes during a test run? You want to know why! To report crash data, the test runner needs access to a
minidump_stackwalk binary and a
minidump_stackwalk binary for your platform (save it wherever). Get it from Breakpad.
$ chmod +x path/to/minidump_stackwalk # Linux and Mac
MINIDUMP_STACKWALK that points to that local path:
$ export MINIDUMP_STACKWALK=path/to/minidump_stackwalk
crashreporter-symbols.zip file for the Firefox build you are testing and extract it. For example, ftp://ftp.mozilla.org/pub/firefox/tinderbox-builds/mozilla-aurora-win32/1427442016/firefox-38.0a2.en-US.win32.crashreporter-symbols.zip
$ $EXTERNAL-MEDIA-TESTS --symbols-path path/to/example/firefox-38.0a2.en-US.win32.crashreporter-symbols
To check whether the above setup is working for you, trigger a (silly) Firefox crash while the tests are running. One way to do this is with the crashme add-on: you can add this to Firefox even while the tests are running. Another way on Linux and Mac OS systems:
$ ps x | grep 'Firefox'
# 1234 is an example of a PID
$ kill -6 1234
Somewhere in the output produced by
$EXTERNAL-MEDIA-TESTS, you should see something like:
0:12.68 CRASH: MainThread pid:1234. Test:test_basic_playback.py TestVideoPlayback.test_playback_starts.
Signature:[@ XUL + 0x2a65900]
Crash dump filename:
Operating system: Mac OS X
family 6 model 69 stepping 1
Crash reason: EXC_SOFTWARE / SIGABRT
Crash address: 0x104616900
browsermob-proxy.bat on Windows) on your machine. You may have to do a lot of work to install and use a Java that browsermobproxy would like.
Preferences->Advanced->Certificates->View Certificates->Import... Navigate to
<browsermob>/ssl-support and select
cybervilliansCA.cer. Select all of the checkboxes.
--browsermob-script <browsermob>/bin/browsermob-proxy --browsermob-port 999 --profile <your saved profile>
You can then call browsermob to shape the network. You can find an example in
external_media_tests/playback/test_playback_limiting_bandwidth.py. Another example can be found at https://dxr.mozilla.org/mozilla-central/source/testing/marionette/harness/marionette/tests/unit/test_browsermobproxy.py.
The ini files in
external_media_tests/urls may contain URLs pulled from Firefox crash or bug data. Automated tests don't care about video content, but you might: visit these at your own risk and be aware that they may be NSFW. We do ever not intend to moderate or filter these URLs.
MediaTestCase (or a subclass). You may also need to inherit from
VideoPlaybackTestsMixin, which gives you
$PROJECT_HOME/external_media_tests/media_utils for other useful methods and tests to inherit.
.ini file and place it in
$PROJECT_HOME/external_media_tests/urls, and pass it to your test via the '
--urls' command-line parameter.
.ini file in the
$PROJECT_HOME/external_media_tests/playback folder, and pass that file in your test command-line.
API testing for the classes in external-media-tests are on readthedocs.org.
API documentation is generated from the python source code itself. The documentation is added with docstring; the resulting documentation is generated with the Sphinx tool.
The document structure is located in
$PROJECT_HOME/docs. The .rst files are written for Sphinx to process. Please consult the Sphinx documentation if you need to add more sections to the .rst files we already have.
If you make changes to the docstrings in the python code, or if you make changes to the .rst files, you need to test your changes and see how they look. Follow these steps:
virtualenv docenv . docenv/bin/activate
pip install sphinx
python setup.py install
cd $PROJECT_HOME/docs make html
Note that your virtualenv must be active to rerun the
make command. Note also that if you change your python code to change documentation, you must rerun
python setup.py install to pick those changes up.
This software is licensed under the Mozilla Public License 2.0