Writing a clear, concise manual test case is an important part of quality assurance. There are a number of things you can do to make sure a manual tester has easy steps to follow when he/she runs a test case in Moztrap (moztrap.mozilla.org), Mozilla's manual testing database. Also, be aware that a good chunk of tests that we write cover "functional testing", which means the emphasis is on how the feature works. Part of manual testing is confirming that an existing feature works as expected. As we continue development on our product, we may occasionally break certain functionality, and the purpose of the manual tests in Moztrap is a "gut check" that everything works as expected. We want our users to be happy!
In Moztrap, we currently have three main types of testing. Each of them contains several sub-types:
The main difference between these tests and those in other categories is that they have been created especially to be run by the community each day. While the community can run any tests in Moztrap, these tests will only be run by them.
Basic Functionality Tests cover the minimum basic functionality of Firefox. A basic functional test of the Firefox toolbar might cover more of the common operations, and wouldn't drill down and test every level of functionality.
Smoke Tests are a much smaller subset of the basic functionality, but cover many of the common operations that browser users may do while browsing. A smoke test of the Firefox toolbar would probably cover customizing the toolbar at one level and then making sure when you restarted that Firefox honored those settings (again, this has to do with the fact that smoke tests are more focused on catching regressions). The 30min Smoke Test covers the main browser functionality and one should be able to run it in only 30 minutes. The 1h Smoke Test covers a little more since the tester has an extra half an hour, while the 2h Smoke Test should cover most of the main functionality of the browser.
Regression tests are created and run in order to ensure that existing functionality has not been broken by recent changes in the code. The Firefox X Regression Tests are created taking into account all the changes made in version X of Firefox. Running them should reveal if anything got broken by them.
The Flash Player regression tests have been created to be run each time a new Flash version is released. They are meant to check if the new Flash Player version has revealed any unknown Firefox issues, or if it has bad enough regressions itself and it needs to be blacklisted.
The GPU Drivers regression tests are run every time new GPU beta drivers are released. Just like with Flash, these tests are meant to check if the new drivers reveal unknown Firefox issues, or if they are buggy enough themselves, that they need to be blacklisted.
Each time a new feature lands in Firefox, QA creates a suite of tests for it. These tests are run several times before the feature is released, to ensure it doesn't get released with any major issues. The corresponding tests and test runs then remain in Moztrap for a while so the community can run them whenever they want. The time a run stays active is decided by the QA that created it considering the changes planned for the feature in the near future.
Q: Where should my test case go once it has been created?
A: Don’t worry too much about which group a testcase should belong to, the Mozilla QA team can help you with putting your test case in the right bucket. Just ask for some help on the #qa irc channel. If you are creating multiple tests for the same purpose, you might just create your own test suite/run.
There is certain terminology associated with many areas of the browser and the email client. An example is the browser chrome, which is another name for the graphical framework and elements of the web browser window. The browser chrome includes things such as the title bar, toolbar buttons, scrollbars, status bar and window frame.
It is probably best not to include terminology such as "chrome" in Moztrap test cases, since a wide variety of testers come to Moztrap and may not know what the "browser chrome" is. A better approach is to call out specifically the elements of the chrome, such as the window frame, status bar or scrollbars. Even if you don't know the technical name for things, be as descriptive as possible. A Mozilla QA person can always go back in and assist you in cleaning up any unclear areas.
There are some standard items in the Browser User Interface that we like to refer to, and these include:
Bottom Line: Don't get too hung up in terminology. A Mozilla QA person can help you navigate these waters.
The set of steps that you will create when writing a testcase are similar to the "Steps to Reproduce" that you might use when filing a bug (some of you might be familiar with this if you have used Bugzilla). When writing a test case, here are some helpful guidelines to use:
When creating a title for a test case, try to drill down and be as concise as possible.
Good examples would be:
Bad Examples would be:
The bad examples don't give enough information about what part of help, home page and fonts will be tested. A better phrase for both might be:
Let's now look at some sample test cases.
You are asked to write a test case to test the functionality of adding an icon to the Firefox toolbar.
When writing this test case, you will first need to think about the series of steps you would need to take to add a toolbar icon. When flushing this out, you also need to remember there are various ways to invoke the "Customize Toolbar" User Interface, and you might need to consider that fact when writing the testcase.
First, let's start with a minimal set of steps and expected results to get us to the additon of a toolbar icon.
Title: Add an icon to the Firefox toolbar
The icon should be added to the toolbar in the place that you specified.
This is an example of a basic test case. But there are a number of assumptions that are not explicity called out there, such as:
Here is the test case, rewritten to include testing the functionality of the Print icon.
Title: Drag/Select a Web Page
How this test case could be improved:
This test case is a good first cut, but could still stand some improvement. For example, in the expected results, will a user definitively be able to identify how content is selected? Usually when the content is selected, it will be highlighted in a color. That would probably be something useful to include. Also, this test case demands a page that has a scroll window, so it would useful to provide a URL for the user so they can navigate quickly to a page that has enough content to be scrolled.
Creating a Variation of this test case
If I wanted to create a variation of this test case, I might do something like this:
Title: Drag/Select a Web Page
The window should scroll downwards and all the scrolled content should be selected. Selected content is highlighted in a color that may vary depending on your system settings.
Hopefully this has given you a basic framework to use when writing a test case. As noted earlier, the Mozilla QA team is ready, willing and able to help out in our IRC channel should you need assistance.