Automated Testing

The automated testing system for the applications running under ROSA Linux allows to check different usage scenarios of these applications by emulating user actions:

• keyboard input
• mouse movement and button clicks

The source code: https://abf.rosalinux.ru/spectre/rosa-autotest/

The system is build upon Python unit testing framework, Python 2.7 is needed. Besides that, the system includes a modified version of Xpresser tool.

Testing GUI Applications

The tests for GUI applications may run in two different modes in our system:

• A test runs locally, that is, on the same machine where the application under test is running. This mode is more suitable during the development and debugging of the tests. It can also be used for testing the applications on the real hardware.
• A test operates on the applications that run on a different (usually, virtual) machine. This is convenient for automated runs of the testing system to check the new builds of the distro. Besides, it is possible to check the boot and installation of the OS in this mode, as well as other things that are difficult to check if the tests run locally.

The tests for the applications in the installed OS should usually be written so that they could run in both modes. If the test uses only Python unittest API and the API provided by our system, this will be achieved automatically.

Testing Applications Running on a Separate Machine

Note. For most developers, it will likely be more convenient to run the tests locally instead (see below).

Setup instructions for the software needed for this mode are available here: https://abf.rosalinux.ru/spectre/rosa-autotest/tree/master/doc

The following script runs the whole test suite: launcher.py

Testing Applications Locally

Here it is assumed that everything is performed on ROSA Desktop Fresh.

• First, install the prerequisites:

# urpmi python-distribute python-xlib opencv-devel python-opencv python-numpy python-imaging
# urpmi git scrot
# easy_install pyuserinput

• Some image editor will also be needed to prepare the images for the tests (usually, parts of the screenshots). For example, GIMP can be used.

• The newest version of the testing system is available in the git repository:

$ git clone https://abf.rosalinux.ru/spectre/rosa-autotest.git

Here we assume the source code of the testing system is now in /home/user/rosa-autotest.

For Python to find the needed modules when the tests are executed, one may prepare symlinks to their directories in /usr/lib/python2.7/site-packages/ (as root):

# ln -s /home/user/rosa-autotest/rosa_autotest /usr/lib/python2.7/site-packages/
# ln -s /home/user/rosa-autotest/rosatest /usr/lib/python2.7/site-packages/
# ln -s /home/user/rosa-autotest/xpresser /usr/lib/python2.7/site-packages/

The path to the modules could be specified in PYTHONPATH environment variable instead.

To check that all the necessary paths are set correctly, one may launch Python and try to import the modules, for example:

$ python -c "import rosa_autotest, rosatest, xpresser"

If this command completes successfully, the paths are OK.

Creating the Tests - Tutorial #1

Here we assume the test developer has at least basic skills in Python (to obtain these skills, it would be more than enough to study the official tutorial: http://docs.python.org/2.7/tutorial/index.html)

Let us prepare a test that launches Firefox, opens a new tab and goes to a particular web site. Everything is done here on ROSA Desktop Fresh with KDE.

First, create a file, say, test_firefox.py and images subdirectory in the same directory. images will contain the images the test will use.

Suppose the desktop looks as follows before the test starts:

To launch Firefox, it is enough to click its icon on the panel. For the test to be able to do so automatically, let us take a screenshot and extract the icon from there: . It is often desirable to take only the inner part of that image (), to avoid depending on the background of the icon.

Save that image as ff-start.png in images subdirectory.

Now open test_firefox.py in a text editor of your choice and write the following there:

import time
import rosatest

class TestFirefox(rosatest.TestCaseInstalled):
    """Tests for Firefox"""

    @rosatest.imagebased
    def test_ff(self):
        """A simple test"""
        self.load_images("images")

        self.click("ff-start")

The needed modules are imported at the beginning, then TestFirefox is defined as a class derived from rosatest.TestCaseInstalled.

The derived classes of rosatest.TestCaseInstalled are used to test software on the installed system, the ones derived from rosatest.TestCaseLive - on the system booted in Live mode.

As usual in the development of the tests based on Python unittest, we should define one or more functions (test methods) in the class, each performing a test as a part of the test case. In this tutorial, it is test_ff().

The test functions that rely on image recognition and emulation of the user actions should be marked with @rosatest.imagebased decorator.

self.load_images("images") loads the images to be used by the test from the specified directory (images that has been created earlier).

self.click("ff-start") looks for the image images/ff-start.png on the screen and clicks the left mouse button at the center of the found image. If it does not find the image, an exception will be raised and the test will fail.

Now we can execute the test and check that it actually launches Firefox (assuming Firefox is not running yet). The easiest way is perhaps to use local.py script. The results of the test will be placed to results subdirectory of the current directory in this case.

python /home/user/rosa-autotest/rosatest/local.py <path to test_firefox.py>

Firefox should start as a result:

The following will be printed to the console:

== Starting test test_firefox.TestFirefox.test_ff ==

Ran 1 test
OK

Note. Each test has an ID in the form file_name.class_name.function_name. When executed with local.py, the test will save its results to results/file_name/class_name/function_name/ subdirectory of the current directory. Subdirectory tmp/file_name/class_name/function_name/ is used to store the temporary files during the execution of the test.

At the moment, the test only launches Firefox but does not check the browser has started successfully. We can try, for example, to check that "Home" button () has appeared.

Take a shot of the appropriate part of the screen and save the image of that button as images/ff-home-button.png. Then add the following to test_ff():

        self.wait("ff-home-button")

wait() waits till the given image appears on the screen. If the image does show up, the execution of the test continues. If the given amount of time (30 seconds by default) has passed but the image has not appeared, wait() raises an exception, the test ends and is marked as failed.

The maximum amount of time to wait for an image can be set in timeout argument, the interval between consecutive checks - in interval, for example:

        self.wait("ff-home-button", timeout=120, interval=5)

timeout is 0, wait() will not wait at all but will rather check if the image is present immediately.

If the image has not appeared so far, the test will end with an error:

== Starting test test_firefox.TestFirefox.test_ff ==

==============================
FAIL: test_ff (test_firefox.TestFirefox)
A simple test
------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/rosatest/test.py", line 349, in internal
    func(self, *args, **kwargs)
  File "/home/eugene/work/tutorial/test_firefox.py", line 13, in test_ff
    self.wait("ff-home-button", timeout=1)
  File "/usr/lib/python2.7/site-packages/rosatest/test.py", line 299, in wait
    match, "Failed to find an image on the screen: " + image)
AssertionError: Failed to find an image on the screen: ff-home-button

Ran 1 test
FAILED (failures=1)

When a test ends with an error, a screenshot will be taken at the time of failure and saved to the results directory of the test as 'scr_failure.png'. This may help analyze what has gone wrong. The log of the test (test.log in the results directory) may also contain information useful when analyzing the results, including the messages output by that test.

Note. The system uses the standard assertion methods from Python unittest. TestCase classes from rosatest module are derived from unittest.TestCase, so the tests may use the facilities from unittest directly.

Now let the test open a new tab by clicking at (save this image as images/ff-new-tab.png), and enter "www.rosalab.com" there. To achieve this, add the following to test_ff():

        self.click("ff-new-tab")

        # Wait a little for the tab to open.
        time.sleep(1)

        self.type_string("www.rosalab.com")

What remains is to press "Enter" and wait until Firefox opens the needed site:

Code:

        self.press_key("enter")
        self.wait("rosa-logo")

"www.rosalab.com" should open as a result:

The test may close Firefox now. To do this, let it press "alt-f4":

        self.press_key("alt-f4")

Note. press_key() can be used to emulate pressing of both single keys ("esc", "f4", etc.) and the combinations of keys. The combinations are written with a hyphen as a separator: "ctrl-alt-v", "shift-f7", "ctrl-a" and so on. The names of the keys used by the testing system are listed here: http://en.wikibooks.org/wiki/QEMU/Monitor#sendkey_keys. In addition to that list, there are some handy aliases: "ret" and "enter" can be used for the "Enter" key, "del" is an alias for "delete", "ins" - for "insert".

One should take into account that Firefox may (or may not) issue a warning about closing multiple tabs:

The test may check, for example, whether "Close tabs" button (, images/ff-close-tabs.png) appears on the screen and if it does, click it:

        button = self.find("ff-close-tabs", timeout=5)
        if button:
            self.click(button)

find() is similar to wait() with a single significant difference. If the image is not found on the screen, find() returns None but does not raise exceptions. The test is not considered failed.

If the image is found, both find() and wait() return an object that can be passed to click(), right_click(), double_click() and the like instead of the name of an image which is demonstrated above.

So, here is the complete test:

import time
import rosatest

class TestFirefox(rosatest.TestCaseInstalled):
    """Tests for Firefox"""

    @rosatest.imagebased
    def test_ff(self):
        """A simple test"""
        self.load_images("images")

        self.click("ff-start")
        self.wait("ff-home-button")

        self.click("ff-new-tab")

        # Wait a little for the tab to open.
        time.sleep(1)

        self.type_string("www.rosalab.com")
        self.press_key("enter")
        self.wait("rosa-logo")

        self.press_key("alt-f4")

        button = self.find("ff-close-tabs", timeout=5)
        if button:
            self.click(button)

Creating the Tests - Tutorial #2

TBD