Installing Avocado¶

sudo curl https://repos-avocadoproject.rhcloud.com/static/avocado-fedora.repo -o /etc/yum.repos.d/avocado.repo

sudo dnf repolist avocado avocado-lts
...
repo id      repo name                          status
avocado      Avocado                            50
avocado-lts  Avocado LTS (Long Term Stability)  disabled

sudo dnf install avocado

# If not already, enable epel (for RHEL7 it's following cmd)
sudo yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
# Add avocado repository and install avocado
sudo curl https://repos-avocadoproject.rhcloud.com/static/avocado-el.repo -o /etc/yum.repos.d/avocado.repo
sudo yum install avocado

sudo zypper install avocado

sudo yum install -y git gcc python-devel python-pip libvirt-devel libyaml-devel redhat-rpm-config xz-devel

git clone git://github.com/avocado-framework/avocado.git
cd avocado
sudo make requirements
sudo python setup.py install

cd optional_plugins/html
sudo python setup.py install

pip install avocado-framework

pip install -r https://raw.githubusercontent.com/avocado-framework/avocado/master/requirements.txt

Using Avocado¶

You should first experience Avocado by using the test runner, that is, the command line tool that will conveniently run your tests and collect their results.

$ avocado run /bin/true
JOB ID    : 381b849a62784228d2fd208d929cc49f310412dc
JOB LOG   : $HOME/avocado/job-results/job-2014-08-12T15.39-381b849a/job.log
TESTS     : 1
 (1/1) /bin/true: PASS (0.01 s)
RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0
TESTS TIME : 0.01 s
JOB HTML  : $HOME/avocado/job-results/job-2014-08-12T15.39-381b849a/html/results.html

avocado run /bin/true --dry-run
JOB ID     : 0000000000000000000000000000000000000000
JOB LOG    : /tmp/avocado-dry-runSeWniM/job-2015-10-16T15.46-0000000/job.log
TESTS      : 1
 (1/1) /bin/true: SKIP
RESULTS    : PASS 0 | ERROR 0 | FAIL 0 | SKIP 1 | WARN 0 | INTERRUPT 0
TESTS TIME : 0.00 s
JOB HTML   : /tmp/avocado-dry-runSeWniM/job-2015-10-16T15.46-0000000/html/results.html

$ avocado list
INSTRUMENTED /usr/share/avocado/tests/abort.py
INSTRUMENTED /usr/share/avocado/tests/datadir.py
INSTRUMENTED /usr/share/avocado/tests/doublefail.py
INSTRUMENTED /usr/share/avocado/tests/doublefree.py
INSTRUMENTED /usr/share/avocado/tests/errortest.py
INSTRUMENTED /usr/share/avocado/tests/failtest.py
INSTRUMENTED /usr/share/avocado/tests/fiotest.py
INSTRUMENTED /usr/share/avocado/tests/gdbtest.py
INSTRUMENTED /usr/share/avocado/tests/gendata.py
INSTRUMENTED /usr/share/avocado/tests/linuxbuild.py
INSTRUMENTED /usr/share/avocado/tests/multiplextest.py
INSTRUMENTED /usr/share/avocado/tests/passtest.py
INSTRUMENTED /usr/share/avocado/tests/sleeptenmin.py
INSTRUMENTED /usr/share/avocado/tests/sleeptest.py
INSTRUMENTED /usr/share/avocado/tests/synctest.py
INSTRUMENTED /usr/share/avocado/tests/timeouttest.py
INSTRUMENTED /usr/share/avocado/tests/trinity.py
INSTRUMENTED /usr/share/avocado/tests/warntest.py
INSTRUMENTED /usr/share/avocado/tests/whiteboard.py
...

$ avocado list | grep ^SIMPLE
SIMPLE       /usr/share/avocado/tests/env_variables.sh
SIMPLE       /usr/share/avocado/tests/output_check.sh
SIMPLE       /usr/share/avocado/tests/simplewarning.sh
SIMPLE       /usr/share/avocado/tests/failtest.sh
SIMPLE       /usr/share/avocado/tests/passtest.sh

$ avocado list examples/gdb-prerun-scripts/ -V
Type       file
NOT_A_TEST examples/gdb-prerun-scripts/README
NOT_A_TEST examples/gdb-prerun-scripts/pass-sigusr1

SIMPLE: 0
INSTRUMENTED: 0
MISSING: 0
NOT_A_TEST: 2

Writing a Simple Test¶

This very simple example of simple test written in shell script:

$ echo '#!/bin/bash' > /tmp/simple_test.sh
$ echo 'exit 0' >> /tmp/simple_test.sh
$ chmod +x /tmp/simple_test.sh

Notice that the file is given executable permissions, which is a requirement for Avocado to treat it as a simple test. Also notice that the script exits with status code 0, which signals a successful result to Avocado.

Running A More Complex Test Job¶

You can run any number of test in an arbitrary order, as well as mix and match instrumented and simple tests:

$ avocado run failtest.py sleeptest.py synctest.py failtest.py synctest.py /tmp/simple_test.sh
JOB ID    : 86911e49b5f2c36caeea41307cee4fecdcdfa121
JOB LOG   : $HOME/avocado/job-results/job-2014-08-12T15.42-86911e49/job.log
TESTS     : 6
 (1/6) failtest.py:FailTest.test: FAIL (0.00 s)
 (2/6) sleeptest.py:SleepTest.test: PASS (1.00 s)
 (3/6) synctest.py:SyncTest.test: PASS (2.43 s)
 (4/6) failtest.py:FailTest.test: FAIL (0.00 s)
 (5/6) synctest.py:SyncTest.test: PASS (2.44 s)
 (6/6) /bin/true: PASS (0.00 s)
 (6/6) /tmp/simple_test.sh.1: PASS (0.02 s)
RESULTS    : PASS 2 | ERROR 2 | FAIL 2 | SKIP 0 | WARN 0 | INTERRUPT 0
TESTS TIME : 5.88 s
JOB HTML  : $HOME/avocado/job-results/job-2014-08-12T15.42-86911e49/html/results.html

Interrupting The Job On First Failed Test (failfast)¶

The Avocado run command has the option --failfast on to exit the job on first failed test:

$ avocado run --failfast on /bin/true /bin/false /bin/true /bin/true
JOB ID     : eaf51b8c7d6be966bdf5562c9611b1ec2db3f68a
JOB LOG    : $HOME/avocado/job-results/job-2016-07-19T09.43-eaf51b8/job.log
TESTS      : 4
 (1/4) /bin/true: PASS (0.01 s)
 (2/4) /bin/false: FAIL (0.01 s)
Interrupting job (failfast).
RESULTS    : PASS 1 | ERROR 0 | FAIL 1 | SKIP 2 | WARN 0 | INTERRUPT 0
TESTS TIME : 0.02 s
JOB HTML   : /home/apahim/avocado/job-results/job-2016-07-19T09.43-eaf51b8/html/results.html

The --failfast option accepts the argument off. Since it’s disabled by default, the off argument only makes sense in replay jobs, when the original job was executed with --failfast on.

Running Tests With An External Runner¶

It’s quite common to have organically grown test suites in most software projects. These usually include a custom built, very specific test runner that knows how to find and run their own tests.

Still, running those tests inside Avocado may be a good idea for various reasons, including being able to have results in different human and machine readable formats, collecting system information alongside those tests (the Avocado’s sysinfo functionality), and more.

Avocado makes that possible by means of its “external runner” feature. The most basic way of using it is:

$ avocado run --external-runner=/path/to/external_runner foo bar baz

In this example, Avocado will report individual test results for tests foo, bar and baz. The actual results will be based on the return code of individual executions of /path/to/external_runner foo, /path/to/external_runner bar and finally /path/to/external_runner baz.

As another way to explain an show how this feature works, think of the “external runner” as some kind of interpreter and the individual tests as anything that this interpreter recognizes and is able to execute. A UNIX shell, say /bin/sh could be considered an external runner, and files with shell code could be considered tests:

$ echo "exit 0" > /tmp/pass
$ echo "exit 1" > /tmp/fail
$ avocado run --external-runner=/bin/sh /tmp/pass /tmp/fail
JOB ID     : 4a2a1d259690cc7b226e33facdde4f628ab30741
JOB LOG    : /home/<user>/avocado/job-results/job-<date>-<shortid>/job.log
TESTS      : 2
(1/2) /tmp/pass: PASS (0.01 s)
(2/2) /tmp/fail: FAIL (0.01 s)
RESULTS    : PASS 1 | ERROR 0 | FAIL 1 | SKIP 0 | WARN 0 | INTERRUPT 0
TESTS TIME : 0.01 s
JOB HTML   : /home/<user>/avocado/job-results/job-<date>-<shortid>/html/results.html

This example is pretty obvious, and could be achieved by giving /tmp/pass and /tmp/fail shell “shebangs” (#!/bin/sh), making them executable (chmod +x /tmp/pass /tmp/fail), and running them as “SIMPLE” tests.

But now consider the following example:

$ avocado run --external-runner=/bin/curl http://local-avocado-server:9405/jobs/ \
                                       http://remote-avocado-server:9405/jobs/
JOB ID     : 56016a1ffffaba02492fdbd5662ac0b958f51e11
JOB LOG    : /home/<user>/avocado/job-results/job-<date>-<shortid>/job.log
TESTS      : 2
(1/2) http://local-avocado-server:9405/jobs/: PASS (0.02 s)
(2/2) http://remote-avocado-server:9405/jobs/: FAIL (3.02 s)
RESULTS    : PASS 1 | ERROR 0 | FAIL 1 | SKIP 0 | WARN 0 | INTERRUPT 0
TESTS TIME : 3.04 s
JOB HTML   : /home/<user>/avocado/job-results/job-<date>-<shortid>/html/results.html

This effectively makes /bin/curl an “external test runner”, responsible for trying to fetch those URLs, and reporting PASS or FAIL for each of them.

Debugging tests¶

When developing new tests, you frequently want to look straight at the job log, without switching screens or having to “tail” the job log.

In order to do that, you can use avocado --show test run ... or avocado run --show-job-log ... options:

$ avocado --show test run examples/tests/sleeptest.py
...
Job ID: f9ea1742134e5352dec82335af584d1f151d4b85

START 1-sleeptest.py:SleepTest.test

PARAMS (key=timeout, path=*, default=None) => None
PARAMS (key=sleep_length, path=*, default=1) => 1
Sleeping for 1.00 seconds
PASS 1-sleeptest.py:SleepTest.test

Test results available in $HOME/avocado/job-results/job-2015-06-02T10.45-f9ea174

As you can see, the UI output is suppressed and only the job log is shown, making this a useful feature for test development and debugging.

Basic example¶

Let’s re-create an old time favorite, sleeptest [1]. It is so simple, it does nothing besides sleeping for a while:

import time

from avocado import Test

class SleepTest(Test):

    def test(self):
        sleep_length = self.params.get('sleep_length', default=1)
        self.log.debug("Sleeping for %.2f seconds", sleep_length)
        time.sleep(sleep_length)

This is about the simplest test you can write for Avocado, while still leveraging its API power.

Saving test generated (custom) data¶

Each test instance provides a so called whiteboard. It can be accessed through self.whiteboard. This whiteboard is simply a string that will be automatically saved to test results (as long as the output format supports it). If you choose to save binary data to the whiteboard, it’s your responsibility to encode it first (base64 is the obvious choice).

Building on the previously demonstrated sleeptest, suppose that you want to save the sleep length to be used by some other script or data analysis tool:

def test(self):
    sleep_length = self.params.get('sleep_length', default=1)
    self.log.debug("Sleeping for %.2f seconds", sleep_length)
    time.sleep(sleep_length)
    self.whiteboard = "%.2f" % sleep_length

The whiteboard can and should be exposed by files generated by the available test result plugins. The results.json file already includes the whiteboard for each test. Additionally, we’ll save a raw copy of the whiteboard contents on a file named whiteboard, in the same level as the results.json file, for your convenience (maybe you want to use the result of a benchmark directly with your custom made scripts to analyze that particular benchmark result).

Accessing test parameters¶

Each test has a set of parameters that can be accessed through self.params.get($name, $path=None, $default=None). Avocado finds and populates self.params with all parameters you define on a Multiplex Config file (see Test variants - Mux). As an example, consider the following multiplex file for sleeptest:

sleeptest:
    type: "builtin"
    length: !mux
        short:
            sleep_length: 0.5
        medium:
            sleep_length: 1
        long:
            sleep_length: 5

When running this example by avocado run $test --mux-yaml $file.yaml three variants are executed and the content is injected into /run namespace (see Test variants - Mux for details). Every variant contains variables “type” and “sleep_length”. To obtain the current value, you need the name (“sleep_length”) and its path. The path differs for each variant so it’s needed to use the most suitable portion of the path, in this example: /run/sleeptest/length/* or perhaps sleeptest/* might be enough. It depends on how your setup looks like.

The default value is optional, but always keep in mind to handle them nicely. Someone might be executing your test with different params or without any params at all. It should work fine.

So the complete example on how to access the “sleep_length” would be:

self.params.get("sleep_length", "/*/sleeptest/*", 1)

There is one way to make this even simpler. It’s possible to define resolution order, then for simple queries you can simply omit the path:

self.params.get("sleep_length", None, 1)
self.params.get("sleep_length", '*', 1)
self.params.get("sleep_length", default=1)

One should always try to avoid param clashes (multiple matching keys for given path with different origin). If it’s not possible (eg. when you use multiple yaml files) you can modify the default paths by modifying --mux-path. What it does is it slices the params and iterates through the paths one by one. When there is a match in the first slice it returns it without trying the other slices. Although relative queries only match from --mux-path slices.

There are many ways to use paths to separate clashing params or just to make more clear what your query for. Usually in tests the usage of ‘*’ is sufficient and the namespacing is not necessarily, but it helps make advanced usage clearer and easier to follow.

When thinking of the path always think about users. It’s common to extend default config with additional variants or combine them with different ones to generate just the right scenarios they need. People might simply inject the values elsewhere (eg. /run/sleeptest => /upstream/sleeptest) or they can merge other clashing file into the default path, which won’t generate clash, but would return their values instead. Then you need to clarify the path (eg. ‘*’ => sleeptest/*)

More details on that are in Test variants - Mux

Using a multiplex file¶

You may use the Avocado runner with a multiplex file to provide params and matrix generation for sleeptest just like:

$ avocado run sleeptest.py --mux-yaml examples/tests/sleeptest.py.data/sleeptest.yaml
JOB ID     : d565e8dec576d6040f894841f32a836c751f968f
JOB LOG    : $HOME/avocado/job-results/job-2014-08-12T15.44-d565e8de/job.log
TESTS      : 3
 (1/3) sleeptest.py:SleepTest.test;1: PASS (0.50 s)
 (2/3) sleeptest.py:SleepTest.test;2: PASS (1.00 s)
 (3/3) sleeptest.py:SleepTest.test;3: PASS (5.00 s)
RESULTS    : PASS 3 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0
TESTS TIME : 6.50 s
JOB HTML   : $HOME/avocado/job-results/job-2014-08-12T15.44-d565e8de/html/results.html

The --mux-yaml accepts either only $FILE_LOCATION or $INJECT_TO:$FILE_LOCATION. As explained in Test variants - Mux without any path the content gets injected into /run in order to be in the default relative path location. The $INJECT_TO can be either relative path, then it’s injected into /run/$INJECT_TO location, or absolute path (starting with '/'), then it’s injected directly into the specified path and it’s up to the test/framework developer to get the value from this location (using path or adding the path to mux-path). To understand the difference execute those commands:

$ avocado multiplex -t -m examples/tests/sleeptest.py.data/sleeptest.yaml
$ avocado multiplex -t -m duration:examples/tests/sleeptest.py.data/sleeptest.yaml
$ avocado multiplex -t -m /my/location:examples/tests/sleeptest.py.data/sleeptest.yaml

Note that, as your multiplex file specifies all parameters for sleeptest, you can’t leave the test ID empty:

$ scripts/avocado run --mux-yaml examples/tests/sleeptest/sleeptest.yaml
Empty test ID. A test path or alias must be provided

You can also execute multiple tests with the same multiplex file:

$ avocado run sleeptest.py synctest.py --mux-yaml examples/tests/sleeptest.py.data/sleeptest.yaml
JOB ID     : cd20fc8d1714da6d4791c19322374686da68c45c
JOB LOG    : $HOME/avocado/job-results/job-2016-05-04T09.25-cd20fc8/job.log
TESTS      : 8
 (1/8) sleeptest.py:SleepTest.test;1: PASS (0.50 s)
 (2/8) sleeptest.py:SleepTest.test;2: PASS (1.00 s)
 (3/8) sleeptest.py:SleepTest.test;3: PASS (5.01 s)
 (4/8) sleeptest.py:SleepTest.test;4: PASS (10.00 s)
 (5/8) synctest.py:SyncTest.test;1: PASS (2.38 s)
 (6/8) synctest.py:SyncTest.test;2: PASS (2.47 s)
 (7/8) synctest.py:SyncTest.test;3: PASS (2.46 s)
 (8/8) synctest.py:SyncTest.test;4: PASS (2.45 s)
RESULTS    : PASS 8 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0
TESTS TIME : 26.26 s
JOB HTML   : $HOME/avocado/job-results/job-2016-05-04T09.25-cd20fc8/html/results.html

Advanced logging capabilities¶

Avocado provides advanced logging capabilities at test run time. These can be combined with the standard Python library APIs on tests.

One common example is the need to follow specific progress on longer or more complex tests. Let’s look at a very simple test example, but one multiple clear stages on a single test:

import logging
import time

from avocado import Test

progress_log = logging.getLogger("progress")

class Plant(Test):

    def test_plant_organic(self):
        rows = self.params.get("rows", default=3)

        # Preparing soil
        for row in range(rows):
            progress_log.info("%s: preparing soil on row %s",
                              self.name, row)

        # Letting soil rest
        progress_log.info("%s: letting soil rest before throwing seeds",
                          self.name)
        time.sleep(2)

        # Throwing seeds
        for row in range(rows):
            progress_log.info("%s: throwing seeds on row %s",
                              self.name, row)

        # Let them grow
        progress_log.info("%s: waiting for Avocados to grow",
                          self.name)
        time.sleep(5)

        # Harvest them
        for row in range(rows):
            progress_log.info("%s: harvesting organic avocados on row %s",
                              self.name, row)

From this point on, you can ask Avocado to show your logging stream, either exclusively or in addition to other builtin streams:

$ avocado --show app,progress run plant.py

The outcome should be similar to:

JOB ID     : af786f86db530bff26cd6a92c36e99bedcdca95b
JOB LOG    : /home/cleber/avocado/job-results/job-2016-03-18T10.29-af786f8/job.log
TESTS      : 1
 (1/1) plant.py:Plant.test_plant_organic: progress: 1-plant.py:Plant.test_plant_organic: preparing soil on row 0
progress: 1-plant.py:Plant.test_plant_organic: preparing soil on row 1
progress: 1-plant.py:Plant.test_plant_organic: preparing soil on row 2
progress: 1-plant.py:Plant.test_plant_organic: letting soil rest before throwing seeds
-progress: 1-plant.py:Plant.test_plant_organic: throwing seeds on row 0
progress: 1-plant.py:Plant.test_plant_organic: throwing seeds on row 1
progress: 1-plant.py:Plant.test_plant_organic: throwing seeds on row 2
progress: 1-plant.py:Plant.test_plant_organic: waiting for Avocados to grow
\progress: 1-plant.py:Plant.test_plant_organic: harvesting organic avocados on row 0
progress: 1-plant.py:Plant.test_plant_organic: harvesting organic avocados on row 1
progress: 1-plant.py:Plant.test_plant_organic: harvesting organic avocados on row 2
PASS (7.01 s)
RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0
TESTS TIME : 7.01 s
JOB HTML   : /home/cleber/avocado/job-results/job-2016-03-18T10.29-af786f8/html/results.html

The custom progress stream is combined with the application output, which may or may not suit your needs or preferences. If you want the progress stream to be sent to a separate file, both for clarity and for persistence, you can run Avocado like this:

$ avocado run plant.py --store-logging-stream progress

The result is that, besides all the other log files commonly generated, there will be another log file named progress.INFO at the job results dir. During the test run, one could watch the progress with:

$ tail -f ~/avocado/job-results/latest/progress.INFO
10:36:59 INFO | 1-plant.py:Plant.test_plant_organic: preparing soil on row 0
10:36:59 INFO | 1-plant.py:Plant.test_plant_organic: preparing soil on row 1
10:36:59 INFO | 1-plant.py:Plant.test_plant_organic: preparing soil on row 2
10:36:59 INFO | 1-plant.py:Plant.test_plant_organic: letting soil rest before throwing seeds
10:37:01 INFO | 1-plant.py:Plant.test_plant_organic: throwing seeds on row 0
10:37:01 INFO | 1-plant.py:Plant.test_plant_organic: throwing seeds on row 1
10:37:01 INFO | 1-plant.py:Plant.test_plant_organic: throwing seeds on row 2
10:37:01 INFO | 1-plant.py:Plant.test_plant_organic: waiting for Avocados to grow
10:37:06 INFO | 1-plant.py:Plant.test_plant_organic: harvesting organic avocados on row 0
10:37:06 INFO | 1-plant.py:Plant.test_plant_organic: harvesting organic avocados on row 1
10:37:06 INFO | 1-plant.py:Plant.test_plant_organic: harvesting organic avocados on row 2

The very same progress logger, could be used across multiple test methods and across multiple test modules. In the example given, the test name is used to give extra context.

unittest.TestCase heritage¶

Since an Avocado test inherits from unittest.TestCase, you can use all the assertion methods that its parent.

The code example bellow uses assertEqual, assertTrue and assertIsInstace:

from avocado import Test

class RandomExamples(Test):
    def test(self):
        self.log.debug("Verifying some random math...")
        four = 2 * 2
        four_ = 2 + 2
        self.assertEqual(four, four_, "something is very wrong here!")

        self.log.debug("Verifying if a variable is set to True...")
        variable = True
        self.assertTrue(variable)

        self.log.debug("Verifying if this test is an instance of test.Test")
        self.assertIsInstance(self, test.Test)

$ nosetests examples/tests/sleeptest.py
.
----------------------------------------------------------------------
Ran 1 test in 1.004s

OK

from avocado import Test
from unittest import main

class Dummy(Test):
    def test(self):
        self.assertTrue(True)

if __name__ == '__main__':
    main()

$ python dummy.py
.
----------------------------------------------------------------------
Ran 1 test in 0.000s

OK

Setup and cleanup methods¶

If you need to perform setup actions before/after your test, you may do so in the setUp and tearDown methods, respectively. We’ll give examples in the following section.

Running third party test suites¶

It is very common in test automation workloads to use test suites developed by third parties. By wrapping the execution code inside an Avocado test module, you gain access to the facilities and API provided by the framework. Let’s say you want to pick up a test suite written in C that it is in a tarball, uncompress it, compile the suite code, and then executing the test. Here’s an example that does that:

#!/usr/bin/env python

import os

from avocado import Test
from avocado import main
from avocado.utils import archive
from avocado.utils import build
from avocado.utils import process


class SyncTest(Test):

    """
    Execute the synctest test suite.
    """
    default_params = {'sync_tarball': 'synctest.tar.bz2',
                      'sync_length': 100,
                      'sync_loop': 10}

    def setUp(self):
        """
        Set default params and build the synctest suite.
        """
        # Build the synctest suite
        self.cwd = os.getcwd()
        tarball_path = os.path.join(self.datadir, self.params.sync_tarball)
        archive.extract(tarball_path, self.srcdir)
        self.srcdir = os.path.join(self.srcdir, 'synctest')
        build.make(self.srcdir)

    def test(self):
        """
        Execute synctest with the appropriate params.
        """
        os.chdir(self.srcdir)
        cmd = ('./synctest %s %s' %
               (self.params.sync_length, self.params.sync_loop))
        process.system(cmd)
        os.chdir(self.cwd)


if __name__ == "__main__":
    main()

Here we have an example of the setUp method in action: Here we get the location of the test suite code (tarball) through avocado.Test.datadir(), then uncompress the tarball through avocado.utils.archive.extract(), an API that will decompress the suite tarball, followed by avocado.utils.build.make(), that will build the suite.

The setUp method is the only place in avocado where you are allowed to call the skip method, given that, if a test started to be executed, by definition it can’t be skipped anymore. Avocado will do its best to enforce this boundary, so that if you use skip outside setUp, the test upon execution will be marked with the ERROR status, and the error message will instruct you to fix your test’s code.

In this example, the test method just gets into the base directory of the compiled suite and executes the ./synctest command, with appropriate parameters, using avocado.utils.process.system().

Fetching asset files¶

To run third party test suites as mentioned above, or for any other purpose, we offer an asset fetcher as a method of Avocado Test class. The asset method looks for a list of directories in the cache_dirs key, inside the [datadir.paths] section from the configuration files. Read-only directories are also supported. When the asset file is not present in any of the provided directories, we will try to download the file from the provided locations, copying it to the first writable cache directory. Example:

cache_dirs = ['/usr/local/src/', '~/avocado/cache']

In the example above, /usr/local/src/ is a read-only directory. In that case, when we need to fetch the asset from the locations, it will be copied to the ~/avocado/cache directory.

If you don’t provide a cache_dirs, we will create a cache directory inside the avocado data_dir location to put the fetched files in.

• Use case 1: no cache_dirs key in config files, only the asset name provided in the full url format:

  ...
      def setUp(self):
          stress = 'http://people.seas.harvard.edu/~apw/stress/stress-1.0.4.tar.gz'
          tarball = self.fetch_asset(stress)
          archive.extract(tarball, self.srcdir)
  ...
  

  In this case, fetch_asset() will download the file from the url provided, copying it to the $data_dir/cache directory. tarball variable will contains, for example, /home/user/avocado/data/cache/stress-1.0.4.tar.gz.

• Use case 2: Read-only cache directory provided. cache_dirs = ['/mnt/files']:

  ...
      def setUp(self):
          stress = 'http://people.seas.harvard.edu/~apw/stress/stress-1.0.4.tar.gz'
          tarball = self.fetch_asset(stress)
          archive.extract(tarball, self.srcdir)
  ...
  

  In this case, we try to find stress-1.0.4.tar.gz file in /mnt/files directory. If it’s not there, since /mnt/files is read-only, we will try to download the asset file to the $data_dir/cache directory.

• Use case 3: Writable cache directory provided, along with a list of locations. cache_dirs = ['~/avocado/cache']:

  ...
      def setUp(self):
          st_name = 'stress-1.0.4.tar.gz'
          st_hash = 'e1533bc704928ba6e26a362452e6db8fd58b1f0b'
          st_loc = ['http://people.seas.harvard.edu/~apw/stress/stress-1.0.4.tar.gz',
                    'ftp://foo.bar/stress-1.0.4.tar.gz']
          tarball = self.fetch_asset(st_name, asset_hash=st_hash,
                                     locations=st_loc)
          archive.extract(tarball, self.srcdir)
  ...
  

  In this case, we try to download stress-1.0.4.tar.gz from the provided locations list (if it’s not already in ~/avocado/cache). The hash was also provided, so we will verify the hash. To do so, we first look for a hashfile named stress-1.0.4.tar.gz.sha1 in the same directory. If the hashfile is not present we compute the hash and create the hashfile for further usage.

  The resulting tarball variable content will be ~/avocado/cache/stress-1.0.4.tar.gz. An exception will take place if we fail to download or to verify the file.

Detailing the fetch_asset() attributes:

• name: The name used to name the fetched file. It can also contains a full URL, that will be used as the first location to try (after serching into the cache directories).
• asset_hash: (optional) The expected file hash. If missing, we skip the check. If provided, before computing the hash, we look for a hashfile to verify the asset. If the hashfile is nor present, we compute the hash and create the hashfile in the same cache directory for further usage.
• algorithm: (optional) Provided hash algorithm format. Defaults to sha1.
• locations: (optional) List of locations that will be used to try to fetch the file from. The supported schemes are http://, https://, ftp:// and file://. You’re required to inform the full url to the file, including the file name. The first success will skip the next locations. Notice that for file:// we just create a symbolic link in the cache directory, pointing to the file original location.
• expire: (optional) time period that the cached file will be considered valid. After that period, the file will be dowloaded again. The value can be an integer or a string containig the time and the unit. Example: ‘10d’ (ten days). Valid units are s (second), m (minute), h (hour) and d (day).

The expected return is the asset file path or an exception.

Test Output Check and Output Record Mode¶

In a lot of occasions, you want to go simpler: just check if the output of a given application matches an expected output. In order to help with this common use case, we offer the option --output-check-record [mode] to the test runner:

--output-check-record OUTPUT_CHECK_RECORD
                      Record output streams of your tests to reference files
                      (valid options: none (do not record output streams),
                      all (record both stdout and stderr), stdout (record
                      only stderr), stderr (record only stderr). Default:
                      none

If this option is used, it will store the stdout or stderr of the process (or both, if you specified all) being executed to reference files: stdout.expected and stderr.expected. Those files will be recorded in the test data dir. The data dir is in the same directory as the test source file, named [source_file_name.data]. Let’s take as an example the test synctest.py. In a fresh checkout of Avocado, you can see:

examples/tests/synctest.py.data/stderr.expected
examples/tests/synctest.py.data/stdout.expected

From those 2 files, only stdout.expected is non empty:

$ cat examples/tests/synctest.py.data/stdout.expected
PAR : waiting
PASS : sync interrupted

The output files were originally obtained using the test runner and passing the option –output-check-record all to the test runner:

$ scripts/avocado run --output-check-record all synctest.py
JOB ID    : bcd05e4fd33e068b159045652da9eb7448802be5
JOB LOG   : $HOME/avocado/job-results/job-2014-09-25T20.20-bcd05e4/job.log
TESTS     : 1
 (1/1) synctest.py:SyncTest.test: PASS (2.20 s)
RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0
TESTS TIME : 2.20 s

After the reference files are added, the check process is transparent, in the sense that you do not need to provide special flags to the test runner. Now, every time the test is executed, after it is done running, it will check if the outputs are exactly right before considering the test as PASSed. If you want to override the default behavior and skip output check entirely, you may provide the flag --output-check=off to the test runner.

The avocado.utils.process APIs have a parameter allow_output_check (defaults to all), so that you can select which process outputs will go to the reference files, should you chose to record them. You may choose all, for both stdout and stderr, stdout, for the stdout only, stderr, for only the stderr only, or none, to allow neither of them to be recorded and checked.

This process works fine also with simple tests, which are programs or shell scripts that returns 0 (PASSed) or != 0 (FAILed). Let’s consider our bogus example:

$ cat output_record.sh
#!/bin/bash
echo "Hello, world!"

Let’s record the output for this one:

$ scripts/avocado run output_record.sh --output-check-record all
JOB ID    : 25c4244dda71d0570b7f849319cd71fe1722be8b
JOB LOG   : $HOME/avocado/job-results/job-2014-09-25T20.49-25c4244/job.log
TESTS     : 1
 (1/1) output_record.sh: PASS (0.01 s)
RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0
TESTS TIME : 0.01 s

After this is done, you’ll notice that a the test data directory appeared in the same level of our shell script, containing 2 files:

$ ls output_record.sh.data/
stderr.expected  stdout.expected

Let’s look what’s in each of them:

$ cat output_record.sh.data/stdout.expected
Hello, world!
$ cat output_record.sh.data/stderr.expected
$

Now, every time this test runs, it’ll take into account the expected files that were recorded, no need to do anything else but run the test. Let’s see what happens if we change the stdout.expected file contents to Hello, Avocado!:

$ scripts/avocado run output_record.sh
JOB ID    : f0521e524face93019d7cb99c5765aedd933cb2e
JOB LOG   : $HOME/avocado/job-results/job-2014-09-25T20.52-f0521e5/job.log
TESTS     : 1
 (1/1) output_record.sh: FAIL (0.02 s)
RESULTS    : PASS 0 | ERROR 0 | FAIL 1 | SKIP 0 | WARN 0 | INTERRUPT 0
TESTS TIME : 0.02 s

Verifying the failure reason:

$ cat $HOME/avocado/job-results/job-2014-09-25T20.52-f0521e5/job.log
20:52:38 test       L0163 INFO | START 1-output_record.sh
20:52:38 test       L0164 DEBUG|
20:52:38 test       L0165 DEBUG| Test instance parameters:
20:52:38 test       L0173 DEBUG|
20:52:38 test       L0176 DEBUG| Default parameters:
20:52:38 test       L0180 DEBUG|
20:52:38 test       L0181 DEBUG| Test instance params override defaults whenever available
20:52:38 test       L0182 DEBUG|
20:52:38 process    L0242 INFO | Running '$HOME/Code/avocado/output_record.sh'
20:52:38 process    L0310 DEBUG| [stdout] Hello, world!
20:52:38 test       L0565 INFO | Command: $HOME/Code/avocado/output_record.sh
20:52:38 test       L0565 INFO | Exit status: 0
20:52:38 test       L0565 INFO | Duration: 0.00313782691956
20:52:38 test       L0565 INFO | Stdout:
20:52:38 test       L0565 INFO | Hello, world!
20:52:38 test       L0565 INFO |
20:52:38 test       L0565 INFO | Stderr:
20:52:38 test       L0565 INFO |
20:52:38 test       L0060 ERROR|
20:52:38 test       L0063 ERROR| Traceback (most recent call last):
20:52:38 test       L0063 ERROR|   File "$HOME/Code/avocado/avocado/test.py", line 397, in check_reference_stdout
20:52:38 test       L0063 ERROR|     self.assertEqual(expected, actual, msg)
20:52:38 test       L0063 ERROR|   File "/usr/lib64/python2.7/unittest/case.py", line 551, in assertEqual
20:52:38 test       L0063 ERROR|     assertion_func(first, second, msg=msg)
20:52:38 test       L0063 ERROR|   File "/usr/lib64/python2.7/unittest/case.py", line 544, in _baseAssertEqual
20:52:38 test       L0063 ERROR|     raise self.failureException(msg)
20:52:38 test       L0063 ERROR| AssertionError: Actual test sdtout differs from expected one:
20:52:38 test       L0063 ERROR| Actual:
20:52:38 test       L0063 ERROR| Hello, world!
20:52:38 test       L0063 ERROR|
20:52:38 test       L0063 ERROR| Expected:
20:52:38 test       L0063 ERROR| Hello, Avocado!
20:52:38 test       L0063 ERROR|
20:52:38 test       L0064 ERROR|
20:52:38 test       L0529 ERROR| FAIL 1-output_record.sh -> AssertionError: Actual test sdtout differs from expected one:
Actual:
Hello, world!

Expected:
Hello, Avocado!

20:52:38 test       L0516 INFO |

As expected, the test failed because we changed its expectations.

Test log, stdout and stderr in native Avocado modules¶

If needed, you can write directly to the expected stdout and stderr files from the native test scope. It is important to make the distinction between the following entities:

• The test logs
• The test expected stdout
• The test expected stderr

The first one is used for debugging and informational purposes. Additionally writing to self.log.warning causes test to be marked as dirty and when everything else goes well the test ends with WARN. This means that the test passed but there were non-related unexpected situations described in warning log.

You may log something into the test logs using the methods in avocado.Test.log class attributes. Consider the example:

class output_test(Test):

    def test(self):
        self.log.info('This goes to the log and it is only informational')
        self.log.warn('Oh, something unexpected, non-critical happened, '
                      'but we can continue.')
        self.log.error('Describe the error here and don't forget to raise '
                       'an exception yourself. Writing to self.log.error '
                       'won't do that for you.')
        self.log.debug('Everybody look, I had a good lunch today...')

If you need to write directly to the test stdout and stderr streams, Avocado makes two preconfigured loggers available for that purpose, named avocado.test.stdout and avocado.test.stderr. You can use Python’s standard logging API to write to them. Example:

import logging

class output_test(Test):

    def test(self):
        stdout = logging.getLogger('avocado.test.stdout')
        stdout.info('Informational line that will go to stdout')
        ...
        stderr = logging.getLogger('avocado.test.stderr')
        stderr.info('Informational line that will go to stderr')

Avocado will automatically save anything a test generates on STDOUT into a stdout file, to be found at the test results directory. The same applies to anything a test generates on STDERR, that is, it will be saved into a stderr file at the same location.

Additionally, when using the runner’s output recording features, namely the --output-check-record argument with values stdout, stderr or all, everything given to those loggers will be saved to the files stdout.expected and stderr.expected at the test’s data directory (which is different from the job/test results directory).

Avocado Tests run on a separate process¶

In order to avoid tests to mess around the environment used by the main Avocado runner process, tests are run on a forked subprocess. This allows for more robustness (tests are not easily able to mess/break Avocado) and some nifty features, such as setting test timeouts.

Setting a Test Timeout¶

Sometimes your test suite/test might get stuck forever, and this might impact your test grid. You can account for that possibility and set up a timeout parameter for your test. The test timeout can be set through 2 means, in the following order of precedence:

• Multiplex variable parameters. You may just set the timeout parameter, like in the following simplistic example:

sleep_length = 5
sleep_length_type = float
timeout = 3
timeout_type = float

$ avocado run sleeptest.py --mux-yaml /tmp/sleeptest-example.yaml
JOB ID    : 6d5a2ff16bb92395100fbc3945b8d253308728c9
JOB LOG   : $HOME/avocado/job-results/job-2014-08-12T15.52-6d5a2ff1/job.log
TESTS     : 1
 (1/1) sleeptest.py:SleepTest.test: ERROR (2.97 s)
RESULTS    : PASS 0 | ERROR 1 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0
TESTS TIME : 2.97 s
JOB HTML  : $HOME/avocado/job-results/job-2014-08-12T15.52-6d5a2ff1/html/results.html

$ cat $HOME/avocado/job-results/job-2014-08-12T15.52-6d5a2ff1/job.log
15:52:51 test       L0143 INFO | START 1-sleeptest.py
15:52:51 test       L0144 DEBUG|
15:52:51 test       L0145 DEBUG| Test log: $HOME/avocado/job-results/job-2014-08-12T15.52-6d5a2ff1/sleeptest.1/test.log
15:52:51 test       L0146 DEBUG| Test instance parameters:
15:52:51 test       L0153 DEBUG|     _name_map_file = {'sleeptest-example.yaml': 'sleeptest'}
15:52:51 test       L0153 DEBUG|     _short_name_map_file = {'sleeptest-example.yaml': 'sleeptest'}
15:52:51 test       L0153 DEBUG|     dep = []
15:52:51 test       L0153 DEBUG|     id = sleeptest
15:52:51 test       L0153 DEBUG|     name = sleeptest
15:52:51 test       L0153 DEBUG|     shortname = sleeptest
15:52:51 test       L0153 DEBUG|     sleep_length = 5.0
15:52:51 test       L0153 DEBUG|     sleep_length_type = float
15:52:51 test       L0153 DEBUG|     timeout = 3.0
15:52:51 test       L0153 DEBUG|     timeout_type = float
15:52:51 test       L0154 DEBUG|
15:52:51 test       L0157 DEBUG| Default parameters:
15:52:51 test       L0159 DEBUG|     sleep_length = 1.0
15:52:51 test       L0161 DEBUG|
15:52:51 test       L0162 DEBUG| Test instance params override defaults whenever available
15:52:51 test       L0163 DEBUG|
15:52:51 test       L0169 INFO | Test timeout set. Will wait 3.00 s for PID 15670 to end
15:52:51 test       L0170 INFO |
15:52:51 sleeptest  L0035 DEBUG| Sleeping for 5.00 seconds
15:52:54 test       L0057 ERROR|
15:52:54 test       L0060 ERROR| Traceback (most recent call last):
15:52:54 test       L0060 ERROR|   File "$HOME/Code/avocado/tests/sleeptest.py", line 36, in action
15:52:54 test       L0060 ERROR|     time.sleep(self.params.sleep_length)
15:52:54 test       L0060 ERROR|   File "$HOME/Code/avocado/avocado/job.py", line 127, in timeout_handler
15:52:54 test       L0060 ERROR|     raise exceptions.TestTimeoutError(e_msg)
15:52:54 test       L0060 ERROR| TestTimeoutError: Timeout reached waiting for sleeptest to end
15:52:54 test       L0061 ERROR|
15:52:54 test       L0400 ERROR| ERROR 1-sleeptest.py -> TestTimeoutError: Timeout reached waiting for sleeptest to end
15:52:54 test       L0387 INFO |

If you pass that multiplex file to the runner multiplexer, this will register a timeout of 3 seconds before Avocado ends the test forcefully by sending a signal.SIGTERM to the test, making it raise a avocado.core.exceptions.TestTimeoutError.

• Default params attribute. Consider the following example:

import time

from avocado import Test
from avocado import main


class TimeoutTest(Test):

    """
    Functional test for Avocado. Throw a TestTimeoutError.
    """
    default_params = {'timeout': 3.0,
                      'sleep_time': 5.0}

    def test(self):
        """
        This should throw a TestTimeoutError.
        """
        self.log.info('Sleeping for %.2f seconds (2 more than the timeout)',
                      self.params.sleep_time)
        time.sleep(self.params.sleep_time)


if __name__ == "__main__":
    main()

This accomplishes a similar effect to the multiplex setup defined in there.

$ avocado run timeouttest.py
JOB ID    : d78498a54504b481192f2f9bca5ebb9bbb820b8a
JOB LOG   : $HOME/avocado/job-results/job-2014-08-12T15.54-d78498a5/job.log
TESTS     : 1
 (1/1) timeouttest.py:TimeoutTest.test: INTERRUPTED (3.04 s)
RESULTS    : PASS 0 | ERROR 1 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0
TESTS TIME : 3.04 s
JOB HTML  : $HOME/avocado/job-results/job-2014-08-12T15.54-d78498a5/html/results.html

$ cat $HOME/avocado/job-results/job-2014-08-12T15.54-d78498a5/job.log
15:54:28 test       L0143 INFO | START 1-timeouttest.py:TimeoutTest.test
15:54:28 test       L0144 DEBUG|
15:54:28 test       L0145 DEBUG| Test log: $HOME/avocado/job-results/job-2014-08-12T15.54-d78498a5/timeouttest.1/test.log
15:54:28 test       L0146 DEBUG| Test instance parameters:
15:54:28 test       L0153 DEBUG|     id = timeouttest
15:54:28 test       L0154 DEBUG|
15:54:28 test       L0157 DEBUG| Default parameters:
15:54:28 test       L0159 DEBUG|     sleep_time = 5.0
15:54:28 test       L0159 DEBUG|     timeout = 3.0
15:54:28 test       L0161 DEBUG|
15:54:28 test       L0162 DEBUG| Test instance params override defaults whenever available
15:54:28 test       L0163 DEBUG|
15:54:28 test       L0169 INFO | Test timeout set. Will wait 3.00 s for PID 15759 to end
15:54:28 test       L0170 INFO |
15:54:28 timeouttes L0036 INFO | Sleeping for 5.00 seconds (2 more than the timeout)
15:54:31 test       L0057 ERROR|
15:54:31 test       L0060 ERROR| Traceback (most recent call last):
15:54:31 test       L0060 ERROR|   File "$HOME/Code/avocado/tests/timeouttest.py", line 37, in action
15:54:31 test       L0060 ERROR|     time.sleep(self.params.sleep_time)
15:54:31 test       L0060 ERROR|   File "$HOME/Code/avocado/avocado/job.py", line 127, in timeout_handler
15:54:31 test       L0060 ERROR|     raise exceptions.TestTimeoutError(e_msg)
15:54:31 test       L0060 ERROR| TestTimeoutError: Timeout reached waiting for timeouttest to end
15:54:31 test       L0061 ERROR|
15:54:31 test       L0400 ERROR| ERROR 1-timeouttest.py:TimeoutTest.test -> TestTimeoutError: Timeout reached waiting for timeouttest to end
15:54:31 test       L0387 INFO |

Test Tags¶

The need may arise for more complex tests, that use more advanced Python features such as inheritance. Due to the fact that Avocado uses a safe test introspection method, that is more limited than actual loading of the test classes, Avocado may need your help to identify those tests. For example, let’s say you are defining a new test class that inherits from the Avocado base test class and putting it in mylibrary.py:

from avocado import Test


class MyOwnDerivedTest(Test):
    def __init__(self, methodName='test', name=None, params=None,
                 base_logdir=None, job=None, runner_queue=None):
        super(MyOwnDerivedTest, self).__init__(methodName, name, params,
                                               base_logdir, job,
                                               runner_queue)
        self.log('Derived class example')

Then implement your actual test using that derived class, in mytest.py:

import mylibrary


class MyTest(mylibrary.MyOwnDerivedTest):

    def test1(self):
        self.log('Testing something important')

    def test2(self):
        self.log('Testing something even more important')

If you try to list the tests in that file, this is what you’ll get:

scripts/avocado list mytest.py -V
Type       Test
NOT_A_TEST mytest.py

ACCESS_DENIED: 0
BROKEN_SYMLINK: 0
EXTERNAL: 0
FILTERED: 0
INSTRUMENTED: 0
MISSING: 0
NOT_A_TEST: 1
SIMPLE: 0
VT: 0

You need to give avocado a little help by adding a docstring tag. That docstring tag is :avocado: enable. That tag tells the Avocado safe test detection code to consider it as an avocado test, regardless of what the (admittedly simple) detection code thinks of it. Let’s see how that works out. Add the docstring, as you can see the example below:

import mylibrary


class MyTest(mylibrary.MyOwnDerivedTest):
    """
    :avocado: enable
    """
    def test1(self):
        self.log('Testing something important')

    def test2(self):
        self.log('Testing something even more important')

Now, trying to list the tests on the mytest.py file again:

scripts/avocado list mytest.py -V
Type         Test
INSTRUMENTED mytest.py:MyTest.test1
INSTRUMENTED mytest.py:MyTest.test2

ACCESS_DENIED: 0
BROKEN_SYMLINK: 0
EXTERNAL: 0
FILTERED: 0
INSTRUMENTED: 2
MISSING: 0
NOT_A_TEST: 0
SIMPLE: 0
VT: 0

You can also use the :avocado: disable tag, that works the opposite way: Something looks like an Avocado test, but we force it to not be listed as one.

Python unittest Compatibility Limitations And Caveats¶

When executing tests, Avocado uses different techniques than most other Python unittest runners. This brings some compatibility limitations that Avocado users should be aware.

def setUp(self):
    print("PID: %s", os.getpid())

from avocado import Test

from avocado.utils import software_manager
from avocado.utils import path as utils_path
from avocado.utils import process


class BinSleep(Test):

    """
    Sleeps using the /bin/sleep binary
    """
    def setUp(self):
        self.sleep = None
        try:
            self.sleep = utils_path.find_command('sleep')
        except utils_path.CmdNotFoundError:
            software_manager.install_distro_packages({'fedora': ['coreutils']})
            self.sleep = utils_path.find_command('sleep')

    def test(self):
        process.run("%s 1" % self.sleep)

Environment Variables for Simple Tests¶

Avocado exports Avocado variables and multiplexed variables as BASH environment to the running test. Those variables are interesting to simple tests, because they can not make use of Avocado API directly with Python, like the native tests can do and also they can modify the test parameters.

Here are the current variables that Avocado exports to the tests:

Simple Tests BASH extensions¶

To enhance simple tests one can use supported set of libraries we created. The only requirement is to use:

PATH=$(avocado "exec-path"):$PATH

which injects path to Avocado utils into shell PATH. Take a look into avocado exec-path to see list of available functions and take a look at examples/tests/simplewarning.sh for inspiration.

Wrap Up¶

We recommend you take a look at the example tests present in the examples/tests directory, that contains a few samples to take some inspiration from. That directory, besides containing examples, is also used by the Avocado self test suite to do functional testing of Avocado itself.

It is also recommended that you take a look at the API Reference. for more possibilities.

Results for human beings¶

Avocado has two different result formats that are intended for human beings:

• Its default UI, which shows the live test execution results on a command line, text based, UI.
• The HTML report, which is generated after the test job finishes running.

$ avocado run sleeptest.py failtest.py synctest.py
JOB ID    : 5ffe479262ea9025f2e4e84c4e92055b5c79bdc9
JOB LOG   : $HOME/avocado/job-results/job-2014-08-12T15.57-5ffe4792/job.log
TESTS     : 3
 (1/3) sleeptest.py:SleepTest.test: PASS (1.01 s)
 (2/3) failtest.py:FailTest.test: FAIL (0.00 s)
 (3/3) synctest.py:SyncTest.test: PASS (1.98 s)
RESULTS    : PASS 1 | ERROR 1 | FAIL 1 | SKIP 0 | WARN 0 | INTERRUPT 0
TESTS TIME : 3.17 s
JOB HTML  : $HOME/avocado/job-results/job-2014-08-12T15.57-5ffe4792/html/results.html

$ avocado run sleeptest.py failtest.py synctest.py
...
JOB HTML  : $HOME/avocado/job-results/job-2014-08-12T15.57-5ffe4792/html/results.html
...

$ avocado run sleeptest --open-browser

Machine readable results¶

Another type of results are those intended to be parsed by other applications. Several standards exist in the test community, and Avocado can in theory support pretty much every result standard out there.

Out of the box, Avocado supports a couple of machine readable results. They are always generated and stored in the results directory in results.$type files, but you can ask for a different location too.

$ avocado run sleeptest.py failtest.py synctest.py --xunit -
<?xml version="1.0" encoding="UTF-8"?>
<testsuite name="avocado" tests="3" errors="0" failures="1" skipped="0" time="3.5769162178" timestamp="2016-05-04 14:46:52.803365">
        <testcase classname="SleepTest" name="1-sleeptest.py:SleepTest.test" time="1.00204920769"/>
        <testcase classname="FailTest" name="2-failtest.py:FailTest.test" time="0.00120401382446">
                <failure type="TestFail" message="This test is supposed to fail"><![CDATA[Traceback (most recent call last):
  File "/home/medic/Work/Projekty/avocado/avocado/avocado/core/test.py", line 490, in _run_avocado
    raise test_exception
TestFail: This test is supposed to fail
]]></failure>
                <system-out><![CDATA[14:46:53 ERROR|
14:46:53 ERROR| Reproduced traceback from: /home/medic/Work/Projekty/avocado/avocado/avocado/core/test.py:435
14:46:53 ERROR| Traceback (most recent call last):
14:46:53 ERROR|   File "/home/medic/Work/Projekty/avocado/avocado/examples/tests/failtest.py", line 17, in test
14:46:53 ERROR|     self.fail('This test is supposed to fail')
14:46:53 ERROR|   File "/home/medic/Work/Projekty/avocado/avocado/avocado/core/test.py", line 585, in fail
14:46:53 ERROR|     raise exceptions.TestFail(message)
14:46:53 ERROR| TestFail: This test is supposed to fail
14:46:53 ERROR|
14:46:53 ERROR| FAIL 2-failtest.py:FailTest.test -> TestFail: This test is supposed to fail
14:46:53 INFO |
]]></system-out>
        </testcase>
        <testcase classname="SyncTest" name="3-synctest.py:SyncTest.test" time="2.57366299629"/>
</testsuite>

$ avocado run sleeptest.py failtest.py synctest.py --json -
{
    "debuglog": "/home/cleber/avocado/job-results/job-2016-08-09T13.53-10715c4/job.log",
    "errors": 0,
    "failures": 1,
    "job_id": "10715c4645d2d2b57889d7a4317fcd01451b600e",
    "pass": 2,
    "skip": 0,
    "tests": [
        {
            "end": 1470761623.176954,
            "fail_reason": "None",
            "logdir": "/home/cleber/avocado/job-results/job-2016-08-09T13.53-10715c4/test-results/1-sleeptest.py:SleepTest.test",
            "logfile": "/home/cleber/avocado/job-results/job-2016-08-09T13.53-10715c4/test-results/1-sleeptest.py:SleepTest.test/debug.log",
            "start": 1470761622.174918,
            "status": "PASS",
            "test": "1-sleeptest.py:SleepTest.test",
            "time": 1.0020360946655273,
            "url": "1-sleeptest.py:SleepTest.test",
            "whiteboard": ""
        },
        {
            "end": 1470761623.193472,
            "fail_reason": "This test is supposed to fail",
            "logdir": "/home/cleber/avocado/job-results/job-2016-08-09T13.53-10715c4/test-results/2-failtest.py:FailTest.test",
            "logfile": "/home/cleber/avocado/job-results/job-2016-08-09T13.53-10715c4/test-results/2-failtest.py:FailTest.test/debug.log",
            "start": 1470761623.192334,
            "status": "FAIL",
            "test": "2-failtest.py:FailTest.test",
            "time": 0.0011379718780517578,
            "url": "2-failtest.py:FailTest.test",
            "whiteboard": ""
        },
        {
            "end": 1470761625.656061,
            "fail_reason": "None",
            "logdir": "/home/cleber/avocado/job-results/job-2016-08-09T13.53-10715c4/test-results/3-synctest.py:SyncTest.test",
            "logfile": "/home/cleber/avocado/job-results/job-2016-08-09T13.53-10715c4/test-results/3-synctest.py:SyncTest.test/debug.log",
            "start": 1470761623.208165,
            "status": "PASS",
            "test": "3-synctest.py:SyncTest.test",
            "time": 2.4478960037231445,
            "url": "3-synctest.py:SyncTest.test",
            "whiteboard": ""
        }
    ],
    "time": 3.4510700702667236,
    "total": 3
}

$ avocado run sleeptest.py --tap -
1..1
# debug.log of sleeptest.py:SleepTest.test:
#   12:04:38 DEBUG| PARAMS (key=sleep_length, path=*, default=1) => 1
#   12:04:38 DEBUG| Sleeping for 1.00 seconds
#   12:04:39 INFO | PASS 1-sleeptest.py:SleepTest.test
#   12:04:39 INFO |
ok 1 sleeptest.py:SleepTest.test

$ avocado --silent run failtest.py
$ echo $?
1

#!/bin/bash
...
$ avocado --silent run /path/to/my/test.py
if [ $? == 0 ]; then
   echo "great success!"
elif
   ...

Multiple results at once¶

You can have multiple results formats at once, as long as only one of them uses the standard output. For example, it is fine to use the xunit result on stdout and the JSON result to output to a file:

$ avocado run sleeptest.py synctest.py --xunit - --json /tmp/result.json
<?xml version="1.0" encoding="UTF-8"?>
<testsuite name="avocado" tests="2" errors="0" failures="0" skipped="0" time="3.64848303795" timestamp="2016-05-04 17:26:05.645665">
        <testcase classname="SleepTest" name="1-sleeptest.py:SleepTest.test" time="1.00270605087"/>
        <testcase classname="SyncTest" name="2-synctest.py:SyncTest.test" time="2.64577698708"/>
</testsuite>

$ cat /tmp/result.json
{
     "debuglog": "/home/cleber/avocado/job-results/job-2016-08-09T13.55-1a94ad6/job.log",
     "errors": 0,
     ...
}

But you won’t be able to do the same without the –json flag passed to the program:

$ avocado run sleeptest.py synctest.py --xunit - --json -
Options --json --xunit are trying to use stdout simultaneously
Please set at least one of them to a file to avoid conflicts

That’s basically the only rule, and a sane one, that you need to follow.

Exit Codes¶

Avocado exit code tries to represent different things that can happen during an execution. That means exit codes can be a combination of codes that were ORed toghether as a simgle exit code. The final exit code can be debundled so users can have a good idea on what happened to the job.

The single individual exit codes are:

• AVOCADO_ALL_OK (0)
• AVOCADO_TESTS_FAIL (1)
• AVOCADO_JOB_FAIL (2)
• AVOCADO_FAIL (4)
• AVOCADO_JOB_INTERRUPTED (8)

If a job finishes with exit code 9, for example, it means we had at least one test that failed and also we had at some point a job interruption, probably due to the job timeout or a CTRL+C.

Implementing other result formats¶

If you are looking to implement a new machine or human readable output format, you can refer to avocado.core.plugins.xunit and use it as a starting point.

If your result is something that is produced at once, based on the complete job outcome, you should create a new class that inherits from avocado.core.plugin_interfaces.Result and implements the avocado.core.plugin_interfaces.Result.render() method.

But, if your result implementation is something that outputs information live before/after each test, have to implement the old-style interface. Create a class that inherits from avocado.core.result.Result and implements all public methods, that perform actions (write to a file/stream) for each test states.

You can take a look at Plugin System for more information on how to write a plugin that will activate and execute the new result format.

Config file parsing order¶

Avocado starts by parsing what it calls system wide config file, that is shipped to all Avocado users on a system wide directory, /etc/avocado/avocado.conf. Then it’ll verify if there’s a local user config file, that is located usually in ~/.config/avocado/avocado.conf. The order of the parsing matters, so the system wide file is parsed, then the user config file is parsed last, so that the user can override values at will. There is another directory that will be scanned by extra config files, /etc/avocado/conf.d. This directory may contain plugin config files, and extra additional config files that the system administrator/avocado developers might judge necessary to put there.

Please note that for base directories, if you chose a directory that can’t be properly used by Avocado (some directories require read access, others, read and write access), Avocado will fall back to some defaults. So if your regular user wants to write logs to /root/avocado/logs, Avocado will not use that directory, since it can’t write files to that place. A new location, by default ~/avocado/job-results will be selected instead.

The order of files described in this section is only valid if avocado was installed in the system. For people using avocado from git repos (usually avocado developers), that did not install it in the system, keep in mind that avocado will read the config files present in the git repos, and will ignore the system wide config files. Running avocado config will let you know which files are actually being used.

Plugin config files¶

Plugins can also be configured by config files. In order to not disturb the main Avocado config file, those plugins, if they wish so, may install additional config files to /etc/avocado/conf.d/[pluginname].conf, that will be parsed after the system wide config file. Users can override those values as well at the local config file level. Considering the config for the hypothethical plugin salad:

[salad.core]
base = ceasar
dressing = ceasar

If you want, you may change dressing in your config file by simply adding a [salad.core] new section in your local config file, and set a different value for dressing there.

Parsing order recap¶

So the file parsing order is:

• /etc/avocado/avocado.conf
• /etc/avocado/conf.d/*.conf
• ~/.config/avocado/avocado.conf

In this order, meaning that what you set on your local config file may override what’s defined in the system wide files.

Order of precedence for values used in tests¶

Since you can use the config system to alter behavior and values used in tests (think paths to test programs, for example), we established the following order of precedence for variables (from least precedence to most):

• default value (from library or test code)
• global config file
• local (user) config file
• command line switch
• multiplexer

So the least important value comes from the library or test code default, going all the way up to the multiplexing system.

Config plugin¶

A configuration plugin is provided for users that wish to quickly see what’s defined in all sections of their Avocado configuration, after all the files are parsed in their correct resolution order. Example:

$ avocado config
Config files read (in order):
    /etc/avocado/avocado.conf
    $HOME/.config/avocado/avocado.conf

    Section.Key     Value
    runner.base_dir /usr/share/avocado
    runner.test_dir $HOME/Code/avocado/examples/tests
    runner.data_dir /usr/share/avocado/data
    runner.logs_dir ~/avocado/job-results

The command also shows the order in which your config files were parsed, giving you a better understanding of what’s going on. The Section.Key nomenclature was inspired in git config --list output.

Avocado Data Directories¶

When running tests, we are frequently looking to:

• Locate tests
• Write logs to a given location
• Grab files that will be useful for tests, such as ISO files or VM disk images

Avocado has a module dedicated to find those paths, to avoid cumbersome path manipulation magic that people had to do in previous test frameworks [1].

If you want to list all relevant directories for your test, you can use avocado config –datadir command to list those directories. Executing it will give you an output similar to the one seen below:

$ avocado config --datadir
Config files read (in order):
    /etc/avocado/avocado.conf
    $HOME/.config/avocado/avocado.conf

Avocado replaces config dirs that can't be accessed
with sensible defaults. Please edit your local config
file to customize values

Avocado Data Directories:
    base  $HOME/avocado
    tests $HOME/Code/avocado/examples/tests
    data  $HOME/avocado/data
    logs  $HOME/avocado/job-results

Note that, while Avocado will do its best to use the config values you provide in the config file, if it can’t write values to the locations provided, it will fall back to (we hope) reasonable defaults, and we notify the user about that in the output of the command.

The relevant API documentation and meaning of each of those data directories is in avocado.data_dir, so it’s highly recommended you take a look.

You may set your preferred data dirs by setting them in the Avocado config files. The only exception for important data dirs here is the Avocado tmp dir, used to place temporary files used by tests. That directory will be in normal circumstances /var/tmp/avocado_XXXXX, (where XXXXX is in actuality a random string) securely created on /var/tmp/, unless the user has the $TMPDIR environment variable set, since that is customary among unix programs.

The next section of the documentation explains how you can see and set config values that modify the behavior for the Avocado utilities and plugins.

[1] For example, autotest.

The order of test loaders¶

Avocado supports different types of test starting with SIMPLE tests, which are simply executable files, then unittest-like tests called INSTRUMENTED up to some tests like the avocado-vt ones, which uses complex matrix of tests from config files that don’t directly map to existing files. Given the number of loaders, the mapping from test names on the command line to executed tests might not always be unique. Additionally some people might always (or for given run) want to execute only tests of a single type.

To adjust this behavior you can either tweak plugins.loaders in avocado settings (/etc/avocado/), or temporarily using --loaders (option of avocado run) option.

This option allows you to specify order and some params of the available test loaders. You can specify either loader_name (file), loader_name + TEST_TYPE (file.SIMPLE) and for some loaders even additional params passed after : (external:/bin/echo -e. You can also supply @DEFAULT, which injects into that position all the remaining unused loaders.

To get help about --loaders:

$ avocado run --loaders ?
$ avocado run --loaders external:?

Example of how --loaders affects the produced tests (manually gathered as some of them result in error):

$ avocado run passtest.py boot this_does_not_exist /bin/echo
    > INSTRUMENTED passtest.py:PassTest.test
    > VT           io-github-autotest-qemu.boot
    > MISSING      this_does_not_exist
    > SIMPLE       /bin/echo
$ avocado run passtest.py boot this_does_not_exist /bin/echo --loaders @DEFAULT "external:/bin/echo -e"
    > INSTRUMENTED passtest.py:PassTest.test
    > VT           io-github-autotest-qemu.boot
    > EXTERNAL     this_does_not_exist
    > SIMPLE       /bin/echo
$ avocado run passtest.py boot this_does_not_exist /bin/echo --loaders file.SIMPLE file.INSTRUMENTED @DEFAULT external.EXTERNAL:/bin/echo
    > INSTRUMENTED passtest.py:PassTest.test
    > VT           io-github-autotest-qemu.boot
    > EXTERNAL     this_does_not_exist
    > SIMPLE       /bin/echo

Tweaking the UI¶

Avocado uses python’s logging system to produce UI and to store test’s output. The system is quite flexible and allows you to tweak the output to your needs either by built-in stream sets, or directly by using the stream name. To tweak them you can use avocado –show STREAM[:LEVEL][,STREAM[:LEVEL],...]. Built-in streams with description (followed by list of associated python streams):

Additionally you can specify “all” or “none” to enable/disable all of pre-defined streams and you can also supply custom python logging streams and they will be passed to the standard output.

Storing custom logs¶

When you run a test, you can also store custom logging streams into the results directory by avocado run –store-logging-stream [STREAM[:LEVEL] [STREAM[:LEVEL] ...]], which will produce $STREAM.$LEVEL files per each (unique) entry in the test results directory.

Paginator¶

Some subcommands (list, plugins, ...) support “paginator”, which, on compatible terminals, basically pipes the colored output to less to simplify browsing of the produced output. One can disable it by –paginator {on|off}.

Mux internals¶

The Mux is a core part of avocado and one can see it as a multiplexed database, which contains key/value pairs associated to given paths and as we are talking about a tree of those, we call the paths Nodes.

Mux allows iterating through all possible combinations which are stored in the database, which is called multiplexation. Mux yields variants, which are lists of leaf nodes with their values, which are then processed into AvocadoParams. Those params are available in tests as self.params and one can query for the current parameters:

self.params.get(key="my_key", path="/some/location/*",
                default="default_value")

Let’s get back to Mux for a while. As mentioned earlier, it’s a database which allows storing multiple variants of test parameters. To fill the database, you can use several commands.

1. --mux-inject - injects directly [path:]key:node values from the cmdline (see avocado multiplex -h)
2. yaml_to_mux plugin - allows parsing yaml files into the Mux database (see yaml_to_mux plugin)
3. Custom plugin using the simple Mux API (see mux_api)

Mux API¶

The Mux object is defined in avocado/core/multiplexer.py, is always instantiated in avocado.core.parser.py and always available in args.mux. The basic workflow is:

1. Initialize Mux in args.mux
2. Fill it with data (plugins or job)
3. Multiplex it (in job)
4. Iterate through all variants on all job’s tests

Once the Mux object is multiplexed (3), it’s restricted to alter the data (2) to avoid changing the already produced data.

The main API needed for your plugins, which we are going to try keeping as stable as possible is:

• mux.is_parsed() - to find out whether the object was already parsed
• data_inject(key, value, path=None) - to inject key/value pairs optionaly to a given path (by default ‘/’)
• data_merge(tree) - to merge avocado.core.tree.TreeNode-like tree into the database.

Given these you should be able to implement any kind of parser or params feeder, should you require one. We favor yaml and therefor we implemented a yaml_to_mux plugin which can be found in avocado/plugins/yaml_to_mux.py and on it we also describe the way Mux works: yaml_to_mux plugin

 1  hw:
 2      cpu: !mux
 3          intel:
 4              cpu_CFLAGS: '-march=core2'
 5          amd:
 6              cpu_CFLAGS: '-march=athlon64'
 7          arm:
 8              cpu_CFLAGS: '-mabi=apcs-gnu -march=armv8-a -mtune=arm8'
 9      disk: !mux
10          scsi:
11              disk_type: 'scsi'
12          virtio:
13              disk_type: 'virtio'
14  distro: !mux
15      fedora:
16          init: 'systemd'
17      mint:
18          init: 'systemv'
19  env: !mux
20      debug:
21          opt_CFLAGS: '-O0 -g'
22      prod:
23          opt_CFLAGS: '-O2'

Nodes¶

They define context of the key=>value pairs allowing us to easily identify for what this values might be used for and also it makes possible to define multiple values of the same keys with different scope.

Due to their purpose the YAML automatic type conversion for nodes names is disabled, so the value of node name is always as written in the yaml file (unlike values, where yes converts to True and such).

Nodes are organized in parent-child relationship and together they create a tree. To view this structure use avocado multiplex --tree -m <file>:

┗━━ run
     ┣━━ hw
     ┃    ┣━━ cpu
     ┃    ┃    ╠══ intel
     ┃    ┃    ╠══ amd
     ┃    ┃    ╚══ arm
     ┃    ┗━━ disk
     ┃         ╠══ scsi
     ┃         ╚══ virtio
     ┣━━ distro
     ┃    ╠══ fedora
     ┃    ╚══ mint
     ┗━━ env
          ╠══ debug
          ╚══ prod

You can see that hw has 2 children cpu and disk. All parameters defined in parent node are inherited to children and extended/overwritten by their values up to the leaf nodes. The leaf nodes (intel, amd, arm, scsi, ...) are the most important as after multiplexation they form the parameters available in tests.

Keys and Values¶

Every value other than dict (4,6,8,11) is used as value of the antecedent node.

Each node can define key/value pairs (lines 4,6,8,11,...). Additionally each children node inherits values of it’s parent and the result is called node environment.

Given the node structure bellow:

devtools:
    compiler: 'cc'
    flags:
        - '-O2'
    debug: '-g'
    fedora:
        compiler: 'gcc'
        flags:
            - '-Wall'
    osx:
        compiler: 'clang'
        flags:
            - '-arch i386'
            - '-arch x86_64'

And the rules defined as:

• Scalar values (Booleans, Numbers and Strings) are overwritten by walking from the root until the final node.
• Lists are appended (to the tail) whenever we walk from the root to the final node.

The environment created for the nodes fedora and osx are:

• Node //devtools/fedora environment compiler: 'gcc', flags: ['-O2', '-Wall']
• Node //devtools/osx environment compiler: 'clang', flags: ['-O2', '-arch i386', '-arch x86_64']

Note that due to different usage of key and values in environment we disabled the automatic value conversion for keys while keeping it enabled for values. This means that the value can be of any YAML supported value, eg. bool, None, list or custom type, while the key is always string.

Variants¶

In the end all leaves are gathered and turned into parameters, more specifically into AvocadoParams:

setup:
    graphic:
        user: "guest"
        password: "pass"
    text:
        user: "root"
        password: "123456"

produces [graphic, text]. In the test code you’ll be able to query only those leaves. Intermediary or root nodes are available.

The example above generates a single test execution with parameters separated by path. But the most powerful multiplexer feature is that it can generate multiple variants. To do that you need to tag a node whose children are ment to be multiplexed. Effectively it returns only leaves of one child at the time.In order to generate all possible variants multiplexer creates cartesian product of all of these variants:

cpu: !mux
    intel:
    amd:
    arm:
fmt: !mux
    qcow2:
    raw:

Produces 6 variants:

/cpu/intel, /fmt/qcow2
/cpu/intel, /fmt/raw
...
/cpu/arm, /fmt/raw

The !mux evaluation is recursive so one variant can expand to multiple ones:

fmt: !mux
    qcow: !mux
        2:
        2v3:
    raw:

Results in:

/fmt/qcow2/2
/fmt/qcow2/2v3
/raw

Resolution order¶

You can see that only leaves are part of the test parameters. It might happen that some of these leaves contain different values of the same key. Then you need to make sure your queries separate them by different paths. When the path matches multiple results with different origin, an exception is raised as it’s impossible to guess which key was originally intended.

To avoid these problems it’s recommended to use unique names in test parameters if possible, to avoid the mentioned clashes. It also makes it easier to extend or mix multiple YAML files for a test.

For multiplex YAML files that are part of a framework, contain default configurations, or serve as plugin configurations and other advanced setups it is possible and commonly desirable to use non-unique names. But always keep those points in mind and provide sensible paths.

Multiplexer also supports default paths. By default it’s /run/* but it can be overridden by --mux-path, which accepts multiple arguments. What it does it splits leaves by the provided paths. Each query goes one by one through those sub-trees and first one to hit the match returns the result. It might not solve all problems, but it can help to combine existing YAML files with your ones:

qa:         # large and complex read-only file, content injected into /qa
    tests:
        timeout: 10
    ...
my_variants: !mux        # your YAML file injected into /my_variants
    short:
        timeout: 1
    long:
        timeout: 1000

You want to use an existing test which uses params.get('timeout', '*'). Then you can use --mux-path '/my_variants/*' '/qa/*' and it’ll first look in your variants. If no matches are found, then it would proceed to /qa/*

Keep in mind that only slices defined in mux-path are taken into account for relative paths (the ones starting with *)

Injecting files¶

You can run any test with any YAML file by:

avocado run sleeptest.py --mux-yaml file.yaml

This puts the content of file.yaml into /run location, which as mentioned in previous section, is the default mux-path path. For most simple cases this is the expected behavior as your files are available in the default path and you can safely use params.get(key).

When you need to put a file into a different location, for example when you have two files and you don’t want the content to be merged into a single place becomming effectively a single blob, you can do that by giving a name to your yaml file:

avocado run sleeptest.py --mux-yaml duration:duration.yaml

The content of duration.yaml is injected into /run/duration. Still when keys from other files don’t clash, you can use params.get(key) and retrieve from this location as it’s in the default path, only extended by the duration intermediary node. Another benefit is you can merge or separate multiple files by using the same or different name, or even a complex (relative) path.

Last but not least, advanced users can inject the file into whatever location they prefer by:

avocado run sleeptest.py --mux-yaml /my/variants/duration:duration.yaml

Simple params.get(key) won’t look in this location, which might be the intention of the test writer. There are several ways to access the values:

• absolute location params.get(key, '/my/variants/duration')
• absolute location with wildcards params.get(key, '/my/*) (or /*/duration/*...)
• set the mux-path avocado run ... --mux-path /my/* and use relative path

It’s recommended to use the simple injection for single YAML files, relative injection for multiple simple YAML files and the last option is for very advanced setups when you either can’t modify the YAML files and you need to specify custom resoltion order or you are specifying non-test parameters, for example parameters for your plugin, which you need to separate from the test parameters.

Multiple files¶

You can provide multiple files. In such scenario final tree is a combination of the provided files where later nodes with the same name override values of the preceding corresponding node. New nodes are appended as new children:

file-1.yaml:
    debug:
        CFLAGS: '-O0 -g'
    prod:
        CFLAGS: '-O2'

file-2.yaml:
    prod:
        CFLAGS: '-Os'
    fast:
        CFLAGS: '-Ofast'

results in:

debug:
    CFLAGS: '-O0 -g'
prod:
    CFLAGS: '-Os'       # overriden
fast:
    CFLAGS: '-Ofast'    # appended

It’s also possible to include existing file into another a given node in another file. This is done by the !include : $path directive:

os:
    fedora:
        !include : fedora.yaml
    gentoo:
        !include : gentoo.yaml

The file location can be either absolute path or relative path to the YAML file where the !include is called (even when it’s nested).

Whole file is merged into the node where it’s defined.

Advanced YAML tags¶

There are additional features related to YAML files. Most of them require values separated by ":". Again, in all such cases it’s mandatory to add a white space (" ") between the tag and the ":", otherwise ":" is part of the tag name and the parsing fails.

!include¶

Includes other file and injects it into the node it’s specified in:

my_other_file:
    !include : other.yaml

The content of /my_other_file would be parsed from the other.yaml. It’s the hardcoded equivalent of the -m $using:$path.

Relative paths start from the original file’s directory.

!using¶

Prepends path to the node it’s defined in:

!using : /foo
bar:
    !using : baz

bar is put into baz becoming /baz/bar and everything is put into /foo. So the final path of bar is /foo/baz/bar.

!remove_node¶

Removes node if it existed during the merge. It can be used to extend incompatible YAML files:

os:
    fedora:
    windows:
        3.11:
        95:
os:
    !remove_node : windows
    windows:
        win3.11:
        win95:

Removes the windows node from structure. It’s different from filter-out as it really removes the node (and all children) from the tree and it can be replaced by you new structure as shown in the example. It removes windows with all children and then replaces this structure with slightly modified version.

As !remove_node is processed during merge, when you reverse the order, windows is not removed and you end-up with /windows/{win3.11,win95,3.11,95} nodes.

!remove_value¶

It’s similar to !remove_node only with values.

!mux¶

Children of this node will be multiplexed. This means that in first variant it’ll return leaves of the first child, in second the leaves of the second child, etc. Example is in section Variants

Complete example¶

Let’s take a second look at the first example:

 1    hw:
 2        cpu: !mux
 3            intel:
 4                cpu_CFLAGS: '-march=core2'
 5            amd:
 6                cpu_CFLAGS: '-march=athlon64'
 7            arm:
 8                cpu_CFLAGS: '-mabi=apcs-gnu -march=armv8-a -mtune=arm8'
 9        disk: !mux
10            scsi:
11                disk_type: 'scsi'
12            virtio:
13                disk_type: 'virtio'
14    distro: !mux
15        fedora:
16            init: 'systemd'
17        mint:
18            init: 'systemv'
19    env: !mux
20        debug:
21            opt_CFLAGS: '-O0 -g'
22        prod:
23            opt_CFLAGS: '-O2'

After filters are applied (simply removes non-matching variants), leaves are gathered and all variants are generated:

$ avocado multiplex -m examples/mux-environment.yaml
Variants generated:
Variant 1:    /hw/cpu/intel, /hw/disk/scsi, /distro/fedora, /env/debug
Variant 2:    /hw/cpu/intel, /hw/disk/scsi, /distro/fedora, /env/prod
Variant 3:    /hw/cpu/intel, /hw/disk/scsi, /distro/mint, /env/debug
Variant 4:    /hw/cpu/intel, /hw/disk/scsi, /distro/mint, /env/prod
Variant 5:    /hw/cpu/intel, /hw/disk/virtio, /distro/fedora, /env/debug
Variant 6:    /hw/cpu/intel, /hw/disk/virtio, /distro/fedora, /env/prod
Variant 7:    /hw/cpu/intel, /hw/disk/virtio, /distro/mint, /env/debug
Variant 8:    /hw/cpu/intel, /hw/disk/virtio, /distro/mint, /env/prod
Variant 9:    /hw/cpu/amd, /hw/disk/scsi, /distro/fedora, /env/debug
Variant 10:    /hw/cpu/amd, /hw/disk/scsi, /distro/fedora, /env/prod
Variant 11:    /hw/cpu/amd, /hw/disk/scsi, /distro/mint, /env/debug
Variant 12:    /hw/cpu/amd, /hw/disk/scsi, /distro/mint, /env/prod
Variant 13:    /hw/cpu/amd, /hw/disk/virtio, /distro/fedora, /env/debug
Variant 14:    /hw/cpu/amd, /hw/disk/virtio, /distro/fedora, /env/prod
Variant 15:    /hw/cpu/amd, /hw/disk/virtio, /distro/mint, /env/debug
Variant 16:    /hw/cpu/amd, /hw/disk/virtio, /distro/mint, /env/prod
Variant 17:    /hw/cpu/arm, /hw/disk/scsi, /distro/fedora, /env/debug
Variant 18:    /hw/cpu/arm, /hw/disk/scsi, /distro/fedora, /env/prod
Variant 19:    /hw/cpu/arm, /hw/disk/scsi, /distro/mint, /env/debug
Variant 20:    /hw/cpu/arm, /hw/disk/scsi, /distro/mint, /env/prod
Variant 21:    /hw/cpu/arm, /hw/disk/virtio, /distro/fedora, /env/debug
Variant 22:    /hw/cpu/arm, /hw/disk/virtio, /distro/fedora, /env/prod
Variant 23:    /hw/cpu/arm, /hw/disk/virtio, /distro/mint, /env/debug
Variant 24:    /hw/cpu/arm, /hw/disk/virtio, /distro/mint, /env/prod

Where the first variant contains:

/hw/cpu/intel/  => cpu_CFLAGS: -march=core2
/hw/disk/       => disk_type: scsi
/distro/fedora/ => init: systemd
/env/debug/     => opt_CFLAGS: -O0 -g

The second one:

/hw/cpu/intel/  => cpu_CFLAGS: -march=core2
/hw/disk/       => disk_type: scsi
/distro/fedora/ => init: systemd
/env/prod/      => opt_CFLAGS: -O2

From this example you can see that querying for /env/debug works only in the first variant, but returns nothing in the second variant. Keep this in mind and when you use the !mux flag always query for the pre-mux path, /env/* in this example.

Running Tests on a Remote Host¶

Avocado lets you run tests directly in a remote machine with SSH connection, provided that you properly set it up by installing Avocado in it.

You can check if this feature (a plugin) is enabled by running:

$ avocado plugins
...
remote  Remote machine options for 'run' subcommand
...

Assuming this feature is enabled, you should be able to pass the following options when using the run command in the Avocado command line tool:

--remote-hostname REMOTE_HOSTNAME
                      Specify the hostname to login on remote machine
--remote-port REMOTE_PORT
                      Specify the port number to login on remote machine.
                      Default: 22
--remote-username REMOTE_USERNAME
                      Specify the username to login on remote machine
--remote-password REMOTE_PASSWORD
                      Specify the password to login on remote machine

From these options, you are normally going to use –remote-hostname and –remote-username in case you did set up your VM with password-less SSH connection (through SSH keys).

$ scripts/avocado run --remote-hostname 192.168.122.30 --remote-username fedora examples/tests/sleeptest.py examples/tests/failtest.py
REMOTE LOGIN  : fedora@192.168.122.30:22
JOB ID    : 60ddd718e7d7bb679f258920ce3c39ce73cb9779
JOB LOG   : $HOME/avocado/job-results/job-2014-10-23T11.45-a329461/job.log
TESTS     : 2
 (1/2) examples/tests/sleeptest.py: PASS (1.00 s)
 (2/2) examples/tests/failtest.py: FAIL (0.00 s)
RESULTS    : PASS 1 | ERROR 0 | FAIL 1 | SKIP 0 | WARN 0 | INTERRUPT 0
TESTS TIME : 1.01 s

Running Tests on a Virtual Machine¶

Sometimes you don’t want to run a given test directly in your own machine (maybe the test is dangerous, maybe you need to run it in another Linux distribution, so on and so forth).

For those scenarios, Avocado lets you run tests directly in VMs defined as libvirt domains in your system, provided that you properly set them up.

You can check if this feature (a plugin) is enabled by running:

$ avocado plugins
...
vm      Virtual Machine options for 'run' subcommand
...

Assuming this feature is enabled, you should be able to pass the following options when using the run command in the Avocado command line tool:

--vm                  Run tests on Virtual Machine
--vm-hypervisor-uri VM_HYPERVISOR_URI
                      Specify hypervisor URI driver connection
--vm-domain VM_DOMAIN
                      Specify domain name (Virtual Machine name)
--vm-hostname VM_HOSTNAME
                      Specify VM hostname to login. By default Avocado
                      attempts to automatically find the VM IP address.
--vm-username VM_USERNAME
                      Specify the username to login on VM
--vm-password VM_PASSWORD
                      Specify the password to login on VM
--vm-cleanup          Restore VM to a previous state, before running the
                      tests

From these options, you are normally going to use –vm-domain, –vm-hostname and –vm-username in case you did set up your VM with password-less SSH connection (through SSH keys).

If your VM has the qemu-guest-agent installed, you can skip the --vm-hostname option. Avocado will then probe the VM IP from the agent.

$ scripts/avocado run --vm-domain fedora20 --vm-username autotest --vm examples/tests/sleeptest.py examples/tests/failtest.py
VM DOMAIN : fedora20
VM LOGIN  : autotest@192.168.122.30
JOB ID    : 60ddd718e7d7bb679f258920ce3c39ce73cb9779
JOB LOG   : $HOME/avocado/job-results/job-2014-09-16T18.41-60ddd71/job.log
TESTS     : 2
 (1/2) examples/tests/sleeptest.py:SleepTest.test: PASS (1.00 s)
 (2/2) examples/tests/failtest.py:FailTest.test: FAIL (0.01 s)
RESULTS    : PASS 1 | ERROR 0 | FAIL 1 | SKIP 0 | WARN 0 | INTERRUPT 0
TESTS TIME : 1.01 s

Running Tests on a Docker container¶

Avocado also lets you run tests on a Docker container, starting and cleaning it up automatically with every execution.

You can check if this feature (a plugin) is enabled by running:

$ avocado plugins
...
docker  Run tests inside docker container
...

$ sudo docker pull ldoktor/fedora-avocado
Using default tag: latest
Trying to pull repository docker.io/ldoktor/fedora-avocado ...
latest: Pulling from docker.io/ldoktor/fedora-avocado
...
Status: Downloaded newer image for docker.io/ldoktor/fedora-avocado:latest

$ git clone github.com/avocado-framework/avocado.git avocado.git

$ cd avocado.git
$ patch -p1 < MY_PATCH

$ docker build -t fedora-avocado-custom -f contrib/docker/Dockerfile.fedora .

$ avocado run --docker fedora-avocado-custom examples/tests/passtest.py

$ avocado run passtest.py warntest.py failtest.py --docker ldoktor/fedora-avocado --docker-cmd "sudo docker"
DOCKER     : Container id '4bcbcd69801211501a0dde5926c0282a9630adbe29ecb17a21ef04f024366943'
JOB ID     : db309f5daba562235834f97cad5f4458e3fe6e32
JOB LOG    : $HOME/avocado/job-results/job-2016-07-25T08.01-db309f5/job.log
TESTS      : 3
 (1/3) /avocado_remote_test_dir/$HOME/passtest.py:PassTest.test: PASS (0.00 s)
 (2/3) /avocado_remote_test_dir/$HOME/warntest.py:WarnTest.test: WARN (0.00 s)
 (3/3) /avocado_remote_test_dir/$HOME/failtest.py:FailTest.test: FAIL (0.00 s)
RESULTS    : PASS 1 | ERROR 0 | FAIL 1 | SKIP 0 | WARN 1 | INTERRUPT 0
TESTS TIME : 0.00 s
JOB HTML   : $HOME/avocado/job-results/job-2016-07-25T08.01-db309f5/html/results.html

Environment Variables¶

Running remote instances os Avocado, for example using remote or vm plugins, the remote environment has a different set of environment variables. If you want to make available remotely variables that are available in the local environment, you can use the run option –env-keep. See the example below:

$ export MYVAR1=foobar
$ env MYVAR2=foobar2 avocado run passtest.py --env-keep MYVAR1,MYVAR2 --remote-hostname 192.168.122.30 --remote-username fedora

By doing that, both MYVAR1 and MYVAR2 will be available in remote environment.

Transparent Execution of Executables¶

This feature adds a few command line options to the Avocado run command:

$ avocado run --help
...
GNU Debugger support:

  --gdb-run-bin EXECUTABLE[:BREAKPOINT]
                        Run a given executable inside the GNU debugger,
                        pausing at a given breakpoint (defaults to "main")
  --gdb-prerun-commands EXECUTABLE:COMMANDS
                        After loading an executable in GDB, but before
                        actually running it, execute the GDB commands in the
                        given file. EXECUTABLE is optional, if omitted
                        COMMANDS will apply to all executables
  --gdb-coredump {on,off}
                        Automatically generate a core dump when the inferior
                        process received a fatal signal such as SIGSEGV or
                        SIGABRT
...

To get started you want to use --gdb-run-bin, as shown in the example bellow.

avocado run --gdb-run-bin=doublefree: examples/tests/doublefree_nasty.py --gdb-prerun-commands examples/tests/doublefree_nasty.py.data/gdb_pre --mux-yaml examples/tests/doublefree_nasty.py.data/iterations.yaml

from avocado import Test
from avocado.utils import process

class HelloOutputTest(Test):

    def test(self):
        result = process.run("/path/to/hello", ignore_status=True)
        self.assertIn("hello\n", result.stdout)

avocado.utils.gdb APIs¶

Avocado’s GDB module, provides three main classes that lets a test writer interact with a gdb process, a gdbserver process and also use the GDB remote protocol for interaction with a remote target.

Please refer to avocado.utils.gdb for more information.

def test(self):
    """
    Execute 'print_variable'.
    """
    path = os.path.join(self.srcdir, 'print_variable')
    app = gdb.GDB()
    app.set_file(path)
    app.set_break(6)
    app.run()
    self.log.info("\n".join(app.read_until_break()))
    app.cmd("set variable a = 0xff")
    app.cmd("c")
    out = "\n".join(app.read_until_break())
    self.log.info(out)
    app.exit()
    self.assertIn("MY VARIABLE 'A' IS: ff", out)

Usage¶

This feature is implemented as a plugin, that adds the –wrapper option to the Avocado run command. For a detailed explanation, please consult the Avocado man page.

Example of a transparent way of running strace as a wrapper:

#!/bin/sh
exec strace -ff -o $AVOCADO_TEST_LOGDIR/strace.log -- $@

To have all programs started by test.py wrapped with ~/bin/my-wrapper.sh:

$ scripts/avocado run --wrapper ~/bin/my-wrapper.sh tests/test.py

To have only my-binary wrapped with ~/bin/my-wrapper.sh:

$ scripts/avocado run --wrapper ~/bin/my-wrapper.sh:*my-binary tests/test.py

Caveats¶

• It is not possible to debug with GDB (–gdb-run-bin) and use wrappers (–wrapper) at the same time. These two options are mutually exclusive.
• You can only set one (global) wrapper. If you need functionality present in two wrappers, you have to combine those into a single wrapper script.
• Only executables that are run with the avocado.utils.process APIs (and other API modules that make use of it, like mod:avocado.utils.build) are affected by this feature.

Listing plugins¶

The avocado command line tool has a builtin plugins command that lets you list available plugins. The usage is pretty simple:

$ avocado plugins
Plugins that add new commands (avocado.plugins.cli.cmd):
exec-path Returns path to avocado bash libraries and exits.
run       Run one or more tests (native test, test alias, binary or script)
sysinfo   Collect system information
...
Plugins that add new options to commands (avocado.plugins.cli):
remote  Remote machine options for 'run' subcommand
journal Journal options for the 'run' subcommand
...

Since plugins are (usually small) bundles of Python code, they may fail to load if the Python code is broken for any reason. Example:

$ avocado plugins
Failed to load plugin from module "avocado.plugins.exec_path": ImportError('No module named foo',)
Plugins that add new commands (avocado.plugins.cli.cmd):
run       Run one or more tests (native test, test alias, binary or script)
sysinfo   Collect system information
...

Writing a plugin¶

What better way to understand how an Avocado plugin works than creating one? Let’s use another old time favorite for that, the “Print hello world” theme.

from avocado.plugins.base import CLICmd


class HelloWorld(CLICmd):

    name = 'hello'
    description = 'The classical Hello World! plugin example.'

    def run(self, args):
        print(self.description)

setup(name='mypluginpack',
...
entry_points={
   'avocado.plugins.cli': [
      'hello = mypluginpack.hello:HelloWorld',
   ]
}
...

'avocado.plugins.job.prepost': [
   'jobscripts = avocado.plugins.jobscripts:JobScripts'
]

[plugins]
disable = ['cli.hello', 'job.prepost.jobscripts']

Job, test and identifiers¶

$ avocado run sleeptest.py
JOB ID     : 49ec339a6cca73397be21866453985f88713ac34
...

JOB LOG    : $HOME/avocado/job-results/job-2015-06-10T10.44-49ec339/job.log

<unique-id>-<test-name>[;<variant-id>]

'/bin/true'
'/bin/grep foobar /etc/passwd'
'passtest.py:Passtest.test'
'file:///tmp/passtest.py:Passtest.test'
'multiple_tests.py:MultipleTests.test_hello'
'type_specific.io-github-autotest-qemu.systemtap_tracing.qemu.qemu_free'

Test Types¶

Avocado at its simplest configuration can run two different types of tests [1]. You can mix and match those in a single job.

Test Statuses¶

Avocado sticks to the following definitions of test statuses:

• `PASS`: The test passed, which means all conditions being tested have passed.
• `FAIL`: The test failed, which means at least one condition being tested has failed. Ideally, it should mean a problem in the software being tested has been found.
• `ERROR`: An error happened during the test execution. This can happen, for example, if there’s a bug in the test runner, in its libraries or if a resource breaks unexpectedly. Uncaught exceptions in the test code will also result in this status.
• `SKIP`: The test runner decided a requested test should not be run. This can happen, for example, due to missing requirements in the test environment or when there’s a job timeout.

Libraries and APIs¶

The Avocado libraries and its APIs are a big part of what Avocado is.

But, to avoid having any issues you should understand what parts of the Avocado libraries are intended for test writers and their respective API stability promises.

Test Resolution¶

When you use the Avocado runner, frequently you’ll provide paths to files, that will be inspected, and acted upon depending on their contents. The diagram below shows how Avocado analyzes a file and decides what to do with it:

It’s important to note that the inspection mechanism is safe (that is, python classes and files are not actually loaded and executed on discovery and inspection stage). Due to the fact Avocado doesn’t actually load the code and classes, the introspection is simple and will not catch things like buggy test modules, missing imports and miscellaneous bugs in the code you want to list or run. We recommend only running tests from sources you trust, use of static checking and reviews in your test development process.

Due to the simple test inspection mechanism, avocado will not recognize test classes that inherit from a class derived from avocado.Test. Please refer to the Writing Avocado Tests documentation on how to use the tags functionality to mark derived classes as avocado test classes.

Results Specification¶

On a machine that executed tests, job results are available under [job-results]/job-[timestamp]-[short job ID], where logdir is the configured Avocado logs directory (see the data dir plugin), and the directory name includes a timestamp, such as job-2014-08-12T15.44-565e8de. A typical results directory structure can be seen below

$HOME/avocado/job-results/job-2014-08-13T00.45-4a92bc0/
├── id
├── jobdata
│   ├── args
│   ├── cmdline
│   ├── config
│   ├── multiplex
│   ├── pwd
│   └── urls
├── job.log
├── results.json
├── results.xml
├── sysinfo
│   ├── post
│   │   ├── brctl_show
│   │   ├── cmdline
│   │   ├── cpuinfo
│   │   ├── current_clocksource
│   │   ├── df_-mP
│   │   ├── dmesg_-c
│   │   ├── dmidecode
│   │   ├── fdisk_-l
│   │   ├── gcc_--version
│   │   ├── hostname
│   │   ├── ifconfig_-a
│   │   ├── interrupts
│   │   ├── ip_link
│   │   ├── ld_--version
│   │   ├── lscpu
│   │   ├── lspci_-vvnn
│   │   ├── meminfo
│   │   ├── modules
│   │   ├── mount
│   │   ├── mounts
│   │   ├── numactl_--hardware_show
│   │   ├── partitions
│   │   ├── scaling_governor
│   │   ├── uname_-a
│   │   ├── uptime
│   │   └── version
│   ├── pre
│   │   ├── brctl_show
│   │   ├── cmdline
│   │   ├── cpuinfo
│   │   ├── current_clocksource
│   │   ├── df_-mP
│   │   ├── dmesg_-c
│   │   ├── dmidecode
│   │   ├── fdisk_-l
│   │   ├── gcc_--version
│   │   ├── hostname
│   │   ├── ifconfig_-a
│   │   ├── interrupts
│   │   ├── ip_link
│   │   ├── ld_--version
│   │   ├── lscpu
│   │   ├── lspci_-vvnn
│   │   ├── meminfo
│   │   ├── modules
│   │   ├── mount
│   │   ├── mounts
│   │   ├── numactl_--hardware_show
│   │   ├── partitions
│   │   ├── scaling_governor
│   │   ├── uname_-a
│   │   ├── uptime
│   │   └── version
│   └── profile
└── test-results
    └── tests
        ├── sleeptest.py.1
        │   ├── data
        │   ├── debug.log
        │   └── sysinfo
        │       ├── post
        │       └── pre
        ├── sleeptest.py.2
        │   ├── data
        │   ├── debug.log
        │   └── sysinfo
        │       ├── post
        │       └── pre
        └── sleeptest.py.3
            ├── data
            ├── debug.log
            └── sysinfo
                ├── post
                └── pre

22 directories, 65 files

From what you can see, the results dir has:

1. A human readable id in the top level, with the job SHA1.
2. A human readable job.log in the top level, with human readable logs of the task
3. Subdirectory jobdata, that contains machine readable data about the job.
4. A machine readable results.xml and results.json in the top level, with a summary of the job information in xUnit/json format.
5. A top level sysinfo dir, with sub directories pre, post and profile, that store sysinfo files pre/post/during job, respectively.
6. Subdirectory test-results, that contains a number of subdirectories (filesystem-friendly test ids). Those test ids represent instances of test execution results.

Job Pre and Post Scripts¶

Avocado ships with a plugin (installed by default) that allows running scripts before and after the actual execution of Jobs. A user can be sure that, when a given “pre” script is run, no test in that job has been run, and when the “post” scripts are run, all the tests in a given job have already finished running.

/etc/avocado/scripts/job

/etc/avocado/scripts/job/pre.d

/etc/avocado/scripts/job/post.d

[plugins.jobscripts]
pre = /my/custom/directory/for/pre/job/scripts/

[plugins.jobscripts]
post = /my/custom/directory/for/post/scripts/

Job Cleanup¶

It’s possible to register a callback function that will be called when all the tests have finished running. This effectively allows for a test job to clean some state it may have left behind.

At the moment, this feature is not intended to be used by test writers, but it’s seen as a feature for Avocado extensions to make use.

To register a callback function, your code should put a message in a very specific format in the “runner queue”. The Avocado test runner code will understand that this message contains a (serialized) function that will be called once all tests finish running.

Example:

from avocado import Test

def my_cleanup(path_to_file):
   if os.path.exists(path_to_file):
      os.unlink(path_to_file)

class MyCustomTest(Test):
...
   cleanup_file = '/tmp/my-custom-state'
   self.runner_queue.put({"func_at_exit": self.my_cleanup,
                          "args": (cleanup_file),
                          "once": True})
...

This results in the my_cleanup function being called with positional argument cleanup_file.

Because once was set to True, only one unique combination of function, positional arguments and keyword arguments will be registered, not matter how many times they’re attempted to be registered. For more information check avocado.utils.data_structures.CallbackRegister.register().

Hacking and Using Avocado¶

Since version 0.31.0, our plugin system requires Setuptools entry points to be registered. If you’re hacking on Avocado and want to use the same, possibly modified, source for running your tests and experiments, you may do so with one additional step:

$ make develop

On POSIX systems this will create an “egg link” to your original source tree under “$HOME/.local/lib/pythonX.Y/site-packages”. Then, on your original source tree, an “egg info” directory will be created, containing, among other things, the Setuptools entry points mentioned before. This works like a symlink, so you only need to run this once (unless you add a new entry-point, then you need to re-run it to make it available).

Avocado supports various plugins, which are distributed as separate projects, for example “avocado-vt” and “avocado-virt”. These also need to be deployed and linked in order to work properly with the avocado from sources (installed version works out of the box). To simplify this you can use make requirements-plugins from the main avocado project to install requirements of the plugins and make link to link and develop the plugins. The workflow could be:

$ cd $AVOCADO_PROJECTS_DIR
$ git clone $AVOCADO_GIT
$ git clone $AVOCADO_PROJECT2
$ # Add more projects
$ cd avocado    # go into the main avocado project dir
$ make requirements-plugins
$ make link

You should see the process and status for each directory.

Contact information¶

• Avocado-devel mailing list: https://www.redhat.com/mailman/listinfo/avocado-devel
• Avocado IRC channel: irc.oftc.net #avocado

Contributing to Avocado¶

Avocado uses github and the github pull request development model. You can find a primer on how to use github pull requests here. Every Pull Request you send will be automatically tested by Travis CI and review will take place in the Pull Request as well.

For people who don’t like the github development model, there is the option of sending the patches to the Mailing List, following a workflow more traditional in Open Source development communities. The patches will be reviewed in the Mailing List, should you opt for that. Then a maintainer will collect the patches, integrate them on a branch, and then those patches will be submitted as a github Pull Request. This process tries to ensure that every contributed patch goes through the CI jobs before it is considered good for inclusion.

$ git checkout master
$ git pull upstream master
$ git push

# To list branches along with time reference:
$ git for-each-ref --sort='-authordate:iso8601' --format=' %(authordate:iso8601)%09%(refname)' refs/heads
# To remove branches from your fork repository:
$ git push origin :my_old_branch

# Google for howto, but generally it works like this
$ gpg --gen-key  # defaults are usually fine (using expiration is recommended)
$ gpg --send-keys $YOUR_KEY    # to propagate the key to outer world

$ git config --global user.signingkey $YOUR_KEY

1. Login to github
2. Go to settings->SSH and GPG keys
3. Add New GPG key
4. run $(gpg -a --export $YOUR_EMAIL) in shell to see your key
5. paste the key there

# You can sign commits by using '-S'
$ git commit -S
# You can sign merges by using '-S'
$ git merge -S

Tests Repositories¶

We encourage you or your company to create public Avocado tests repositories so the community can also benefit of your tests. We will be pleased to advertise your repository here in our documentation.

List of known community and third party maintained repositories:

• https://github.com/avocado-framework-tests/avocado-misc-tests: Community maintained Avocado miscellaneous tests repository. There you will find, among others, performance tests like lmbench, stress, cpu tests like ebizzy and generic tests like ltp. Some of them were ported from Autotest Client Tests repository.
• https://github.com/scylladb/scylla-cluster-tests: Avocado tests for Scylla Clusters. Those tests can automatically create a scylla cluster, some loader machines and then run operations defined by the test writers, such as database workloads.

Interrupting test¶

In case you want to “pause” the running test, you can use SIGTSTP (ctrl+z) signal sent to the main avocado process. This signal is forwarded to test and it’s children processes. To resume testing you repeat the same signal.

Note: that the job/test timeouts are still enabled on stopped processes.

In tree utils¶

You can find handy utils in avocado.utils.debug:

from avocado.utils import debug
...
@debug.measure_duration
def your_function(...):

PERF: <function your_function at 0x29b17d0>: (0.1s, 11.3s)
PERF: <function your_function at 0x29b17d0>: (0.2s, 11.5s)

Line-profiler¶

You can measure line-by-line performance by using line_profiler. You can install it using pip:

pip install line_profiler

and then simply mark the desired function with @profile (no need to import it from anywhere). Then you execute:

kernprof -l -v ./scripts/avocado run ...

and when the process finishes you’ll see the profiling information. (sometimes the binary is called kernprof.py)

Remote debug with Eclipse¶

Eclipse is a nice debugging frontend which allows remote debugging. It’s very simple. The only thing you need is Eclipse with pydev plugin. Then you need to locate the pydevd path (usually $INSTALL_LOCATION/plugins/org.python.pydev_*/pysrc or ~/.eclipse/plugins/org.python.pydev_*/pysrc. Then you set the breakpoint by:

import sys
sys.path.append("$PYDEV_PATH")
import pydevd
pydevd.settrace("$IP_ADDR_OF_ECLIPSE_MACHINE")

Alternatively you can export PYTHONPATH=$PYDEV_PATH and use only last 2 lines.

Before you run the code, you need to start the Eclipse’s debug server. Switch to Debug perspective (you might need to open it first Window->Perspective->Open Perspective). Then start the server from Pydev->Start Debug Server.

Now whenever the pydev.settrace() code is executed, it contacts Eclipse debug server (port 8000 by default, don’t forget to open it) and you can easily continue in execution. This works on every remote machine which has access to your Eclipse’s port 8000 (you can override it).

Using Trello cards in Eclipse¶

Eclipse allows us to create tasks. They are pretty cool as you see the status (not started, started, current, done) and by switching tasks it automatically resumes where you previously finished (opened files, ...)

Avocado is planned using Trello, which is not yet supported by Eclipse. Anyway there is a way to at least get read-only list of your commits. This guide is based on https://docs.google.com/document/d/1jvmJcCStE6QkJ0z5ASddc3fNmJwhJPOFN7X9-GLyabM/ which didn’t work well with lables and descriptions. The only difference is you need to use Query Pattern:

\"url\":\"https://trello.com/[^/]*/[^/]*/({Id}[^\"]+)({Description})\"

Setup Trello key:

1. Create a Trello account
2. Get (developer_key) here: https://trello.com/1/appKey/generate
3. Get user_token from following address (replace key with your key): https://trello.com/1/authorize?key=$developer_key&name=Mylyn%20Tasks&expiration=never&response_type=token
4. Address with your assigned tasks (task_addr) is: https://trello.com/1/members/my/cards?key=developer_key&token=$user_token Open it in web browser and you should see [] or [$list_of_cards] without any passwords.

Configure Eclipse:

1. We’re going to need Web Templates, which are not yet upstream. We need to use incubator version.
2. Help->Install New Software...
3. -> Add
4. Name: Incubator
5. Location: http://download.eclipse.org/mylyn/incubator/3.10
6. -> OK
7. Select Mylyn Tasks Connector: Web Templates (Advanced) (Incubation) (use filter text to find it)
8. Install it (Next->Agree->Next...)
9. Restart Eclipse
10. Open the Mylyn Team Repositories Window->Show View->Other...->Mylyn->Team Repositories
11. Right click the Team Repositories and select New->Repository
12. Use Task Repository -> Next
13. Use Web Template (Advanced) -> Next
14. In the Properties for Task Repository dialog box, enter https://trello.com
15. In the Server field and give the repository a label (eg. Trello API).
16. In the Additional Settings section set applicationkey = $developer_key and userkey = $user_token.
17. In the Advanced Configuration set the Task URL to https://trello.com/c/
18. Set New Task URL to https://trello.com
19. Set the Query Request URL (no changes required): https://trello.com/1/members/my/cards?key=${applicationkey}&token=${userkey}
20. For the Query Pattern enter “url”:”https://trello.com/[^/]*/[^/]*/({Id}[^”]+)({Description})”
21. -> Finish

Create task query:

1. Create a query by opening the Mylyn Task List.
2. Right click the pane and select New Query.
3. Select Trello API as the repository.
4. -> Next
5. Enter the name of your query.
6. Expand the Advanced Configuration and make sure the Query Pattern is filled in
7. Press Preview to confirm that there are no errors.
8. Press Finish.
9. Trello tasks assigned to you will now appear in the Mylyn Task List.

Noy you can start using tasks by clicking the small bubble in front of the name. This closes all editors. Try openning some and then click the bubble again. They should get closed. When you click the bubble third time, it should resume all the open editors from before.

My usual workflow is:

1. git checkout $branch
2. Eclipse: select task
3. git commit ...
4. Eclipse: unselect task
5. git checkout $other_branch
6. Eclipse: select another_task

This way you always have all the files present and you can easily resume your work.

Bump the version number¶

Go through the avocado code base and update the release number. At the time of this writing, the diff looked like this:

diff --git a/avocado.spec b/avocado.spec
index eb910e8..21313ca 100644
--- a/avocado.spec
+++ b/avocado.spec
@@ -1,7 +1,7 @@
 Summary: Avocado Test Framework
 Name: avocado
-Version: 0.28.0
-Release: 2%{?dist}
+Version: 0.29.0
+Release: 0%{?dist}
 License: GPLv2
 Group: Development/Tools
 URL: http://avocado-framework.github.io/
@@ -104,6 +104,9 @@ examples of how to write tests on your own.
 %{_datadir}/avocado/wrappers

 %changelog
+* Wed Oct 7 2015 Lucas Meneghel Rodrigues <lmr@redhat.com> - 0.29.0-0
+- New upstream release 0.29.0
+
 * Wed Sep 16 2015 Lucas Meneghel Rodrigues <lmr@redhat.com> - 0.28.0-2
 - Add pystache, aexpect, psutil, sphinx and yum/dnf dependencies for functional/unittests

diff --git a/avocado/core/version.py b/avocado/core/version.py
index c927b19..a555af5 100755
--- a/avocado/core/version.py
+++ b/avocado/core/version.py
@@ -18,7 +18,7 @@ __all__ = ['MAJOR', 'MINOR', 'RELEASE', 'VERSION']


 MAJOR = 0
-MINOR = 28
+MINOR = 29
 RELEASE = 0

 VERSION = "%s.%s.%s" % (MAJOR, MINOR, RELEASE)
diff --git a/setup.cfg b/setup.cfg
index 76953b9..5cf90e9 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -1,6 +1,6 @@
 [metadata]
 name = avocado
-version = 0.28.0
+version = 0.29.0
 summary = Avocado Test Framework
 description-file =
     README.rst

You can find on git such commits that will help you get oriented for other repos.

Which repositories you should pay attention to¶

In general, a release of avocado includes taking a look and eventually release content in the following repositories:

• avocado
• avocado-vt

Tag all repositories¶

When everything is in good shape, commit the version changes and tag that commit in master with:

$ git tag -u $(GPG_ID) -s $(RELEASE) -m 'Avocado Release $(RELEASE)'

Then the tag should be pushed to the GIT repository with:

$ git push --tags

Build RPMs¶

Go to the source directory and do:

$ make rpm
...
+ exit 0

This should be all. It will build packages using mock, targeting your default configuration. That usually means the same platform you’re currently on.

Sign Packages¶

All the packages should be signed for safer public consumption. The process is, of course, dependent on the private keys, put is based on running:

$ rpm --resign

For more information look at the rpmsign(8) man page.

Upload packages to repository¶

The current distribution method is based on serving content over HTTP. That means that repository metadata is created locally and synchronized to the well know public Web server. A process similar to:

$ cd $REPO_ROOT && for DIR in epel-?-noarch fedora-??-noarch; \
do cd $DIR && createrepo -v . && cd ..; done;

Creates the repo metadata locally. Then a command similar to:

$ rsync -va $REPO_ROOT user@repo_web_server:/path

Is used to copy the content over.

Write release notes¶

Release notes give an idea of what has changed on a given development cycle. Good places to go for release notes are:

1. Git logs
2. Trello Cards (Look for the Done lists)
3. Github compare views: https://github.com/avocado-framework/avocado/compare/0.28.0...0.29.0

Go there and try to write a text that represents the changes that the release encompasses.

Upload package to PyPI¶

Users may also want to get Avocado from the PyPI repository, so please upload there as well. To help with the process, please run:

$ make pypi

And follow the URL and brief instructions given.

Send e-mails to avocado-devel and other places¶

Send the e-mail with the release notes to avocado-devel and virt-test-devel.

Module contents¶

avocado.main¶

alias of TestProgram

class avocado.Test(methodName='test', name=None, params=None, base_logdir=None, job=None, runner_queue=None)¶

Bases: unittest.case.TestCase

Base implementation for the test class.

You’ll inherit from this to write your own tests. Typically you’ll want to implement setUp(), test*() and tearDown() methods on your own tests.

Initializes the test.

Parameters:

methodName – Name of the main method to run. For the sake of compatibility with the original unittest class, you should not set this. name (avocado.core.test.TestName) – Pretty name of the test name. For normal tests, written with the avocado API, this should not be set. This is reserved for internal Avocado use, such as when running random executables as tests. base_logdir – Directory where test logs should go. If None provided, it’ll use avocado.data_dir.create_job_logs_dir(). job – The job that this test is part of.

Raises:

avocado.core.test.NameNotTestNameError

basedir¶

The directory where this test (when backed by a file) is located at

cache_dirs = None¶

datadir¶

Returns the path to the directory that contains test data files

default_params = {}¶

error(message=None)¶

Errors the currently running test.

After calling this method a test will be terminated and have its status as ERROR.

Parameters:	message (str) – an optional message that will be recorded in the logs

fail(message=None)¶

Fails the currently running test.

After calling this method a test will be terminated and have its status as FAIL.

Parameters:	message (str) – an optional message that will be recorded in the logs

fetch_asset(name, asset_hash=None, algorithm='sha1', locations=None, expire=None)¶

Method o call the utils.asset in order to fetch and asset file supporting hash check, caching and multiple locations.

Parameters:	name – the asset filename or URL asset_hash – asset hash (optional) algorithm – hash algorithm (optional, defaults to sha1) locations – list of URLs from where the asset can be fetched (optional) expire – time for the asset to expire
Raises:	EnvironmentError – When it fails to fetch the asset
Returns:	asset file local path

filename¶

Returns the name of the file (path) that holds the current test

get_state()¶

Serialize selected attributes representing the test state

Returns:	a dictionary containing relevant test state data
Return type:	dict

report_state()¶

Send the current test state to the test runner process

run_avocado()¶

Wraps the run method, for execution inside the avocado runner.

Result:	Unused param, compatibility with unittest.TestCase.

skip(message=None)¶

Skips the currently running test.

This method should only be called from a test’s setUp() method, not anywhere else, since by definition, if a test gets to be executed, it can’t be skipped anymore. If you call this method outside setUp(), avocado will mark your test status as ERROR, and instruct you to fix your test in the error message.

Parameters:	message (str) – an optional message that will be recorded in the logs

srcdir = None¶

workdir = None¶

avocado.fail_on(exceptions=None)¶

Fail the test when decorated function produces exception of the specified type.

(For example, our method may raise IndexError on tested software failure. We can either try/catch it or use this decorator instead)

Parameters:	exceptions – Tuple or single exception to be assumed as test fail [Exception]
Note:	self.error and self.skip behavior remains intact
Note:	To allow simple usage param “exceptions” must not be callable

Subpackages¶

Submodules¶

avocado.utils.archive module¶

Module to help extract and create compressed archives.

exception avocado.utils.archive.ArchiveException¶

Bases: exceptions.Exception

Base exception for all archive errors.

class avocado.utils.archive.ArchiveFile(filename, mode='r')¶

Bases: object

Class that represents an Archive file.

Archives are ZIP files or Tarballs.

Creates an instance of ArchiveFile.

Parameters:	filename – the archive file name. mode – file mode, r read, w write.

add(filename, arcname=None)¶

Add file to the archive.

Parameters:	filename – file to archive. arcname – alternative name for the file in the archive.

close()¶

Close archive.

extract(path='.')¶

Extract all files from the archive.

Parameters:	path – destination path.

list()¶

List files to the standard output.

classmethod open(filename, mode='r')¶

Creates an instance of ArchiveFile.

Parameters:	filename – the archive file name. mode – file mode, r read, w write.

avocado.utils.archive.compress(filename, path)¶

Compress files in an archive.

Parameters:	filename – archive file name. path – origin directory path to files to compress. No individual files allowed.

avocado.utils.archive.create(filename, path)¶

Compress files in an archive.

Parameters:	filename – archive file name. path – origin directory path to files to compress. No individual files allowed.

avocado.utils.archive.extract(filename, path)¶

Extract files from an archive.

Parameters:	filename – archive file name. path – destination path to extract to.

avocado.utils.archive.is_archive(filename)¶

Test if a given file is an archive.

Parameters:	filename – file to test.
Returns:	True if it is an archive.

avocado.utils.archive.uncompress(filename, path)¶

Extract files from an archive.

Parameters:	filename – archive file name. path – destination path to extract to.

avocado.utils.asset module¶

Asset fetcher from multiple locations

class avocado.utils.asset.Asset(name, asset_hash, algorithm, locations, cache_dirs, expire=None)¶

Bases: object

Try to fetch/verify an asset file from multiple locations.

Initialize the Asset() class.

Parameters:	name – the asset filename. url is also supported asset_hash – asset hash algorithm – hash algorithm locations – list of locations fetch asset from cache_dirs – list of cache directories expire – time in seconds for the asset to expire

fetch()¶

Fetches the asset. First tries to find the asset on the provided cache_dirs list. Then tries to download the asset from the locations list provided.

Raises:	EnvironmentError – When it fails to fetch the asset
Returns:	The path for the file on the cache directory.

avocado.utils.astring module¶

Operations with strings (conversion and sanitation).

The unusual name aims to avoid causing name clashes with the stdlib module string. Even with the dot notation, people may try to do things like

import string ... from avocado.utils import string

And not notice until their code starts failing.

avocado.utils.astring.bitlist_to_string(data)¶

Transform from bit list to ASCII string.

Parameters:	data – Bit list to be transformed

avocado.utils.astring.iter_tabular_output(matrix, header=None)¶

Generator for a pretty, aligned string representation of a nxm matrix.

This representation can be used to print any tabular data, such as database results. It works by scanning the lengths of each element in each column, and determining the format string dynamically.

Parameters:	matrix – Matrix representation (list with n rows of m elements). header – Optional tuple or list with header elements to be displayed.

avocado.utils.astring.shell_escape(command)¶

Escape special characters from a command so that it can be passed as a double quoted (” ”) string in a (ba)sh command.

Parameters:	command – the command string to escape.
Returns:	The escaped command string. The required englobing double quotes are NOT added and so should be added at some point by the caller.

See also: http://www.tldp.org/LDP/abs/html/escapingsection.html

avocado.utils.astring.string_safe_encode(string)¶

People tend to mix unicode streams with encoded strings. This function tries to replace any input with a valid utf-8 encoded ascii stream.

avocado.utils.astring.string_to_bitlist(data)¶

Transform from ASCII string to bit list.

Parameters:	data – String to be transformed

avocado.utils.astring.string_to_safe_path(string)¶

Convert string to a valid file/dir name. :param string: String to be converted :return: String which is safe to pass as a file/dir name (on recent fs)

avocado.utils.astring.strip_console_codes(output, custom_codes=None)¶

Remove the Linux console escape and control sequences from the console output. Make the output readable and can be used for result check. Now only remove some basic console codes using during boot up.

Parameters:	output (string) – The output from Linux console custom_codes – The codes added to the console codes which is not covered in the default codes
Returns:	the string without any special codes
Return type:	string

avocado.utils.astring.tabular_output(matrix, header=None)¶

Pretty, aligned string representation of a nxm matrix.

This representation can be used to print any tabular data, such as database results. It works by scanning the lengths of each element in each column, and determining the format string dynamically.

Parameters:	matrix – Matrix representation (list with n rows of m elements). header – Optional tuple or list with header elements to be displayed.
Returns:	String with the tabular output, lines separated by unix line feeds.
Return type:	str

avocado.utils.aurl module¶

URL related functions.

The strange name is to avoid accidental naming collisions in code.

avocado.utils.aurl.is_url(path)¶

Return True if path looks like an URL.

Parameters:	path – path to check.
Return type:	Boolean.

avocado.utils.build module¶

avocado.utils.build.make(path, make='make', env=None, extra_args='', ignore_status=False, allow_output_check='none')¶

Run make, adding MAKEOPTS to the list of options.

Parameters:

make – what make command name to use. env – dictionary with environment variables to be set before calling make (e.g.: CFLAGS). extra – extra command line arguments to pass to make. allow_output_check (str) – Whether to log the command stream outputs (stdout and stderr) of the make process in the test stream files. Valid values: ‘stdout’, for allowing only standard output, ‘stderr’, to allow only standard error, ‘all’, to allow both standard output and error, and ‘none’, to allow none to be recorded (default). The default here is ‘none’, because usually we don’t want to use the compilation output as a reference in tests.

Returns:

exit status of the make process

avocado.utils.build.run_make(path, make='make', env=None, extra_args='', ignore_status=False, allow_output_check='none')¶

Run make, adding MAKEOPTS to the list of options.

Parameters:

make – what make command name to use. env – dictionary with environment variables to be set before calling make (e.g.: CFLAGS). extra – extra command line arguments to pass to make. allow_output_check (str) – Whether to log the command stream outputs (stdout and stderr) of the make process in the test stream files. Valid values: ‘stdout’, for allowing only standard output, ‘stderr’, to allow only standard error, ‘all’, to allow both standard output and error, and ‘none’, to allow none to be recorded (default). The default here is ‘none’, because usually we don’t want to use the compilation output as a reference in tests.

Returns:

the make command result object

avocado.utils.cpu module¶

Get information from the current’s machine CPU.

avocado.utils.cpu.cpu_has_flags(flags)¶

Check if a list of flags are available on current CPU info

Parameters:	flags (list) – A list of cpu flags that must exists on the current CPU.
Returns:	bool True if all the flags were found or False if not
Return type:	list

avocado.utils.cpu.cpu_online_list()¶

Reports a list of indexes of the online cpus

avocado.utils.cpu.get_cpu_arch()¶

Work out which CPU architecture we’re running on

avocado.utils.cpu.get_cpu_vendor_name()¶

Get the current cpu vendor name

Returns:	string ‘intel’ or ‘amd’ or ‘power7’ depending on the current CPU architecture.
Return type:	string

avocado.utils.crypto module¶

avocado.utils.crypto.hash_file(filename, size=None, algorithm='md5')¶

Calculate the hash value of filename.

If size is not None, limit to first size bytes. Throw exception if something is wrong with filename. Can be also implemented with bash one-liner (assuming size%1024==0):

dd if=filename bs=1024 count=size/1024 | sha1sum -

Parameters:	filename – Path of the file that will have its hash calculated. method – Method used to calculate the hash. Supported methods: * md5 * sha1 size – If provided, hash only the first size bytes of the file.
Returns:	Hash of the file, if something goes wrong, return None.

avocado.utils.crypto.hash_wrapper(algorithm='md5', data=None)¶

Returns an hash object of data using either md5 or sha1 only.

Parameters:	input – Optional input string that will be used to update the hash.
Returns:	Hash object.

avocado.utils.data_factory module¶

Generate data useful for the avocado framework and tests themselves.

avocado.utils.data_factory.generate_random_string(length, ignore='!"#$%&\'()*+, -./:;<=>?@[\\]^_`{|}~', convert='')¶

Generate a random string using alphanumeric characters.

Parameters:	length (int) – Length of the string that will be generated. ignore (str) – Characters that will not include in generated string. convert (str) – Characters that need to be escaped (prepend “”).
Returns:	The generated random string.

avocado.utils.data_factory.make_dir_and_populate(basedir='/tmp')¶

Create a directory in basedir and populate with a number of files.

The files just have random text contents.

Parameters:	basedir (str) – Base directory where directory should be generated.
Returns:	Path of the dir created and populated.
Return type:	str

avocado.utils.data_structures module¶

This module contains handy classes that can be used inside avocado core code or plugins.

class avocado.utils.data_structures.Borg¶

Multiple instances of this class will share the same state.

This is considered a better design pattern in Python than more popular patterns, such as the Singleton. Inspired by Alex Martelli’s article mentioned below:

See:	http://www.aleax.it/5ep.html

class avocado.utils.data_structures.CallbackRegister(name, log)¶

Bases: object

Registers pickable functions to be executed later.

Parameters:	name – Human readable identifier of this register

register(func, args, kwargs, once=False)¶

Register function/args to be called on self.destroy() :param func: Pickable function :param args: Pickable positional arguments :param kwargs: Pickable keyword arguments :param once: Add unique (func,args,kwargs) combination only once

run()¶

Call all registered function

unregister(func, args, kwargs)¶

Unregister (func,args,kwargs) combination :param func: Pickable function :param args: Pickable positional arguments :param kwargs: Pickable keyword arguments

class avocado.utils.data_structures.LazyProperty(f_get)¶

Bases: object

Lazily instantiated property.

Use this decorator when you want to set a property that will only be evaluated the first time it’s accessed. Inspired by the discussion in the Stack Overflow thread below:

See:	http://stackoverflow.com/questions/15226721/

avocado.utils.data_structures.compare_matrices(matrix1, matrix2, threshold=0.05)¶

Compare 2 matrices nxm and return a matrix nxm with comparison data and stats. When the first columns match, they are considered as header and included in the results intact.

Parameters:	matrix1 – Reference Matrix of floats; first column could be header. matrix2 – Matrix that will be compared; first column could be header threshold – Any difference greater than this percent threshold will be reported.
Returns:	Matrix with the difference in comparison, number of improvements, number of regressions, total number of comparisons.

avocado.utils.data_structures.geometric_mean(values)¶

Evaluates the geometric mean for a list of numeric values. This implementation is slower but allows unlimited number of values. :param values: List with values. :return: Single value representing the geometric mean for the list values. :see: http://en.wikipedia.org/wiki/Geometric_mean

avocado.utils.data_structures.ordered_list_unique(object_list)¶

Returns an unique list of objects, with their original order preserved

avocado.utils.data_structures.time_to_seconds(time)¶

Convert time in minutes, hours and days to seconds. :param time: Time, optionally including the unit (i.e. ‘10d’)

avocado.utils.debug module¶

This file contains tools for (not only) Avocado developers.

avocado.utils.debug.log_calls(length=None, cls_name=None)¶

Use this as decorator to log the function call altogether with arguments. :param length: Max message length :param cls_name: Optional class name prefix

avocado.utils.debug.log_calls_class(length=None)¶

Use this as decorator to log the function methods’ calls. :param length: Max message length

avocado.utils.debug.measure_duration(func)¶

Use this as decorator to measure duration of the function execution. The output is “Function $name: ($current_duration, $accumulated_duration)”

avocado.utils.disk module¶

Disk utilities

avocado.utils.disk.freespace(path)¶

avocado.utils.distro module¶

This module provides the client facilities to detect the Linux Distribution it’s running under.

class avocado.utils.distro.LinuxDistro(name, version, release, arch)¶

Bases: object

Simple collection of information for a Linux Distribution

Initializes a new Linux Distro

Parameters:

name (str) – a short name that precisely distinguishes this Linux Distribution among all others. version (str) – the major version of the distribution. Usually this is a single number that denotes a large development cycle and support file. release (str) – the release or minor version of the distribution. Usually this is also a single number, that is often omitted or starts with a 0 when the major version is initially release. It’s often associated with a shorter development cycle that contains incremental a collection of improvements and fixes. arch (str) – the main target for this Linux Distribution. It’s common for some architectures to ship with packages for previous and still compatible architectures, such as it’s the case with Intel/AMD 64 bit architecture that support 32 bit code. In cases like this, this should be set to the 64 bit architecture name.

class avocado.utils.distro.Probe¶

Bases: object

Probes the machine and does it best to confirm it’s the right distro

CHECK_FILE = None¶

Points to a file that can determine if this machine is running a given Linux Distribution. This servers a first check that enables the extra checks to carry on.

CHECK_FILE_CONTAINS = None¶

Sets the content that should be checked on the file pointed to by CHECK_FILE_EXISTS. Leave it set to None (its default) to check only if the file exists, and not check its contents

CHECK_FILE_DISTRO_NAME = None¶

The name of the Linux Distribution to be returned if the file defined by CHECK_FILE_EXISTS exist.

CHECK_VERSION_REGEX = None¶

A regular expression that will be run on the file pointed to by CHECK_FILE_EXISTS

check_name_for_file()¶

Checks if this class will look for a file and return a distro

The conditions that must be true include the file that identifies the distro file being set (CHECK_FILE) and the name of the distro to be returned (CHECK_FILE_DISTRO_NAME)

check_name_for_file_contains()¶

Checks if this class will look for text on a file and return a distro

The conditions that must be true include the file that identifies the distro file being set (CHECK_FILE), the text to look for inside the distro file (CHECK_FILE_CONTAINS) and the name of the distro to be returned (CHECK_FILE_DISTRO_NAME)

check_release()¶

Checks if this has the conditions met to look for the release number

check_version()¶

Checks if this class will look for a regex in file and return a distro

get_distro()¶

Returns the LinuxDistro this probe detected

name_for_file()¶

Get the distro name if the CHECK_FILE is set and exists

name_for_file_contains()¶

Get the distro if the CHECK_FILE is set and has content

release()¶

Returns the release of the distro

version()¶

Returns the version of the distro

avocado.utils.distro.register_probe(probe_class)¶

Register a probe to be run during autodetection

avocado.utils.distro.detect()¶

Attempts to detect the Linux Distribution running on this machine

Returns:	the detected LinuxDistro or UNKNOWN_DISTRO
Return type:	LinuxDistro

avocado.utils.download module¶

Methods to download URLs and regular files.

avocado.utils.download.get_file(src, dst, permissions=None, hash_expected=None, hash_algorithm='md5', download_retries=1)¶

Gets a file from a source location, optionally using caching.

If no hash_expected is provided, simply download the file. Else, keep trying to download the file until download_failures exceeds download_retries or the hashes match.

If the hashes match, return dst. If download_failures exceeds download_retries, raise an EnvironmentError.

Parameters:	src – source path or URL. May be local or a remote file. dst – destination path. permissions – (optional) set access permissions. hash_expected – Hash string that we expect the file downloaded to have. hash_algorithm – Algorithm used to calculate the hash string (md5, sha1). download_retries – Number of times we are going to retry a failed download.
Raise:	EnvironmentError.
Returns:	destination path.

avocado.utils.download.url_download(url, filename, data=None, timeout=300)¶

Retrieve a file from given url.

Parameters:	url – source URL. filename – destination path. data – (optional) data to post. timeout – (optional) default timeout in seconds.
Returns:	None.

avocado.utils.download.url_download_interactive(url, output_file, title='', chunk_size=102400)¶

Interactively downloads a given file url to a given output file.

Parameters:	url (string) – URL for the file to be download output_file (string) – file name or absolute path on which to save the file to title (string) – optional title to go along the progress bar chunk_size (integer) – amount of data to read at a time

avocado.utils.download.url_open(url, data=None, timeout=5)¶

Wrapper to urllib2.urlopen() with timeout addition.

Parameters:	url – URL to open. data – (optional) data to post. timeout – (optional) default timeout in seconds.
Returns:	file-like object.
Raises:	URLError.

avocado.utils.filelock module¶

Utility for individual file access control implemented via PID lock files.

exception avocado.utils.filelock.AlreadyLocked¶

Bases: exceptions.Exception

class avocado.utils.filelock.FileLock(filename, timeout=0)¶

Bases: object

Creates an exclusive advisory lock for a file. All processes should use and honor the advisory locking scheme, but uncooperative processes are free to ignore the lock and access the file in any way they choose.

exception avocado.utils.filelock.LockFailed¶

Bases: exceptions.Exception

avocado.utils.gdb module¶

Module that provides communication with GDB via its GDB/MI interpreter

class avocado.utils.gdb.GDB(path='/usr/bin/gdb', *extra_args)¶

Bases: object

Wraps a GDB subprocess for easier manipulation

DEFAULT_BREAK = 'main'¶

REQUIRED_ARGS = ['--interpreter=mi', '--quiet']¶

cli_cmd(command)¶

Sends a cli command encoded as an MI command

Parameters:	command (str) – a regular GDB cli command
Returns:	a CommandResult instance
Return type:	CommandResult

cmd(command)¶

Sends a command and parses all lines until prompt is received

Parameters:	command (str) – the GDB command, hopefully in MI language
Returns:	a CommandResult instance
Return type:	CommandResult

cmd_exists(command)¶

Checks if a given command exists

Parameters:	command (str) – a GDB MI command, including the dash (-) prefix
Returns:	either True or False
Return type:	bool

connect(port)¶

Connects to a remote debugger (a gdbserver) at the given TCP port

This uses the “extended-remote” target type only

Parameters:	port (int) – the TCP port number
Returns:	a CommandResult instance
Return type:	CommandResult

del_break(number)¶

Deletes a breakpoint by its number

Parameters:	number (int) – the breakpoint number
Returns:	a CommandResult instance
Return type:	CommandResult

disconnect()¶

Disconnects from a remote debugger

Returns:	a CommandResult instance
Return type:	CommandResult

exit()¶

Exits the GDB application gracefully

Returns:	the result of subprocess.POpen.wait(), that is, a subprocess.POpen.returncode
Return type:	int or None

read_gdb_response(timeout=0.01, max_tries=100)¶

Read raw responses from GDB

Parameters:	timeout (float) – the amount of time to way between read attempts max_tries (int) – the maximum number of cycles to try to read until a response is obtained
Returns:	a string containing a raw response from GDB
Return type:	str

read_until_break(max_lines=100)¶

Read lines from GDB until a break condition is reached

Parameters:	max_lines (int) – the maximum number of lines to read
Returns:	a list of messages read
Return type:	list of str

run(args=[])¶

Runs the application inside the debugger

Parameters:	args (builtin.list) – the arguments to be passed to the binary as command line arguments
Returns:	a CommandResult instance
Return type:	CommandResult

send_gdb_command(command)¶

Send a raw command to the GNU debugger input

Parameters:	command (str) – the GDB command, hopefully in MI language
Returns:	None

set_break(location, ignore_error=False)¶

Sets a new breakpoint on the binary currently being debugged

Parameters:	location (str) – a breakpoint location expression as accepted by GDB
Returns:	a CommandResult instance
Return type:	CommandResult

set_file(path)¶

Sets the file that will be executed

Parameters:	path (str) – the path of the binary that will be executed
Returns:	a CommandResult instance
Return type:	CommandResult

class avocado.utils.gdb.GDBServer(path='/usr/bin/gdbserver', port=None, wait_until_running=True, *extra_args)¶

Bases: object

Wraps a gdbserver instance

Initializes a new gdbserver instance

Parameters:	path (str) – location of the gdbserver binary port (int) – tcp port number to listen on for incoming connections wait_until_running (bool) – wait until the gdbserver is running and accepting connections. It may take a little after the process is started and it is actually bound to the allocated port extra_args – optional extra arguments to be passed to gdbserver

INIT_TIMEOUT = 2.0¶

The time to optionally wait for the server to initialize itself and be ready to accept new connections

PORT_RANGE = (20000, 20999)¶

The range from which a port to GDB server will try to be allocated from

REQUIRED_ARGS = ['--multi']¶

The default arguments used when starting the GDB server process

exit(force=True)¶

Quits the gdb_server process

Most correct way of quitting the GDB server is by sending it a command. If no GDB client is connected, then we can try to connect to it and send a quit command. If this is not possible, we send it a signal and wait for it to finish.

Parameters:	force (bool) – if a forced exit (sending SIGTERM) should be attempted
Returns:	None

class avocado.utils.gdb.GDBRemote(host, port, no_ack_mode=True, extended_mode=True)¶

Bases: object

Initializes a new GDBRemote object.

A GDBRemote acts like a client that speaks the GDB remote protocol, documented at:

https://sourceware.org/gdb/current/onlinedocs/gdb/Remote-Protocol.html

Caveat: we currently do not support communicating with devices, only with TCP sockets. This limitation is basically due to the lack of use cases that justify an implementation, but not due to any technical shortcoming.

Parameters:	host (str) – the IP address or host name port (int) – the port number where the the remote GDB is listening on no_ack_mode (bool) – if the packet transmission confirmation mode should be disabled extended_mode – if the remote extended mode should be enabled

cmd(command_data, expected_response=None)¶

Sends a command data to a remote gdb server

Limitations: the current version does not deal with retransmissions.

Parameters:	command_data (str) – the remote command to send the the remote stub expected_response (str) – the (optional) response that is expected as a response for the command sent
Raises:	RetransmissionRequestedError, UnexpectedResponseError
Returns:	raw data read from from the remote server
Return type:	str

connect()¶

Connects to the remote target and initializes the chosen modes

set_extended_mode()¶

Enable extended mode. In extended mode, the remote server is made persistent. The ‘R’ packet is used to restart the program being debugged. Original documentation at:

https://sourceware.org/gdb/current/onlinedocs/gdb/Packets.html#extended-mode

start_no_ack_mode()¶

Request that the remote stub disable the normal +/- protocol acknowledgments. Original documentation at:

https://sourceware.org/gdb/current/onlinedocs/gdb/General-Query-Packets.html#QStartNoAckMode

avocado.utils.genio module¶

Avocado generic IO related functions.

avocado.utils.genio.ask(question, auto=False)¶

Prompt the user with a (y/n) question.

Parameters:	question (str) – Question to be asked auto (bool) – Whether to return “y” instead of asking the question
Returns:	User answer
Return type:	str

avocado.utils.genio.close_log_file(filename)¶

avocado.utils.genio.log_line(filename, line)¶

Write a line to a file.

Parameters:	filename – Path of file to write to, either absolute or relative to the dir set by set_log_file_dir(). line – Line to write.

avocado.utils.genio.read_all_lines(filename)¶

Return all lines of a given file

This utility method returns an empty list in any error scenario, that is, it doesn’t attempt to identify error paths and raise appropriate exceptions. It does exactly the opposite to that.

This should be used when it’s fine or desirable to have an empty set of lines if a file is missing or is unreadable.

Parameters:	filename (str) – Path to the file.
Returns:	all lines of the file as list
Return type:	builtin.list

avocado.utils.genio.read_file(filename)¶

Read the entire contents of file.

Parameters:	filename (str) – Path to the file.
Returns:	File contents
Return type:	str

avocado.utils.genio.read_one_line(filename)¶

Read the first line of filename.

Parameters:	filename (str) – Path to the file.
Returns:	First line contents
Return type:	str

avocado.utils.genio.set_log_file_dir(directory)¶

Set the base directory for log files created by log_line().

Parameters:	dir – Directory for log files.

avocado.utils.genio.write_file(filename, data)¶

Write data to a file.

Parameters:	filename (str) – Path to the file. line (str) – Line to be written.

avocado.utils.genio.write_one_line(filename, line)¶

Write one line of text to filename.

Parameters:	filename (str) – Path to the file. line (str) – Line to be written.

avocado.utils.git module¶

APIs to download/update git repositories from inside python scripts.

class avocado.utils.git.GitRepoHelper(uri, branch='master', lbranch=None, commit=None, destination_dir=None, base_uri=None)¶

Bases: object

Helps to deal with git repos, mostly fetching content from a repo

Instantiates a new GitRepoHelper

Parameters:	uri (string) – git repository url branch (string) – git remote branch lbranch (string) – git local branch name, if different from remote commit (string) – specific commit to download destination_dir (string) – path of a dir where to save downloaded code base_uri (string) – a closer, usually local, git repository url from where to fetch content first from

checkout(branch=None, commit=None)¶

Performs a git checkout for a given branch and start point (commit)

Parameters:	branch – Remote branch name. commit – Specific commit hash.

execute()¶

Performs all steps necessary to initialize and download a git repo.

This includes the init, fetch and checkout steps in one single utility method.

fetch(uri)¶

Performs a git fetch from the remote repo

get_top_commit()¶

Returns the topmost commit id for the current branch.

Returns:	Commit id.

get_top_tag()¶

Returns the topmost tag for the current branch.

Returns:	Tag.

git_cmd(cmd, ignore_status=False)¶

Wraps git commands.

Parameters:	cmd – Command to be executed. ignore_status – Whether we should suppress error.CmdError exceptions if the command did return exit code !=0 (True), or not suppress them (False).

init()¶

Initializes a directory for receiving a verbatim copy of git repo

This creates a directory if necessary, and either resets or inits the repo

avocado.utils.git.get_repo(uri, branch='master', lbranch=None, commit=None, destination_dir=None, base_uri=None)¶

Utility function that retrieves a given git code repository.

Parameters:	uri (string) – git repository url branch (string) – git remote branch lbranch (string) – git local branch name, if different from remote commit (string) – specific commit to download destination_dir (string) – path of a dir where to save downloaded code base_uri (string) – a closer, usually local, git repository url from where to fetch content first from

avocado.utils.iso9660 module¶

Basic ISO9660 file-system support.

This code does not attempt (so far) to implement code that knows about ISO9660 internal structure. Instead, it uses commonly available support either in userspace tools or on the Linux kernel itself (via mount).

avocado.utils.iso9660.iso9660(path)¶

Checks the available tools on a system and chooses class accordingly

This is a convenience function, that will pick the first available iso9660 capable tool.

Parameters:	path (str) – path to an iso9660 image file
Returns:	an instance of any iso9660 capable tool
Return type:	Iso9660IsoInfo, Iso9660IsoRead, Iso9660Mount or None