52.0 LTS

The Avocado team is proud to present another release: Avocado version 52.0, the second Avocado LTS version.

What’s new?

When compared to the last LTS (v36), the main features introduced by this versions are:

  • TAP[2] version 12 support.
  • The download of assets in tests now allow for an expiration time.
  • Environment variables can be propagated into tests running on remote systems.
  • The plugin interfaces have been moved into the avocado.core.plugin_interfaces module.
  • Support for running tests in a Docker container.
  • Introduction of the “Fail Fast” feature (--failfast on option) to the run command, which interrupts the Job on a first test failure.
  • Special keyword latest for replaying previous jobs.
  • Support to replay a Job by path (in addition to the Job ID method and the latest keyword).
  • The introduction of a tool that generated a diff-like report of two jobs.
  • The introduction of a rr based wrapper.
  • The automatic VM IP detection that kicks in when one uses --vm-domain without a matching --vm-hostname, now uses a more reliable method (libvirt/qemu-gust-agent query).
  • Set LC_ALL=C by default on sysinfo collection to simplify avocado diff comparison between different machines.
  • Result plugins system is now pluggable and the results plugins (JSON, XUnit, HTML) were turned into stevedore plugins. They are now listed in the avocado plugins command.
  • Multiplexer was replaced with Varianter plugging system with defined API to register plugins that generate test variants.
  • Old --multiplex argument, which used to turn yaml files into variants, is now handled by an optional plugin called yaml_to_mux and the --multiplex option is being deprecated in favour of the --mux-yaml option, which behaves the same way.
  • It’s now possible to disable plugins by using the configuration file.
  • Better error handling of the virtual machine plugin (--vm-domain and related options).
  • When discovering tests on a directory, the result now is a properly alphabetically ordered list of tests.
  • Plugins can now be setup in Avocado configuration file to run at a specific order.
  • Support for filtering tests by user supplied “tags”.
  • Users can now see the tags set on INSTRUMENTED test when listing tests with the -V (verbose) option.
  • Users can now choose to keep the complete set of files, including temporary ones, created during an Avocado job run by using the --keep-tmp option (e.g. to keep those files for rr).
  • Tests running with the external runner (--external-runner) feature will now have access to the extended behavior for SIMPLE tests, such as being able to exit a test with the WARNING status.
  • Encoding support was improved and now Avocado should safely treat localized test-names.
  • Test writers now have access to a test temporary directory that will last not only for the duration of the test, but for the duration of the whole job execution to allow sharing state/exchanging data between tests.
  • When not disabled, “results.tap” is by default generated in results directory.
  • Avocado is now available on Fedora standard repository. The package name is python2-avocado. The optional plugins and examples packages are also available. Run dnf search avocado to list them all.
  • Optional plugins and examples packages are also Avocado on PyPI.
  • Avocado test writers can now use a family of decorators, namely avocado.skip(), avocado.skipIf() and avocado.skipUnless() to skip the execution of tests.
  • self.skip() is under deprecation process and users should use the skip decorators instead.
  • Sysinfo collection based on command execution now allows a timeout to be set.
  • The non-local runner plugins, the html plugin and the yaml_to_mux plugin are now distributed in separate packages.
  • Users sending SIGTERM to Avocado can now expect it to be properly handled.
  • Support for new type of test status, CANCEL, and of course the mechanisms to set a test with this status.
  • avocado.TestFail, avocado.TestError and avocado.TestCancel are now public Avocado Test APIs, available from the main avocado namespace.
  • Introduction of the robot plugin, which allows Robot Framework tests to be listed and executed natively within Avocado.
  • A brand new ResultsDB optional plugin.
  • Listing of supported loaders (--loaders \?) was refined.
  • Variant-IDs generated by yaml_to_mux plugin now include leaf node names to make them more meaningful, making easier to skim through the results.
  • yaml_to_mux now supports internal filters defined inside the YAML file expanding the filtering capabilities even further.
  • Avocado now supports resuming jobs that were interrupted.
  • The HTML report now presents the test ID and variant ID in separate columns, allowing users to also sort and filter results based on those specific fields.
  • The HTML report will now show the test parameters used in a test when the user hovers the cursor over the test name.
  • Avocado now reports the total job execution time on the UI, instead of just the tests execution time.
  • The Avocado multiplex command has been renamed to variants.
  • Loaders were tweaked to provide more info on avocado list -V especially when they don’t recognize the reference.
  • Users can use --ignore-missing-references on to run a job with undiscovered test references
  • Users can now choose in which order the job will execute tests (from its suite) and variants. The two available options are --execution-order=variants-per-test (default) or --execution-order=tests-per-variant.
  • Test methods can be recursively discovered from parent classes by upon the :avocado: recursive docstring directive.

Those are the main, but not all, features that are available to users. Besides that, we had several improvements in our utils libraries that are important for test writers, some of them are listed below:

  • time_to_seconds, geometric_mean and compare_matrices were added in avocado.utils.data_structures.
  • avocado.utils.distro was refined.
  • avocado.utils.{filelock,lv_utils,multipath,partition,pci} wereintroduced.
  • avocado.utils.memory contains several new methods.
  • New avocado.utils.process.SubProcess.get_pid() method.
  • sudo support in avocado.utils.process was improved
  • The avocado.utils.process library makes it possible to ignore spawned background processes that never finish.
  • New avocado.utils.linux_modules.check_kernel_config().
  • Users of the avocado.utils.process module will now be able to access the process ID in the avocado.utils.process.CmdResult.
  • Improved avocado.utils.iso9660 with a more complete standard API across all back-end implementations.
  • Improved avocado.utils.build.make(), which will now return the make process exit status code.
  • The avocado.Test class now better exports (and protects) the core class attributes members (such as params and runner_queue).
  • avocado.utils.linux_modules functions now returns module name, size, submodules if present, filename, version, number of modules using it, list of modules it is dependent on and finally a list of params.

It is also worth mentioning:

  • Improved documentation, with new sections to Release Notes and Optional Plugins, very improved Contribution and Community Guide. New content and new examples everywhere.
  • The avocado-framework-tests GitHub organization was founded to encourage companies to share Avocado tests.
  • Bugs were always handled as high priority and every single version was delivered with all the reported bugs properly fixed.

And here are some statistics. When compared to the last LTS, we had:

  • 1187 commits (and counting).
  • 15 new versions.
  • 4811 more lines of Python code (+27,42%).
  • 1800 more lines of code comment (+24,67%).
  • 31 more Python files (+16,48%).
  • 69 closed GitHub issues.
  • 34 contributors from at least 12 different companies, 26 of them contributing for the fist time to the project.