Difference between revisions of "Automated Testing - Suggestions"

From Rosalab Wiki
Jump to: navigation, search
(→‎The System)
(→‎Enhancements)
 
(4 intermediate revisions by the same user not shown)
Line 6: Line 6:
 
# It creates virtual machines (currently based on qemu-kvm) and checks that the OS boots in Live mode from the ISO images on these machines.
 
# It creates virtual machines (currently based on qemu-kvm) and checks that the OS boots in Live mode from the ISO images on these machines.
 
# It automatically installs the OS to these virtual machines from the ISO images and performs several basic adjustments and checks. For example, it checks whether SSH connection to the machine is working, tries to enable the default package repositories and perform a package update, etc.
 
# It automatically installs the OS to these virtual machines from the ISO images and performs several basic adjustments and checks. For example, it checks whether SSH connection to the machine is working, tries to enable the default package repositories and perform a package update, etc.
−
# [http://docs.python.org/2.7/library/unittest.html Python inittest] is used to run functional tests on the live and installed systems.
+
# [http://docs.python.org/2.7/library/unittest.html Python unittest ("PyUnit")] is used to run functional tests on the live and installed systems.
 
# The results of the above steps are available [http://fba.rosalinux.ru/autotest/ at FBA]
 
# The results of the above steps are available [http://fba.rosalinux.ru/autotest/ at FBA]
  
Line 32: Line 32:
  
 
* Xfstests and/or other file system tests - at least for ext4 FS, other file systems can also be tested this way in the future
 
* Xfstests and/or other file system tests - at least for ext4 FS, other file systems can also be tested this way in the future
−
** <strike>ext4 with default mount options</strike> (''Done. At least the tests that run no more than several minutes run now.'')
+
** ext4 with default mount options
−
** ext4 with ''noatime'' and ''discard'' (commonly used for the FS on SSD drives) (''Done for noatime.'')
+
** ext4 with ''noatime'' and ''discard'' (commonly used for the FS on SSD drives)
−
** <strike>btrfs</strike> (''Done. At least the tests that run no more than several minutes run now.'')
+
** btrfs
 
** ? XFS
 
** ? XFS
 
** ? ReiserFS + Reiser4
 
** ? ReiserFS + Reiser4
Line 40: Line 40:
 
** ? iso9660
 
** ? iso9660
 
** ? UDF
 
** ? UDF
−
** <strike>NTFS (with FUSE-based driver)</strike> (''Done. Adapted xfstests for this purpose.'')
+
** NTFS (with FUSE-based driver)
 
** ? SquashFS
 
** ? SquashFS
 
** CIFS
 
** CIFS
 
** ? NFS
 
** ? NFS
−
* <strike>LSB test suites (see LSB Distribution Checker) - at least libcheck and cmdcheck.</strike> (''Done. The system runs libchk and cmdchk daily.'')
+
* LSB test suites (see LSB Distribution Checker) - at least libcheck and cmdcheck.
 
* Mozmill tests - at least for Firefox (similar to how it is done in OpenQA for OpenSUSE)
 
* Mozmill tests - at least for Firefox (similar to how it is done in OpenQA for OpenSUSE)
 
* Checking for errors in the kernel modules with KEDR tools
 
* Checking for errors in the kernel modules with KEDR tools
Line 54: Line 54:
 
==== Benchmarks ====
 
==== Benchmarks ====
  
−
* ? (''Are they really needed here?'')
+
* TODO
  
 
==== Enhancements ====
 
==== Enhancements ====
Line 62: Line 62:
 
''Comments: it is already possible to use a particular ISO image rather than the newest one. It is also possible to write Autotest scenarios that enable the given repositories, update the specified packages from these repositories and test that software.''
 
''Comments: it is already possible to use a particular ISO image rather than the newest one. It is also possible to write Autotest scenarios that enable the given repositories, update the specified packages from these repositories and test that software.''
  
−
* ''(done)'' Facilities for UI testing similar to what is used in [http://os-autoinst.org/ os-autoinst] / [http://openqa.opensuse.org/ OpenQA]
+
* <strike>Facilities for UI testing similar to what is used in [http://os-autoinst.org/ os-autoinst] / [http://openqa.opensuse.org/ OpenQA]</strike> ''(done)''
 
''This is now possible. See the description and the tutorials here: [http://wiki.rosalab.ru/en/index.php/Automated_Testing English] [http://wiki.rosalab.ru/ru/index.php/Автоматизированное_тестирование_ПО Russian]''
 
''This is now possible. See the description and the tutorials here: [http://wiki.rosalab.ru/en/index.php/Automated_Testing English] [http://wiki.rosalab.ru/ru/index.php/Автоматизированное_тестирование_ПО Russian]''
  
 
* Distinction of known and unknown failures.
 
* Distinction of known and unknown failures.
−
 
−
* Enhancements of the report generation mechanism in Autotest:
 
−
** Fix broken links in the generated pages (dmesg, etc.).
 
−
** Show the description of the tests in the report.
 
−
** Something to make it easier to see how exactly the tests have failed (currently, it is needed to look through the logs).
 
  
 
* When the repository for pending updates is available, it will be useful to add it on the client machines to check the updates early.
 
* When the repository for pending updates is available, it will be useful to add it on the client machines to check the updates early.
 +
 +
* Use some OCR facilities like tesseract or whatever to allow the tests to check if the appropriate text is displayed in the application's controls. Matching of the images that contain only text may be unreliable due to font anti-aliasing, etc.
  
 
== Development ==
 
== Development ==
  
 
The sources of the system are available at [https://abf.rosalinux.ru/spectre/rosa-autotest ABF].
 
The sources of the system are available at [https://abf.rosalinux.ru/spectre/rosa-autotest ABF].

Latest revision as of 16:56, 29 July 2013

The System

The automated testing system for ROSA Linux distros operates as follows:

  1. It looks for the new ISO images for the appropriate distros on ABF. If they are found, it downloads the images to a local machine.
  2. It creates virtual machines (currently based on qemu-kvm) and checks that the OS boots in Live mode from the ISO images on these machines.
  3. It automatically installs the OS to these virtual machines from the ISO images and performs several basic adjustments and checks. For example, it checks whether SSH connection to the machine is working, tries to enable the default package repositories and perform a package update, etc.
  4. Python unittest ("PyUnit") is used to run functional tests on the live and installed systems.
  5. The results of the above steps are available at FBA

All this is currently done for ROSA Desktop Fresh, both 32- and 64-bit.

Suggestions

Listed here are the suggestions collected so far about the tests and the system features that could be useful, in no particular order.

Profiles

  • Profile for ROSA Desktop Fresh GNOME Edition
  • Profile for ROSA Desktop Fresh LXDE Edition

Installation Options

  • /var on a separate partition
  • Installation on a RAID

Kernels

  • It could be useful to test the ROSA distros with different kernel flavours available for them.

Tests

  • Xfstests and/or other file system tests - at least for ext4 FS, other file systems can also be tested this way in the future
    • ext4 with default mount options
    • ext4 with noatime and discard (commonly used for the FS on SSD drives)
    • btrfs
    •  ? XFS
    •  ? ReiserFS + Reiser4
    • fat/vfat (USB Frash drives)
    •  ? iso9660
    •  ? UDF
    • NTFS (with FUSE-based driver)
    •  ? SquashFS
    • CIFS
    •  ? NFS
  • LSB test suites (see LSB Distribution Checker) - at least libcheck and cmdcheck.
  • Mozmill tests - at least for Firefox (similar to how it is done in OpenQA for OpenSUSE)
  • Checking for errors in the kernel modules with KEDR tools
  • Regression tests for some of the bugs fixed in the distros
  •  ? Fuzz tests like Trinity, etc. (may be destructive)
  • Check the selected software, esp., the software developed by ROSA, for memory-related errors. The tests may be based on Valdrind's Memcheck or other tools.
  •  ? (What else?)

Benchmarks

  • TODO

Enhancements

  • Possibility run test for specified iso-image + update specified package from specified repository. For example, we would like to use autotest for testing ROSA utilities like SimpleWelcome (development version).

Comments: it is already possible to use a particular ISO image rather than the newest one. It is also possible to write Autotest scenarios that enable the given repositories, update the specified packages from these repositories and test that software.

This is now possible. See the description and the tutorials here: English Russian

  • Distinction of known and unknown failures.
  • When the repository for pending updates is available, it will be useful to add it on the client machines to check the updates early.
  • Use some OCR facilities like tesseract or whatever to allow the tests to check if the appropriate text is displayed in the application's controls. Matching of the images that contain only text may be unreliable due to font anti-aliasing, etc.

Development

The sources of the system are available at ABF.

Retrieved from "http://wiki.rosalab.com/en/index.php?title=Automated_Testing_-_Suggestions&oldid=7553"