The X11R6.9/X11R7 Release Testing Plan
Detailed test instructions
This section outlines the test procedure to follow when testing a release candidate. We need for everyone to test both the X11R6.9 (monolithic source tree) and the X11R7.0 (modular source tree). When you have completed a test run, please submit your test report to our results database so that we can track the progress of the release. We need these reports to determine if the code is ready for release or needs more work.
There are four main areas that we are interested in testing:
- Build tests: Does the release build properly?
- Install tests: Does the release install properly?
- Conformance tests: Does the release conform to the specifications (i.e., does it pass the X Test Suite?)
- Run tests: Does the release run the standard desktop and applications properly?
Each of these are described below in detail.
Note that for any test(s) that fail, please file a bugzilla report in bugzilla and mark it as a blocker of the release bug.
Build tests
Monolithic build tests
For the monolithic source tree, each of the following build tests can be performed by copying the sample host.def file (or the alternate) to the xc/config/cf directory and the running make World >& World.LOG (or other such command as appropriate for your platform), and then checking the World.LOG file for any failures.
Build with BuildServersOnly defined as YES (sample alternate)
Build with DoLoadableServer defined as NO (sample alternate)
Note that some systems do not have a compatible version of Freetype2 installed on their system, so in addition to each build requirement above, defining HasFreetype2 as NO is permitted. Each alternate host.def file above have this define included.
Modular build tests
For the modular source tree, we ask that you build from CVS using the methods described in the Modular developer's guide. The build.sh script in CVS will allow you to build and install all of the module components in the appropriate order. By default, the build.sh script will exit if it fails to build or install any component.
If you would like to have the build.sh script warn you about build or install failures, then you can use the -n command-line flag to tell it to print error messages instead of exiting. For example, you could run build.sh -n /tmp/modular >& build.log and then once the script completes, you could grep '*****' build.log to see if there are any errors and which packages failed.
One additional test that further helps find problems is to have the build.sh script also run make distcheck on each package, which can be enabled with the -d command-line flag.
Install tests
Monlithic install tests
For the monolithic source tree, each of the following install tests can be performed by building the release (as described above using the sample or alternate host.def file provided), running make Install >& Install.LOG (or other such command as appropriate for your platform), and checking the Install.LOG output for any failures.
Build and install with: ProjectRoot defined to be something other than the default, and NothingOutsideProjectRoot defined as YES (sample alternate)
Note that some systems do not have a compatible version of Freetype2 installed on their system, so in addition to each install requirement above, defining HasFreetype2 as NO is permitted. Each alternate host.def file above have this define included.
Modular install tests
Since most of the modular source tree components have dependencies on other module components being built and installed, the build.sh script will both build and install all of the components in the appropriate dependency order. If the script completed without errors in the build step above, then the modular source tree both built and installed correctly.
Conformance tests
After installing the full release of either the monolithic, modular or both source trees, the conformance tests can be run using the X Test Suite (XTS). Note that XTS has been updated recently and the build instructions have changed from what they were for past releases.
Setting up the X Test Suite
We have prepared an XTS package for those that are running Linux on an x86 machine. You can download the prebuilt package here. Untar this package and cd into the xts5-Linux-i686 subdirectory.
For everyone else or if you have problems with the pre-built package, you will need to build XTS on your local system. To build it, download the mktestpkg.sh script from CVS and run it, which you can download here. This script will get the appropriate files, check out the latest version of XTS from CVS (Note when it asks you for a password, just hit enter). Then, change to the subdir that is created by the build script.
Now you should be ready to begin testing.
Note that an updated helper script (called xreg), which is used to run XTS, is already included in the test suite subdirectory.
Examples of how to use the xreg script
Here are some examples of how to use xreg to run the X test suite:
xreg -xtest -xvfb
- This runs xtest at all default depths using the Xvfb server.
- The default depths are 8, 15, 16, and 24+32.
- The "24+32" depth is one that uses a depth of 24 with a frame buffer bits per pixel of 32 (i.e., -depth 24 -fbbpp 32).
xreg -xtest -xorg -d 16
- This runs xtest at depth 16 using the Xorg server.
xreg -xtest -xvfb -d 15 -test XCopyArea
This runs xtest at depth 15 using the Xvfb server, but it only runs the XCopyArea test.
- Selecting individual tests is very useful to track down test failures.
xreg -xtest -xvfb -d 16 -xvfbwidth 1280 -xvfbheight 1024 -test XFillRectangles
This runs xtest at depth 16 using the Xvfb server running at 1280x1024, but it only runs the XFillRectangles test.
Notes on using xreg:
The output from these test runs are stored in results subdirectory by default. You can change the default output dir using the -O xreg command-line option.
The material below assumes that you have done a full install of the system to /usr/X11R6. However, if you are using a different ProjectRoot (for the monolithic tree) or a different --prefix (for the modular tree), you can use the following command-line option to the xreg script to run from that alternate location: -projroot path-to-your-project-root-or-prefix
- Running XTS at all bit depths can take a long time (6 hours on a P3 1GHz), so we suggest running them overnight.
- The files that are generated from an xreg run of xtest are:
X-setup.DEPTH.DATE.TIME.output -- this file contains the output of the X server during the setup phase
xts5.DEPTH.DATE.TIME.errors -- this file contains the list of errors found during the test run at depth DEPTH made on date DATE at time TIME.
xts5.DEPTH.DATE.TIME.report -- this file contains the report of all tests run at depth DEPTH made on date DATE at time TIME.
xts5.DEPTH.DATE.TIME.summary -- this file contains a summary of the errors found during the test run at depth DEPTH made on date DATE at time TIME. The summary file is only useful during full test runs (e.g., not when running individual tests).
xts5.DEPTH.DATE.TIME.results -- this directory contains the journal from the tests run at depth DEPTH made on date DATE at time TIME as well as any error images generated.
- After running xtest, you can check to see if everything passed by looking at the summary/errors/report file(s) to see if there are any failures.
As we get more test results in, we will be collecting a list of known failures. We can then update the xreg script to take those known errors into account in the summary file.
The xreg script has only been tested on Linux systems. If there are problems with these scripts, please post patches to the xorg@lists.freedesktop.org mailing list.
There are many other options to xreg (and it can be used to run other tests such as x11perf). Run xreg -help to see the usage message.
After the XTS has been run with xreg, the report file lists the failures and a summary of the test run. Here is an example of the test report generated:
VSW5 SUMMARY RESULTS REPORT
Test suite version: 5.1.5
Specification version: Open Group Window Management (X11R5) document set
Test run by: guest
System: Linux t5 2.6.13-1.1576_FC5 #1 Sat Sep 24 15:23:34 EDT 2005 i686
Test run started: Monday November 07, 2005 11:52:31 PM
Test run ended: Monday November 07, 2005 01:23:18 AM
Journal file: /tmp/xtest/xts5-Linux-i686/results/xts5.16bpp.20051107.204759.results/journal
TCC command line: tcc -e -s /tmp/xreg.tet_scen.26369 -x /tmp/xreg.tetexec.cfg.26369 -i /tmp/xtest/xts5-Linux-i686/results/xts5.16bpp.20051107.204759.results xts5 all
Report type: -d 1 -s 1
CASES TESTS PASS UNSUP UNTST NOTIU WARN FIP FAIL UNRES UNIN ABORT
Xproto 122 389 267 2 120 0 0 0 0 0 0 0
Xlib3 109 161 129 3 28 1 0 0 0 0 0 0
Xlib4 29 324 275 17 27 5 0 0 0 0 0 0
Xlib5 15 84 77 2 5 0 0 0 0 0 0 0
Xlib6 8 50 20 0 30 0 0 0 0 0 0 0
Xlib7 58 172 150 9 13 0 0 0 0 0 0 0
Xlib8 29 165 133 10 22 0 0 0 0 0 0 0
Xlib9 46 1472 1176 46 36 201 8 0 5 0 0 0
Xlib10 23 95 58 2 35 0 0 0 0 0 0 0
Xlib11 33 195 87 22 43 43 0 0 0 0 0 0
Xlib12 27 138 108 2 15 12 0 0 1 0 0 0
Xlib13 32 269 161 3 102 3 0 0 0 0 0 0
Xlib14 45 58 49 0 5 0 0 0 4 0 0 0
Xlib15 45 159 126 0 33 0 0 0 0 0 0 0
Xlib16 30 105 82 1 22 0 0 0 0 0 0 0
Xlib17 55 131 110 0 21 0 0 0 0 0 0 0
Xopen 8 127 125 2 0 0 0 0 0 0 0 0
Xt3 21 73 73 0 0 0 0 0 0 0 0 0
Xt4 33 192 94 0 98 0 0 0 0 0 0 0
Xt5 10 69 28 0 41 0 0 0 0 0 0 0
Xt6 7 71 71 0 0 0 0 0 0 0 0 0
Xt7 11 106 97 0 6 3 0 0 0 0 0 0
Xt8 7 43 35 0 4 0 0 0 4 0 0 0
Xt9 33 189 132 2 55 0 0 0 0 0 0 0
Xt10 8 17 16 0 1 0 0 0 0 0 0 0
Xt11 58 285 249 0 34 0 0 0 1 1 0 0
Xt12 22 67 56 0 11 0 0 0 0 0 0 0
Xt13 39 178 129 0 47 0 0 0 1 1 0 0
Xt14 2 18 18 0 0 0 0 0 0 0 0 0
Xt15 1 2 0 2 0 0 0 0 0 0 0 0
XtC 29 147 90 1 56 0 0 0 0 0 0 0
XtE 1 1 1 0 0 0 0 0 0 0 0 0
TOTAL 996 5552 4222 126 910 268 8 0 16 2 0 0If the TOTAL line at the end of your report is similar to the one shown above, then we consider that your system has passed XTS. We are working to resolve the remaining failures and unresolved tests. From above, there are 16 failed and 2 unresolved tests. We will update the list above as we resolve more of these tests. If you are successful in fixing them, please let us know on the xorg@lists.freedesktop.org mailing list.
Actually running the conformance tests
For the purposes of the testing the monolithic and modular source trees for this release, we ask that you run one of the following scenarios:
For platforms based on a XFree86-style DDX, run XTS on Xorg either using the card driver for your video card or the dummy driver.
For example: xreg -xtest -xorg
- This will run xtest at all the default depths using the Xorg server.
- Check the output of each report or summary file to make sure that all tests that are expected to pass do actually pass.
For all other platforms, Xvfb should be used. This step is optional for those who ran with the Xorg server above.
For example: xreg -xtest -xvfb
- This will run xtest at all of the default depths using the Xvfb server.
- Check the output of each report or summary file to make sure that all tests that are expected to pass do actually pass.
Additional notes:
The Xvfb server is special X server that uses a virtual framebuffer. It is normally built and installed with the full release. See the Xvfb(1) for more information about this server.
The dummy driver is a special driver available with the XFree86 DDX. To use the dummy driver, simply substitue it for your normal card driver in the Device section of your xorg.conf configuration file and comment out any card-specific options from that section. For example, if you normally uses an ATI driver, then you will have a Device section with Driver "ati" to let the X server know that you want it to load and use the ATI driver. It is not necessary to run both the card driver and the dummy driver for the conformance tests, but if you run into issues with the card driver not passing XTS, we recommend that you try running the dummy driver to see if the failures are resolved. This will help us narrow down any failures.
Run tests
After installing the full release and running the conformance tests, run the subset of tests listed below that applies to the platform being tested. We would like to get as much coverage as possible, so please run these tests on as many cards as possible.
Tests to run for each card:
- X Test Suite (in case you haven't already run it)
x11perf (Note that you can use xreg to automatically run x11perf -- see the script for details)
- rendertest (found in the xapps CVS repository on freedesktop.org)
- Standard graphical environment (e.g., Gnome or KDE desktop)
GL tests: glxgears, gloss, quake3
- Switch to/from VTs (on Linux)
After running these tests, please enter them into our results database. This will allow us to track progress of the release. To see the current set of test results, go to the results summary page.