XorgTesting

The X.Org Foundation Testing Plan

Detailed test instructions

This section outlines the test procedure to follow when testing a release candidate. When a test run has been completed, please e-mail the xorg@lists.freedesktop.org mailing list with the following information so that we can track what has and has not yet been tested. Note that we are interested in progress: please do not wait to complete all phases of testing to send in reports.

1. Your Name
2. The date tested
3. The platform you tested:
   • The operating system tested (e.g., AIX, Cygwin, FreeBSD, HP-UX, Linux, etc.)
   • The architecture tested (e.g., Alpha, AMD64, EM64T, IA-32, IA-64, Sparc, etc.)
   • The distribution and release tested (e.g., Red Hat FC2, SUSE 9.1, Debian unstable, Solaris 9, etc.)
4. The snapshot or release candidate tag tested (e.g., XORG-6_7_99_1, etc.)
5. Build test status: passed or failed or untested
6. Install test status: passed or failed or untested
7. Conformance test status: passed or failed or untested
8. Run test status: passed or failed or untested
   • List the tests run

For any test(s) that failed, please include in your report the test(s) that failed, and file a bugzilla report if no one has already filed one against the failure(s) you found.

Build tests

Each of the following build tests can be performed by copying the sample host.def file (or the alternate) to the xc/config/cf directory and the running make World >& World.LOG (or other such command as appropriate for your platform), and then checking the World.LOG file for any failures.

1. Build with empty host.def file (sample alternate)

2. Build with BuildServersOnly defined as YES (sample alternate)

3. Build with DoLoadableServer defined as NO (sample alternate)

Note that some systems do not have a compatible version of Freetype2 installed on their system, so in addition to each build requirement above, defining HasFreetype2 as NO is permitted. Each alternate host.def file above have this define included.

Install tests

Each of the following install tests can be performed by building the release (as described above using the sample or alternate host.def file provided), running make Install >& Install.LOG (or other such command as appropriate for your platform), and checking the Install.LOG output for any failures.

1. Build and install with no host.def file (sample alternate)

2. Build and install with: ProjectRoot defined to be something other than the default, and NothingOutsideProjectRoot defined as YES (sample alternate)

Note that some systems do not have a compatible version of Freetype2 installed on their system, so in addition to each install requirement above, defining HasFreetype2 as NO is permitted. Each alternate host.def file above have this define included.

Conformance tests

After installing the full release, the conformance tests can be run using the X test suite, which can be downloaded here. A helper script (called xreg) is used to run the X test suite, which can be downloaded here. See the next two sections below for more information on how to setup and use these tools.

Setting up the X test suite

Here are some brief instructions on how to download and set up everything that you will need to run the X test suite:

1. Follow the directions at BuildingXtest

2. wget http://www.freedesktop.org/~kem/testing/xreg

Now you should be ready to begin testing.

Examples of how to use the xreg script

Here are some examples of how to use xreg to run the X test suite:

1. xreg -xtest -xvfb

   • This runs xtest at all default depths using the Xvfb server.
   • The default depths are 8, 15, 16, and 24+32.
   • The "24+32" depth is one that uses a depth of 24 with a frame buffer bits per pixel of 32 (i.e., -depth 24 -fbbpp 32).

2. xreg -xtest -xorg -d 16

   • This runs xtest at depth 16 using the Xorg server.

3. xreg -xtest -xvfb -d 15 -test XCopyArea

   • This runs xtest at depth 15 using the Xvfb server, but it only runs the XCopyArea test.

   • Selecting individual tests is very useful to track down test failures.

4. xreg -xtest -xvfb -d 16 -xvfbwidth 1280 -xvfbheight 1024 -test XFillRectangles -n 3-5

   • This runs xtest at depth 16 using the Xvfb server running at 1280x1024, but only runs the third through the fifth tests of the XFillRectangles test.

Notes on using xreg:

• The output from these test runs are stored in pwd/results by default. You can change the default output dir using the -O command line option.

• The material below assumes that you have done a full install of the system to /usr/X11R6. However, if you are using a different ProjectRoot, you can use the following command line option to the xreg script to run from that alternate location: -projroot path-to-your-project-root

• The files that are generated from an xreg run of xtest are:

  1. X-setup..output -- this file contains the output of the X server during the setup phase

  2. xtest.DEPTH.DATE.TIME.errors -- this file contains the list of errors found during the test run at depth DEPTH made on date DATE at time TIME.

  3. xtest.DEPTH.DATE.TIME.report -- this file contains the report of all tests run at depth DEPTH made on date DATE at time TIME.

  4. xtest.DEPTH.DATE.TIME.summary -- this file contains a summary of the errors found during the test run at depth DEPTH made on date DATE at time TIME. The summary file is only useful during full test runs (e.g., not when running individual tests).

  5. xtest.DEPTH.DATE.TIME.results -- this directory contains the journal from the tests run at depth DEPTH made on date DATE at time TIME as well as any error images generated.

• After running xtest, you can check to see if everything passed by looking at the summary/errors/report file(s) to see if there are any failures.

• There are some known failures that the summary file attempts to take into account. The first part of the summary file is the list of failures, and at the end of the summary file is a diff between the known failures (e.g., XDrawArcs) and what the failures were for this run.

• The xreg script has only been tested on Linux systems. If there are problems with these scripts, please post patches to the xorg@lists.freedesktop.org mailing list.

• There are many other options to xreg (and it can be used to run other tests such as x11perf). Run xreg -help to see the usage message.

Actually running the conformance tests

For this section, one of the following should be used for testing:

• For platforms based on a XFree86-style DDX, the dummy driver should be used.

  • For example: xreg -xtest -xorg

  • This will run xtest at all the default depths using the Xorg server.
  • Check the output of each report or summary file to make sure that all tests that are expected to pass do actually pass.

• For all other platforms, Xvfb should be used.

  • For example: xreg -xtest -xvfb -d "15 16 24+32"

  • This will run xtest at depths 15, 16 and 24+32 using the Xvfb server.
  • Check the output of each report or summary file to make sure that all tests that are expected to pass do actually pass.
  • Note that Xvfb does not currently run at depth 8, so the example above limits the testing to depths 15, 16 and 24+32. Update, this problem has been fixed in CVS now and Xvfb at depth 8 works again.

Additional notes:

• The Xvfb server is special X server that uses a virtual framebuffer. It is normally built and installed with the full release. See the Xvfb(1) for more information about this server.

• The dummy driver is a special driver available with the XFree86 DDX. To use the dummy driver, simply substitue it for your normal card driver in the Device section of your xorg.conf configuration file. For example, if you normally uses an ati driver, then you will have a Device section with Driver "ati" to let the X server know that you want it to load and use the ati driver; however, for these conformance tests, you would change that line to Driver "dummy" and remove any other ati specific options from the Device section.

Run tests

After installing the full release, you can run the subset of tests listed below that applies to the platform being tested. Please run these tests on at least two different driver families (where applicable). For example, on an IA-32 system running Linux, you could run the tests using one card from the ATI driver family and another card from the NVIDIA driver family.

Tests for each driver family:

1. X test suite (listed above)
2. x11perf
3. rendertest (found in the xapps CVS repository on freedesktop.org)
4. Standard graphical environment

5. GL tests: glxgears, gloss, quake3

6. Switch to/from VTs (on Linux)

-- KevinMartin - 18 Jan 2005

XorgTesting (last edited 2008-05-16 05:48:32 by localhost)

• Immutable Page

DeleteCache (cached 2013-04-26 18:54:32)

Or try one of these actions: AttachFile, Despam, LikePages, Load, LocalSiteMap, PackagePages, Save, SlideShow, SpellCheck, SubscribeUser, SyncPages