These tests are provided for Xalan contributors to evaluate the impact of code changes. Run the tests on the unchanged code, make the change and rebuild. Then run the tests again and compare the results. Results are automatically compared to the files in the "-gold" directory trees. Even though not all tests have "gold" files, it's still valuable to run the tests before and after a code change. That way you can at least ensure that your changes didn't cause any regressions to the code before you check your changes in. In the future, we hope to provide a tool to compare test results from one run to another (without necessarily having to re-run the test) to make this process even simpler.

The Xalan tests include a richly featured and fully automated testing framework that the great majority of the tests use. The org.apache.qetest package provides an independent testing and automation harness for Xalan, including logging and reporting mechanisims. The basic framework is easily used in testing other programs as well since dependencies on Xalan and an XML parser are clearly compartmented.

Nearly all tests are automated, run without any user interaction and with a minimum amount of setup, and produce a rolled-up report of their pass/fail/other status. Our existing testing library includes a wide array of tests, from XSLT conformance to detailed API tests, and welcomes user-submitted tests as well. The 'smoketest' target (a subset of the most important tests) are also integrated into the GUMP nightly build system, and thus ensure a minimum baseline of functionality on a daily basis. Developers can run the smoketest and ensure it passes before checking in code changes.

Brief overview of directory structure:

xml-xalan/test

Top level dir for all Xalan versions/products tests

xml-xalan/test/tools

Tools required by the test harness, such as JTidy. Note that all .jars required to run Xalan, including Ant, the Xerces parser, etc, are included in the lib and tools directories under xml-xalan/java.

xml-xalan/test/java/src

Java test automation source tree - this includes a generic testing framework as well as specific API tests for parts of Xalan and several test drivers for testing conformance / performance / etc. over a large number of xsl test stylesheets.
Primary packages are:
org.apache.qetest
org.apache.qetest.xsl
org.apache.qetest.trax
org.apache.qetest.trax.dom
org.apache.qetest.trax.stream
org.apache.qetest.trax.sax
org.apache.qetest.xalanj2
org.apache.qetest.dtm

xml-xalan/test/tests

Top level for XSLT stylesheet trees and special API tests

xml-xalan/test/tests/conf

Directory tree of specific conformance testing stylesheets

xml-xalan/test/tests/conf-gold

Directory tree of specific conformance testing stylesheets gold output reference files (this tree should mirror the structure of contrib)

xml-xalan/test/tests/contrib

Directory tree of user-contributed stylesheets

xml-xalan/test/tests/contrib-gold

Directory tree of user-contributed stylesheets gold output reference files (this tree should mirror the structure of contrib)

xml-xalan/test/tests/api

Directory tree for stylesheets used in Java API tests

xml-xalan/test/tests/api/trax

Stylesheets used in Java API tests in org.apache.qetest.trax

xml-xalan/test/tests/api/trax/dom

Stylesheets used in Java API tests in org.apache.qetest.trax.dom

etc. - often the directory tree in the stylesheet area will match the Java sources directory/package tree.

xml-xalan/test/tests/api-gold

Matching Directory tree of gold files for Java API tests

xml-xalan/test/tests/extensions

Directory tree for stylesheets used in Xalan-specific extension tests

xml-xalan/test/tests/extensions/java

Tests for extensions written in Java

xml-xalan/test/tests/extensions/javascript

Tests for extensions written in Javascript

xml-xalan/test/tests/extension-gold

Matching Directory tree of gold files for extensions tests

xml-xalan/test/tests/bugzilla

Special directory of stylesheets and automated Testlets reproducing selected Bugzilla bug reports

This section is a sort of catalog of existing tests. Beginning users will probably want to see how to run tests as well.

Java Test Drivers (data driven testing)

A Java Test Driver executes a test for each xml/xsl file pair in the specified directory tree or each pair in the specified fileList. The driver iterates over the inputDir tree or list of files and asks a Testlet to execute() a test on each one. This is also similar to data driven testing, where a common algorithim is defined for a test case, and then a large number of data points (in this case, the xml/xsl file pairs) are run through the test case in order. The best example is StylesheetTestletDriver. Another generic example is FileTestletDriver.

The Test Drivers rely on various Testlet implementations to define the actual testing algorithim to apply to each xml/xsl file pair. This defines any options to be used when processing the file as well as logging out information about the test in progress. Examples include StylesheetTestlet and PerformanceTestlet

The Testlets rely on TransformWrapper subclasses to perform the actual test of processing or transformation of the xml/xsl file pair into the output file. We can then plug in different TransformWrapper "flavors" easily. Different TransformWrapper can process or transform in various ways, like using DOM trees, SAX events, or input/output streams.

The three levels of iteration, test algorithim, and processor flavor are all independently changeable, so we can easily try out different kinds of tests. This technique is used to run the full sets of 'conf' (conformance), 'perf' (performance), extensions, and 'contrib' (user contributed stylesheets) tests.

org.apache.qetest.xsl.XalanCTestlet

This is similar to the StylesheetTestlet, but for Xalan-C. It simply shells out to a command prompt to run each stylesheet through the TestXSLT.exe program from Xalan-C.

Java API tests for the TRAX (or javax.xml.transform) interface, that Xalan-J 2.x implements.
All in package: org.apache.qetest.trax

(This Section needs updating: many new tests have been added; see the Javadoc for a list -sc)

REPLACE_template_for_new_tests.java

a template for creating new TRAX API tests, see Submitting New Tests

LoggingErrorListener.java

utility: wraps javax.xml.transform.ErrorListener, and logs info; this class also supports setting expected errors to trap, and it will call logger.checkPass/checkFail for you when it gets an expected or unexpected event. This allows us to write very detailed negative tests and have them be fully automated.

LoggingURIResolver.java

utility: wraps javax.xml.transform.URIResolver, and logs info

ExamplesTest.java

A testing version of samples/trax/Examples.java, a sample file provided in Xalan-J 2.x showing various uses of the TRAX or javax.xml.transform API to process stylesheets.

TransformerAPITest.java

API coverage tests for javax.xml.transform.Transformer

TransformerFactoryAPITest.java

API coverage tests for javax.xml.transform.TransformerFactory

TemplatesAPITest.java

API coverage tests for javax.xml.transform.Templates

EmbeddedStylesheetTest.java

Testing various types and kinds of stylesheets embedded with the xml-stylesheet PI

ErrorListenerAPITest.java

API Coverage test for ErrorListener

ErrorListenerTest.java

Functionality test of error listeners when using illegal stylesheets

OutputPropertiesTest.java

Various tests of programmatic access and changing of output properties

SystemIdImpInclTest.java

Testing various forms of URLs in setSystemID with imported and included stylesheets

SystemIdTest.java

Testing various forms of URLs in setSystemID

TestThreads.java

MANUALLY executed test for running multiple threads and transforming multiple stylesheets simultaneously. An updated and automated test is now available, org.apache.qetest.xsl.ThreadedTestletDriver, which should be used instead.

All in subpackages of: org.apache.qetest.trax

stream.StreamSourceAPITest.java

API coverage tests for javax.xml.transform.stream.StreamSource

stream.StreamResultAPITest.java

API coverage tests for javax.xml.transform.stream.StreamResult

etc.

API coverage tests are available with a similar naming scheme for most javax.xml.transform.* classes

A few tests are ones that Xalan does not currently pass due to Bugzilla reports, but we know the correct ("gold") result by analysis or by trying the test on other processors. A number of tests may also be missing matching "gold" files, if we haven't yet had time to confirm the correct output. It's still useful to run these tests (although the test driver will report an AMBG or 'Ambiguous' result) because you can still see if the output looks basically correct, and compare the output to previous test runs before you submit your code changes, etc.

The tests have several different types of results beyond just pass or fail, which are documented in org.apache.qetest.Logger.

• Shane Curcuru
• Paul Dick
• David Marston
• Tom Amiro
• Gary L Peskin
• Many other xalan-dev subscribers
• Test cases written by Carmelo Montanez at NIST for general use
• Many other helpers who we still need to credit! Sorry!