3.4. Invoking QMTest

All QMTest functionality is available using the qmtest command.

3.4.1. qmtest

3.4.1.1. Synopsis

qmtest [ option ...] command [ command-option ...] [ argument ...]

3.4.1.2. Options

These options can be used with any QMTest command, and must precede the command name on the command line.

All options are available in a "long form" prefixed with "--" (two hyphens). Some options also may be specified in a "short form" consisting of a single hyphen and a one-letter abbreviation. Short-form options may be combined; for example, -abc is equivalent to -a -b -c.

-D path, --tdb path

Use the test database located in the directory given by path. This flag overrides the value of the environment variable QMTEST_DB_PATH. If neither this flag nor the environment variable is specified, QMTest assumes that the current directory should be used as the database. See Section 3.1.6.

-h, --help

Display help information, listing commands and general options for the qmtest command.

-v, --verbose

Generate progress and status messages while executing. This option may be specified more than once; the more times it is specified, the higher the verbose level, and the more messages are printed.

Additional options are available for specific commands; these are presented with each command. Options specific to a command must follow the command on the command line. Specify the --help (-h) option after the command for a description of the command and a list of of available options for that command.

3.4.2. qmtest create-tdb

3.4.2.1. Summary

Create a new test database.

3.4.2.2. Synopsis

qmtest create-tdb [ option ...]

3.4.2.3. Description

The qmtest create-tdb command creates a new, empty test database. A test database is a directory in which QMTest stores configuration files, tests, and other data. Certain test database classes may also store data elsewhere, such as in an external relational database.

The test database is created in the directory specified by --tdb (-D) option or by setting the QMTEST_DB_PATH environment variable. The path you specify for the new test database must not exist. (If no database path is specified, QMTest assumes that the current directory is the test database. Since the current directory already exists, QMTest will issue an error. Therefore, when using the create-tdb command, the database directory must be explicitly specified.)

By default, QMTest creates a new test database that uses the standard XML-based implementation. (See Section 3.8 for information about writing a test database class.)

The create-tdb command accepts these options:

-a name=value, --attribute name=value

Set the database attribute name to value. The set of attribute names and valid values is dependent on the database class in use. The default database class accepts no attributes.

-c class, --class class

Use the test database class given by class. Once you create a test database, you cannot change the test database implementation it uses. If you do not use this option, QMTest will use the default test database implementation, which uses an XML file format to store tests.

3.4.3. qmtest gui

3.4.3.1. Summary

Start the graphical user interface.

3.4.3.2. Synopsis

qmtest gui [ option ...]

3.4.3.3. Description

The qmtest gui starts the graphical user interface. The graphical user interface is accessed through a web browser. You must have a web browser that supports JavaScript to use the graphical interface. QMTest has been tested with recent versions of Internet Explorer and Netscape Navigator. Other web browsers may or may nor work with QMTest.

The gui command accepts these options:

-A address, --address address

Bind the server to the indicated internet address, which should be a dotted quad. By default, the server binds itself to the address 127.0.0.1, which is the address of the local machine. If you specify another address, the server will be accessible to users on other machines. QMTest does not perform any authentication of remote users, so you should not use this option unless you have a firewall in place that blocks all untrusted users.

-c name=value, --context name=value

For details about this option, see the description of the qmtest run command.

-C file, --load-context file

For details about this option, see the description of the qmtest run command.

-j count, --concurrency count

For details about this option, see the description of the qmtest run command.

--no-browser

Do not attempt to start a web browser when starting the GUI. QMTest will still print out the URL at which the server can be accessed. You can then connect to this URL manually using the browser of your choice.

--port port

Specify the port on which the QMTest GUI will listen for connections. If this option is not provided, QMTest will select an available port automatically.

-T file, --targets file

For details about this option, see the description of the qmtest run command.

3.4.4. qmtest run

3.4.4.1. Summary

Run tests or test suites.

3.4.4.2. Synopsis

qmtest run [ option ...] [test-name | suite-name...]

3.4.4.3. Description

The qmtest run command runs tests and displays the results. If no test or suite names are specified, QMTest runs all of the tests in the test database. If test or suite names are specified, only those tests or suites are run. Tests listed more than once (directly or by inclusion in a test suite) are run only once.

The run command accepts these options:

-c name=value, --context name=value

Add a property to the test execution context. The name of the property is name, and its value is set to the string value.

This option may be specified multiple times.

-C file, --load-context file

Read properties for the test execution context from the file file.

The file should be a text file with one context property on each line, in the format name=value. Leading and trailing whitespace on each line are ignored. Also, blank lines and lines that begin with "#" (a hash mark) are ignored as comments.

This option may be specified more than once, and used in conjunction with the --context option.

-f format, --format format

Control the format used when displaying results. The format specified must be one of full, brief, stats, or none. The brief format is the default. In the full format, QMTest displays any annotations provided in test results. In the brief mode only the causes of failures are shown; detailed annotations are not shown. In the stats format, no details about failing tests are displayed; only statistics showing the number of passing and failing tests are displayed. In the none mode, no results are displayed, but a results file is still created, unless the --no-output option is also provided.

-j count, --concurrency count

Run tests in multiple count concurrent processes on the local computer. On multiprocessor machines, the processes may be scheduled to run in parallel on different processors. QMTest automatically collects results from the processes and presents combines test results and summary. By default, one process is used.

This option may not be combined with the --targets (-T) option.

--no-output

Do not produce a test results file.

-o file, --output file

Write full test results to file. Specify "-" (a hyphen) to write results to standard output. If neither this option nor --no-output is specified, the results are written to the file named results.qmr in the current directory.

-O file, --option file

Treat file as a set of expected outcomes. The file must have be a results file created either by qmtest run, or by saving results in the graphical user interface. QMTest will expect the results of the current test run to match those specified in the file and will highlight differences from those results.

-s file, --summary file

Write a summary of the test run to file. Specify "-" (a hyphen) to write results to standard output (the default).

QMTest prints a summary of test results, including statistics and the names of tests that did not pass. If expected outcomes were specified, the the names of tests that had unexpected results are printed, instead of the names of tests that did not pass.

-S, --no-summary

Do not produce a summary of the test run.

--seed integer

For each test run, QMTest randomizes the order in which tests are run, subject to the constraints described in Section 3.2.1. The random number generator is seeded using the system time. This maximizes the chance of detecting unanticipated dependencies among tests across multiple test runs.

For debugging purposes, it is sometimes necessary to obtain a reproducible sequence of tests. Use the --seed option to specify the seed for the random number generator.

Note that even with the same random number seed, if tests are run on targets with a concurrency greater than one, scheduling uncertainty may still produce variation in the order in which tests are run.

-T file, --targets file

Use targets specified in target specification file file. See Section 3.6.2 for a description of the target file syntax.

3.4.5. qmtest summarize

3.4.5.1. Summary

The qmtest summarize displays information stored in a results file.

3.4.5.2. Synopsis

qmtest summarize [ option ...] [test-name | suite-name...]

3.4.5.3. Description

The qmtest summarize extracts information stored in a results file and displays this information on the console. The information is formatted just as if the tests had just been run, but QMTest does not actually run the tests.

The summarize command accepts the following options:

-f format, --format format

For details about this option, see the description of the qmtest run command.

-O file, --option file

For details about this option, see the description of the qmtest run command.

3.4.6. Environment Variables

QMTest recognizes the following environment variables:

QM_PYTHON

If this environment variable is set, QMTest uses it as as the path to the Python interpreter. If this environment variable is not set, QMTest looks for a file named python in the bin directory where QM is installed. If this file does not exist, but /usr/bin/python2 exists, QMTest will use that path. Otherwise, QMTest searches for python in the directories listed in the PATH environment variable.

QMTEST_CLASS_PATH

If this environment variable is set, it should contain a list of directories in the same format as used for the system's PATH environment variable. These directories are searched (before the directories that QMTest searches by default) when looking for extension classes such as test classes and database classes.

QMTEST_DB_PATH

If this environment variable is set, its value is used as the location of the test database, unless the --tdb (-D) option is used. If this environment variable is not set and the --tdb option is not used, the current directory is used as the test database.