All QMTest functionality is available using the qmtest command.
qmtest [ option ...] command [ command-option ...] [ argument ...]
These options can be used with any QMTest command, and must precede the command name on the command line.
All options are available in a "long form" prefixed with "--" (two hyphens). Some options also may be specified in a "short form" consisting of a single hyphen and a one-letter abbreviation. Short-form options may be combined; for example, -abc is equivalent to -a -b -c.
Use the test database located in the directory given by path. This flag overrides the value of the environment variable QMTEST_DB_PATH. If neither this flag nor the environment variable is specified, QMTest assumes that the current directory should be used as the database. See Section 3.1.6.
Display help information, listing commands and general options for the qmtest command.
Generate progress and status messages while executing. This option may be specified more than once; the more times it is specified, the higher the verbose level, and the more messages are printed.
Additional options are available for specific commands; these are presented with each command. Options specific to a command must follow the command on the command line. Specify the --help (-h) option after the command for a description of the command and a list of of available options for that command.
Create a new test database.
qmtest create-tdb [ option ...]
The qmtest create-tdb command creates a new, empty test database. A test database is a directory in which QMTest stores configuration files, tests, and other data. Certain test database classes may also store data elsewhere, such as in an external relational database.
The test database is created in the directory specified by --tdb (-D) option or by setting the QMTEST_DB_PATH environment variable. The path you specify for the new test database must not exist. (If no database path is specified, QMTest assumes that the current directory is the test database. Since the current directory already exists, QMTest will issue an error. Therefore, when using the create-tdb command, the database directory must be explicitly specified.)
By default, QMTest creates a new test database that uses the standard XML-based implementation. (See Section 3.8 for information about writing a test database class.)
The create-tdb command accepts these options:
Set the database attribute name to value. The set of attribute names and valid values is dependent on the database class in use. The default database class accepts no attributes.
Use the test database class given by class. Once you create a test database, you cannot change the test database implementation it uses. If you do not use this option, QMTest will use the default test database implementation, which uses an XML file format to store tests.
Start the graphical user interface.
qmtest gui [ option ...]
The qmtest gui starts the graphical user interface. The graphical user interface is accessed through a web browser. You must have a web browser that supports JavaScript to use the graphical interface. QMTest has been tested with recent versions of Internet Explorer and Netscape Navigator. Other web browsers may or may nor work with QMTest.
The gui command accepts these options:
Bind the server to the indicated internet address, which should be a dotted quad. By default, the server binds itself to the address 127.0.0.1, which is the address of the local machine. If you specify another address, the server will be accessible to users on other machines. QMTest does not perform any authentication of remote users, so you should not use this option unless you have a firewall in place that blocks all untrusted users.
For details about this option, see the description of the qmtest run command.
For details about this option, see the description of the qmtest run command.
For details about this option, see the description of the qmtest run command.
Do not attempt to start a web browser when starting the GUI. QMTest will still print out the URL at which the server can be accessed. You can then connect to this URL manually using the browser of your choice.
Specify the port on which the QMTest GUI will listen for connections. If this option is not provided, QMTest will select an available port automatically.
For details about this option, see the description of the qmtest run command.
Run tests or test suites.
qmtest run [ option ...] [test-name | suite-name...]
The qmtest run command runs tests and displays the results. If no test or suite names are specified, QMTest runs all of the tests in the test database. If test or suite names are specified, only those tests or suites are run. Tests listed more than once (directly or by inclusion in a test suite) are run only once.
The run command accepts these options:
Add a property to the test execution context. The name of the property is name, and its value is set to the string value.
This option may be specified multiple times.
Read properties for the test execution context from the file file.
The file should be a text file with one context property on each line, in the format name=value. Leading and trailing whitespace on each line are ignored. Also, blank lines and lines that begin with "#" (a hash mark) are ignored as comments.
This option may be specified more than once, and used in conjunction with the --context option.
Control the format used when displaying results. The format specified must be one of full, brief, stats, or none. The brief format is the default. In the full format, QMTest displays any annotations provided in test results. In the brief mode only the causes of failures are shown; detailed annotations are not shown. In the stats format, no details about failing tests are displayed; only statistics showing the number of passing and failing tests are displayed. In the none mode, no results are displayed, but a results file is still created, unless the --no-output option is also provided.
Run tests in multiple count concurrent processes on the local computer. On multiprocessor machines, the processes may be scheduled to run in parallel on different processors. QMTest automatically collects results from the processes and presents combines test results and summary. By default, one process is used.
This option may not be combined with the --targets (-T) option.
Do not produce a test results file.
Write full test results to file. Specify "-" (a hyphen) to write results to standard output. If neither this option nor --no-output is specified, the results are written to the file named results.qmr in the current directory.
Treat file as a set of expected outcomes. The file must have be a results file created either by qmtest run, or by saving results in the graphical user interface. QMTest will expect the results of the current test run to match those specified in the file and will highlight differences from those results.
Write a summary of the test run to file. Specify "-" (a hyphen) to write results to standard output (the default).
QMTest prints a summary of test results, including statistics and the names of tests that did not pass. If expected outcomes were specified, the the names of tests that had unexpected results are printed, instead of the names of tests that did not pass.
Do not produce a summary of the test run.
For each test run, QMTest randomizes the order in which tests are run, subject to the constraints described in Section 3.2.1. The random number generator is seeded using the system time. This maximizes the chance of detecting unanticipated dependencies among tests across multiple test runs.
For debugging purposes, it is sometimes necessary to obtain a reproducible sequence of tests. Use the --seed option to specify the seed for the random number generator.
Note that even with the same random number seed, if tests are run on targets with a concurrency greater than one, scheduling uncertainty may still produce variation in the order in which tests are run.
Use targets specified in target specification file file. See Section 3.6.2 for a description of the target file syntax.
The qmtest summarize displays information stored in a results file.
qmtest summarize [ option ...] [test-name | suite-name...]
The qmtest summarize extracts information stored in a results file and displays this information on the console. The information is formatted just as if the tests had just been run, but QMTest does not actually run the tests.
The summarize command accepts the following options:
For details about this option, see the description of the qmtest run command.
For details about this option, see the description of the qmtest run command.
QMTest recognizes the following environment variables:
If this environment variable is set, QMTest uses it as as the path to the Python interpreter. If this environment variable is not set, QMTest looks for a file named python in the bin directory where QM is installed. If this file does not exist, but /usr/bin/python2 exists, QMTest will use that path. Otherwise, QMTest searches for python in the directories listed in the PATH environment variable.
If this environment variable is set, it should contain a list of directories in the same format as used for the system's PATH environment variable. These directories are searched (before the directories that QMTest searches by default) when looking for extension classes such as test classes and database classes.
If this environment variable is set, its value is used as the location of the test database, unless the --tdb (-D) option is used. If this environment variable is not set and the --tdb option is not used, the current directory is used as the test database.