Checklists

Checklists are everywhere. The GTD crowd lives by them. A checklist is a tool in testers arsenal when used well makes mundane tasks easy to handle, complex tasks more structured and overall provides systematic way of handling testing.

Marathon Checklists

Marathon brings the goodness of checklists to the automation - making it possible to create semi automatic test scripts.
Marathon checklist is a form containing either checkbox items (along with an explanatory text area) or plain text areas. The forms are stored as plain XML files. These can be created through an (rudimentary) UI while recording or using you favorite text editor.
While recording a test script, you can insert a checklist action. You will be prompted with the existing forms and you can either select a predefined form or create a new one.
While playing a test script with a checklist action, the user will be presented with the form. The form will have options to succeed or fail the test script. In both the cases, the results are written into a XML file. At the end of test run, you can create a checklist report that formats the results and display either in text or html format.
The checklist actions by default are ignored during batch run (from commandline or JUnit view). You can enable the checklist actions by passing a command line parameter (for batch run) or a menu option (for JUnit run).

Exploratory Testing With Marathon

Having the checklists is good and we can make it better by adding an exploratory test mode. The exploratory test mode is similar to recording mode (may be the same option) - where the application is started and the script is recorded. But the recorded script is not used for playback. It is used to provide the step by step operations for reproducing a bug. During this mode, you have an option of adding a test failure. When using this option, you can either provide free form text or use an existing checklist. Marathon will store the results including the recorded part of the script in a XML file. A report option provides the Text or HTML view of the results.
Another feature of exploratory testing mode - for a later release - is the ability to add a window screen shot and making annotation markings on the screenshot image. This screenshot will also be added to the results.

How does these features sound? Do you think these will be useful? Please leave feedback in the comments.

Marathon to the rescue.

We named each of the components and panels that we are adding (using setName) and ran the application under Marathon (version 2.0b4 - yet to be released). The new version has a different assertion mechanism where by the component selected is highlighted.

Assertions

You can see the semi-transparent Address field? Marathon by default shows each of the recognizable components by highlighting them. It is a nice side benefit.

Updated. Thanks to Arun for giving me a pointer to pstools.

This will be part of Marathon User Guide when it comes out. I am writing these snippets so that you can benefit from them now.

Now that you wrote a lot tests using marathon how do you run them? The problem with GUI testing is that your desktop is tied up for the duration of the tests. The monitor blinks like a tube-light with a starter problem with Windows popping up all over with crazy titles - ‘testStartAndStopRecordingWithSetupAndTearDown’ or ‘testDeleteFilesWithSomeFailures’. Even if you can live with this - all these popping up windows will be stealing your application’s focus (beside stealing yours) and some of your keystrokes and mouse clicks will go there. In the end you end up not knowing whether a test case failed because of you using the keyboard or a bug in the application under test.

Using java.awt.headless

Java since 1.4.2 supports a headless mode that is half-baked and not useful for the purposes we are talking about. Googling for ‘java headless’ turns up this page, so I am pointing this out. Java support for headless is really for applications that need to run without a display and keyboard. Anyhow if you do not need a display then the whole of this article does not make sense.

Running headless on Linux

An elegant solution I found to run GUI test suites on Linux is to use Xvbf - a X server that simulates a display, mouse and keyboard. On ubuntu, the platform of choice for me, the Xvfb package comes with a script xvfb-run. Executing an application under Xvfb is just typing:

xvfb-run marathon <path-to-your-mpf>

The standard error and output will be on your terminal, so you can see the progress of your test case execution and any errors thrown up by the test run.

Running headless on Windows

Using psexec

psexec is a small utility from Sysinternals from the pstools package. You can Download it here.

psexec allows you to run applications on remote machines. We will use it to run marathon headless on Windows. Download the pstools archive and install on the machine. Add the installed directory to the path. There are two ways you can execute Marathon - with a console and without a console.

For using it with a console (so that you can see the progress):

psexec -s cmd /c marathon -batch <path-to-your-mpf>

If you are not interested in a console:

psexec -s -d cmd /c marathon -batch <path-to-your-mpf>

Note that -s uses the system account (i.e the administrator) to execute the command. I found it easier to create a batch file that runs Marathon with all options and execute the batch file using psexec:

psexec -s cmd /c <path-to-batch-file>

Using telnet

You can run the tests headless on Windows using the telnet service that is available on Windows XP Professional. You can enable telnet service from Control Panel -> Services. Once you enable the service log-in using the command:

telnet localhost

Give your user name and password. You will land up at C:> prompt. Run your command:

marathon -batch <path-to-your-mpf>

That is it. Your display, keyboard and mouse belong to you, no annoying pop-up windows to disturb you and Marathon chugs along running the application tests. You can use -text, -html or -xml arguments along with -batch so that the test results are captured in a report.

Note:
Running a telnet server is insecure. Modify your windows firewall settings to disable incoming connections. Default is to disable incoming connections from Internet - it should be OK.

Note:
It will be possible to automate the log-in part provided we have a telnet client that can take input from a file and send the commands to
the server. I don’t know of any. Drop me a line if you know such a client for Windows.