Merilo

What

Merilo (russian: "measurement") is an automated toolkit that helps you test non-functional requirements of web sites. That is: All the aspects of your site that unit tests normally do not cover. Some examples:

• markup validation
• performance and scalability
• simple content errors
• character encoding problems
• invisible errors that write something in the log
• resource usage and release (memory, cpu, threads, connections etc)

Example output

Have a look here.

Why

We strongly believe that testing these aspects in a continuous manner in a project is just as important as executing functional (unit) tests. There is much more to a running system than the sum of its unit tests, and we would like to discover also these kinds of issues at an early stage. We have so far seen few toolkits that do that, so here is our go at it.

The toolkit consists of various open-source libraries and executables that does most the job for us. We have provided the glue code and worked on finding the right tool for each subtask.

How

We start by running a client against the system as a whole, using a relevant protocol (http, for example). The client must generate traffic that is reasonably realistic (a crawler can be used to help that). During testing, we gather various kinds of output from the system, such as application and server logs, cpu and memory usage, system responses and response metadata, etc. We then put this information into simple but easily readable reports.

The toolkit is pipeline-based. Each step in the pipeline consists of executing a command-line program that does some processing and stores the output in a text file. The following step reads that output, executes a new program that does some processing, and again that output is stored in another text file. This orchestration of information gathering and processing turns out to be a simple, but very powerful tool. By the way, a detailed description of example steps in the pipeline is available here.

We use ant as the overall script container due to easy integration with CI tools that support automation. Each step in the ant script calls a shell script that in turn runs an executable and/or does text parsing.

Installation

Descriptions on how to download the code from our Subversion repository is here. Please use the unix2/ subdirectory in the Subversion repository as it contains the most recent version. The code runs well on Ubuntu 8.04. There are READMEs in the root directory and subdirectories that provide further help. Please note that using all the features requires installing Java, Ant, Ruby, FireWatir, Siege and HTMLTidy. The last two need to be compiled by yourself.

Please note that this is very much work in progress and that we currently try to reduce the number of dependencies (Ruby, Java, grep, awk, etc) on both the Windows and *nix platforms. Quite likely we will end up having two separate implementations due to the various limitations and incompatibilities in scripting and text parsing across Windows and *nix.

Licensing

Go here for an overview of the licences in use.

History

The first version of Merilo was developed at The Research Council of Norway and has been kindly donated to Open Source.

Comments ( Hide )