Building a visual regression testing tool in Jenkins

In what seemed like a blink of an eye, late May had turned into mid-August. My time as a trainee in the Exove trainee program was reaching its end. This also meant that I was up to prepare a presentation on something I had done, a highlight of my learning experience so far. I decided to share my progress on an internal testing tool I had designed and developed during July, whenever I wasn’t involved with a client project.

Where it all started

There were hopes among coworkers to have a form of visual regression testing for client projects. Some earlier, internal prototypes had been created before, but the code behind them was largely obsolete and undocumented. Since client work can get sparse around the peak holiday season, I was given the task to research a new solution in the meantime.

Being a test engineer, one would usually choose from existing tools, rather than build their own. After a bit of research on feasible methods that would also match my personal skill level, I managed to find a solution that I felt comfortable attempting.

In a nutshell, the application browses a website and captures screenshots of each page it visits. These screenshots are organized in a gallery. When the application is run a second time, it produces a new gallery, which is then compared to the original. If differences are detected between the old and new screenshots, the user will be notified.

Development

The browsing part of the application was very easy to come up with. During my trainee period I often found myself doing browser automation for testing purposes with Selenium. It was an obvious choice to use it once again, with Robot Framework making the development process faster.

Dealing with the image files and folder structures was done by adding some bespoke Python code along the built-in functionality. The trickier part was to figure out an image comparison algorithm. After some searching, I settled on ImageMagick. It was quick to learn, and it runs directly from a command line, making implementation painless.

Where to host the application?

The original version of the application required a local installation, with all the technologies used to create it. This felt too tedious, so I had to come up with a hosting solution that wasn’t too excessive for this simple program.

Then I had an idea. Continuous integration tools, such as Jenkins, are no strangers to use cases involving all sorts of test automation. What is more challenging is to build a standalone, stable application to entirely run within the service itself, in place of something more disposable, usually based on one-off builds. Even though it is a somewhat unusual environment for my needs, it still seemed plausible the more I looked into it.

Since it is possible to keep the same workspace environment between builds, I could indefinitely call the program without losing the image galleries in between. Jenkins has a built-in feature for hosting HTML-based reports after the software has run, so I could build my own test reports as HTML files. Each run can be supplied with custom parameters, so the run configuration could be submitted to a graphical form instead of using only a command line. Finally, by installing all required technologies to Jenkins itself, there was no need to install anything as a user.

How it worked out

After doing some modifications to accommodate the existing code in the new environment, I was finally ready to see how it turned out. The end result is best described as a proof of concept. It does the intended job, but the current version still has lots of room for further development.

See my presentation about this project here.

Overall, I enjoyed building this application. It allowed me to expand my skill set beyond my usual duties and there were a lot of valuable lessons to learn regarding design, programming and ease of use. But in the end, I created something on my own that can also be useful to others, which I find very fulfilling.

Thoughts by