Test Automation Tool

One of the main priorities we have when developing Quimera Engine is to provide reliable code. We sacrifice development speed in exange for getting souce code that works exactly as we designed (it sounds logical, right?) so users know how and when to use a feature and what to expect from it, feeling they really have control over the engine. In order to assure this, we need to implement tons of tests for every function or method, taking into consideration as many kinds of input values as we can imagine; and not only that, the same tests may fail for different compilation configurations, different flag values (preprocessor directives), different compilers or even different operative systems. Just to let you get an idea of what magnitude are we talking about: currently there are more than 1600 unit tests for our Math library only. What a madness!

Yes, trying to execute all the test batteries for every flag value, compiler and operative system would force us to spend hours or even days. Luckily we are developers: we like automating things and hate repetitive work. Due to this necessity the Test Automation Tool was born. After working in a first approach, it was divided in 3 parts: the configuration file, where one can define the test system (includes an editor); the configuration window, which let us customize the test process using a simple UI; and the execution window, where the test execution is monitored and the results are displayed graphically.

Apart from the functional part, designing the infrastructure was not so easy due to the variety of operative systems, compilers and project file formats available with which the tool would have to work. It should be as flexible as possible but at the same time it should be easy to maintain (no duplicate code, not too much generic) and easy to extend since lot of improvements are to come for sure when we have time. To summarize, we end up with a tool that has the following characteristics:

  • Cross-platform: It works on Windows and Linux currently. To achieve this, we based it on wxWidgets 2.9.4.
  • Works with any kind of project file: Any kind of file format can be used as project file for the project under test or the test system, including makefiles.
  • Works with any compiler: It doesn't matter whether you use GCC, MinGW, MSBuild (VC++) or whatever you want.
  • Easy to adapt to any test system: We use the Boost's Unit Test Framework for testing Quimera Engine, which creates executable files that generate results in form of XML files. The Test Automation Tool expects this kind of test system but it wouldn't be hard to modify the source code to make it work with another one.
  • Reliable: The Test Automation Tool must pass several test batteries too! (you know, we have to watch the watchman...). It's worth to say that the frontend and backend layers are decoupled enough so we can easily test the functionality without worrying about the UI. This also makes the infrastructure almost independent from the visual representation.
  • Easy to extend: It's organized in 3 functional groups, totally isolated, and a common part, so it shouldn't be difficult to add a new functional group.
  • Simple: This can be seen as a feature or a defect. What we can say is that it's planned to improve it substantially (maybe you can see some of the future tasks in the issue tracker).
  • Multilanguage: It's translated to English and Spanish and it's very simple to add new translations using tools like poedit.
  • Its GUI can be modified using a visual designer: We used wxFormBuilder for this purpose, which generates C++ code automatically from the graphic design. Since our own code is separated from the autogenerated code, you can overwrite it as many times as you need without having to re-apply our customizations.

So how does it works?

Let's think about a real case. In a Windows environment we can compile Quimera Engine using VC++ or MinGW. The engine can also be configured at compile time using some flags (preprocessor directives) which can take 1 to N different concrete values. Besides, there is one project to be used along with each compiler: one is a Visual Studio solution (.sln) and the other is a makefile (.mak). And finally, there are 2 test projects, of the same type than the 2 mentioned before, to be compiled with the same compilers.

The first thing to do is to define the configuration file respecting the expected format. This has to be done only the first time. There you define where are the compilers, which options do they expect, where are the project files, the test system, the test result files, etc. After that, you only have to open the tool, select what do you want to test from some lists and start the execution. If all was correctly set up, the application will compile every project with its corresponding compiler for every flag value combination; then it will test them and will display the results on the screen.

Great, isn't it?

Further information

You can find more information in the wiki (take a look to the user manual), or you can just ask if you prefer.

You can also download the source code from the repository, see how in the wiki.

To know about the software license, please refer to the license section.

We hope this is useful for you.

Working in darkness

Although Quimera Engine has just come to light, it hasn't sprout suddenly. A lot of work and sacrifice has been necessary which is not easy to assess nowadays. How did we got here? Have we actually progressed that much? There is lot of work to do for sure, but we have managed to create a solid infrastructure to advance in a more secure way, we have sown the seed of what will become a steady and prolific tree.

It's about a long-term investment. It's clear that, if we had focused our efforts on the development of the API only, now we would be watching triangles and points dancing in the screen, we would be implementing the sound engine, the collisions or the scripts. But would it bear that structure the weight of the time? Would a mediocre code and design, which doesn't assure that will behave as expected in the 99% of cases, be adopted? Would somebody use it if not every functionality is documented? Writer's opinion is clear: We would sunk in a short period of time. This project is not Facebook, it's not Youtube; leaving the difference bewteen offering an API or a service application in the Net to one side, this is not a super-novel product that can break the market and generate enormous profits which could be spent to rebuild the entire library later, with more quality. This project requires patience and perseverance to pay attention to details and be able to plan ahead for the problems.

Following, a very summarized list of the main project's milestones, in chronological order:

2010 Creation of the first designs
Conversations with potential collaborators
Register of the project in Google Code
Creation of the development forum
13/08/2010 Creation of the SVN repository in Google Code
08/2010 a 09/2010 Start of the wiki creation in Google Code
09/2010 a 11/2010 Creation of the proofs of concept for different libraries for their evaluation with the aim of deciding which one to include in the project. This includes testing Boost libraries (multithreading, logging, serialization), POCO (multithreading, logging), PugiXML (XML), Xerces (XML), TinyXML (XML), s11n (serialization) and Eternity (serialization)
17/09/2010 Creation of the Add-in for documenting automatically C++ code in XML format, used by .Net in Visual Studio 2010, with the aim of standardizing the documentation that will be parsed by Doxygen.
03/11/2010 First lines of code uploaded (project structure for Visual Studio 2010)
08/11/2010 First task description documents created (QBaseVector3)
10/11/2010 First code-related task completed by a collaborator (QBaseVector3, by Jwladi)
24/11/2010 DataTypes library development begins
17/01/2011 Creation of the testing projects and the first unit tests (for QVector3)
19/01/2011 Task #100 created
23/01/2011 MathJax introduced to improve documented mathematical formulas with Doxygen. A plug-in was also included for generating formulas in the forum.
02/2011 Mutual inclusion problems solved in Math.
03/2011 Linux 32 bits portability achieved using Code::Blocks 10.05
01/03/2011 Beginning of the implementation of geometry-related classes of Math library
24/10/2011 Task #200 created
12/2011 Intensive code cleaning, homogenizing and reviews.
01/2012 Beginning of the unit tests creation for the Math library by collaborators
14/01/2012 Creation of an application for testing the Math library with 2D graphics
04/2012 a 05/2012 wxWidgets libraries research
14/06/2012 Test Automation Tool's structure creation
21/06/2012 Buy of quimeraengine.org domain. Beginning of the design of the portal with Joomla
25/06/2012 Task #300 created
26/06/2012 Beggining of the design of the wiki with Dokuwiki
29/09/2012 Creation of the testing environment for the Test Automation Tool
28/07/2012 The development of Math is done (1600+ unit tests implemented)
08/2012 The development of the Test Automation Tool begins
09/2012 Completion of the design of the portal, the wiki and the API reference
10/2012 Completion of the contents writing for the portal
13/10/2012 Webs registered in Google Analytics
20/10/2012 Added the notification bar to the wiki
21/10/2012 Public forum's layout / style finished
18/01/2013 Test Automation Tool development finished
30/01/2013 Planned wiki contents written, in Spanish
11/02/2013 Wiki contents translated into English
15/02/2013 The web portal, the public forum and the wiki were published.

Ignition

Hello world!

At last we have reached our first goal, we have something to demonstrate our work's reliability and a place to show it to the world. This is the first time, after a lot of work in the darkness, that we make the Quimera Engine project public and we announce it with fanfare. We are very excited about the idea of other professionals being able to participate in the project and making it grow up to all directions, until a solid community is created that makes Quimera Engine a more useful and reliable product every day.

Stay tuned

This website will be the central point to contact with the project, anyone can be informed daily about our progress through periodical publications on the main section, which will act as a blog (RSS link available). Besides, many publications will be also replicated in the Facebook wall and the project's Twitter account.

Participate

If apart from being informed about what's being cooked in our kitchen you want to give your opinion, make proposals, ask your questions, etc., you can use our forum, where we will respond as soon as possible. For a direct and private contact, you can write an email to This email address is being protected from spambots. You need JavaScript enabled to view it. .

Thanks to the team for making it real.

Spread it!


Kinesis Team.

Twitter