Phase 3 ends

After thousands of battles and hundreds of working hours throughout almost 3 months, the development Phase 3 of Quimera Engine comes to an end. Many new features have been added, most of them related to operating system abstraction and usability improvements on existing components; compilation times and file sizes, along with the design in general have been improved. Progresses are described in detail below. I'm very glad I could perform many of the improvements I had postponed since the beginning. Now the engine as a whole is both more robust and more usable, although there is so much work pending yet.

This stage has been a bit special: it's been the first one I've developed alone from the start to the end. It's been interesting due to the possibility of comparing the way of working and the amount of progress, the pros (like being able to change the design quickly) and the cons (like having to review your own code or not having the possibility of asking opinions). I'm neither going to talk in this post about the reasons that made the team diminish from up to 7 people to 1, nor about my experience managing a team of people who didn't know each other and were dispersed all along the country, each one with its own personal problems and objectives of life, in a barely incipient project with an abstract future; someday I'll write about that to share it with those who could be interested. Apart from the errors any of us committed, some of them avoidable and some of them inevitable, I keep all what I've learned and have confirmed that there are more people out there who seek to fulfill themselves and are capable of enlisting in crazy projects like this. It's a comforting thought for me. "If you want to walk fast, walk alone. If you want to walk far, walk together", some people say; my intention in the beginning was to walk far so I looked for company; what that phrase does not tell is "...and prepare provisions to feed your company during the entire trip, an official map, a compass and a beautiful postcard from the place you are going to, otherwise only Dora the explorer will follow you". Looking at it with some perspective, it doesn't discourage me: I think we went a long way.

Now that I'm walking alone, it's time to change the way I do things and embrace the advantages it implies regarding agility. I already changed the road map in the past, postposed the development of certain less urgent components of Common and Tools layers and advanced others required to develop higher layers; it's the time to jump to the most interesting and satisfactory part of all: graphics. The next phase will consist in the creation of a graphics engine prototype based on OpenGL that will be later integrated in Quimera Engine. It will be a proof of concept, not a "proof of production" (somebody shall understand what I mean); once it implements some minimum functions a clean version will be re-written, a user interface will be designed and it will be joined to the rest of the project as a plug-in. I'll provide more details in posterior posts.

To the point

Next some numbers extracted from the project, as done in the end of the previous phase:

  • 274.856* code lines, without either blank lines or file header comments, counting the test code.
  • 84.544* code lines, without either blank lines or file header comments, not counting the test code.
  • 35.000 code lines, approximately, written during this phase.
  • 461* source code files, counting the test code files.
  • 276* source code files, not counting the test code files.
  • 196* classes, 40 of them created during this phase.
  • 43%* of the code lines (without tests) are internal documentation or public documentation.
  • More than 7.750 unit tests in total, 650 of them written during this phase.
  • 155 revisions in Subversion during this phase (the code is only uploaded when the task is completely done, including unit tests, and after it has passed the revision).
  • 85 tasks completed during this phase.
  • More than 542 tasks finished until now in total, from 702 created in the tracker.

*These numbers were obtained using the Source Monitor tool.

Summary of the finished work

New classes

  • Call stack tracing tools: QCallTrace, QCallStackTrace, QAbstractCallStackTracePrinter, IQCallStackTraceFormatter, QCallStackTracePlainTextFormatter, QCallStackTraceConsolePrinter, QCallStackTracer, QScopedCallTraceNotifier, QTypeWithToString, QTypeWithGetObjectType, macros para facilitar el uso de este mecanismo interno.
  • Iterators QConstHashtableIterator, QConstDictionaryIterator.
  • Timing: QStopwatchEnclosed.
  • File system: SQFile, SQDirectory, QFileStream.
  • IO: QBinaryStreamWriter, QTextStreamReader, QTextStreamWriter.
  • Data types: QArrayBasic, QArrayResult.
  • Containers: QHashtable, QDictionary.
  • Container components: QKeyValuePair, SQStringHashProvider, SQIntegerHashProvider, SQKeyValuePairComparator, SQNoComparator, SQEqualityComparator.
  • Threading: QThread, SQThisThread, QMutex, QSharedMutex, QRecursiveMutex, QScopedExclusiveLock, QScopedSharedLock, QConditionVariable, QScopedLockPair.
  • Others: QEvent, QReferenceWrapper, QAssertException.

Improvements

  • QDynamicArray and QFixedArray were renamed to QArrayDynamic and QArrayFixed, respectively. This makes them more discoverable, since the user expects arrays to be called "Array" + something. Although it may seem a simple decision, it wasn't, due to it also has negative consequences: we often use the "control+spacebar" combination for the contextual help to complete the name, and then we press "Enter" to continue writing; there are 4 classes whose name starts with "QArray" now, so it is necessary to type all those letters before we can choose the class we want, whereas it was enough to write "QDy" or "QFix" to obtain the full name before. The readability improvement is more subjective in this case although reading the words in such order it is less intuitive for English speakers.
  • Enumerated types refactoring. The dependency on STL containers have been removed (the intention is, in the whole project, not to depend on the STL at all). Now enumeration classes are simpler, faster and take less time to compile. Besides, now they don't necessarily depend on the string_q type, although they can; this allows the string_q type to include in its own header file the enumerations it needs, so the user doesn't need to worry about including them every time he needs them (obviously, this has a small impact on the compilation time that, apparently, is worth).
  • Basic data type to string conversion methods refactoring. There are specific static classes for every basic data type, this means, SQInteger for integers, SQFloat for floating point types, SQBoolean for booleans and SQVF32 for 128 bits vectors. So far, every class provided a ToString method to convert the corresponding type to its representation as a character string. Now all that functionality has been moved to the QStringUnicode class as overloads for operator+ and Append, and From*** static methods. This way, 2 things are achieved: First, it is not necessary to include the QStringUnicode class if only basic data types operations are needed; second, concatenating variables is much easier and more readable not having to write method calls when, for example, one wants to log a message. E. g.: QE_LOG(myString + 5 + " + " + 3 + " = " + (5 + 3) would result in "Sum: 5 + 3 = 8".
  • Internal RTTI system refactoring. The internal RTTI system allows a remarkable reduction of the required space to store information about the data types utilized in the project, letting us to know and compare some types even dynamically, when integrated classes are polymorphic. So far the intention was to implement this in a simple manner enforcing all the classes integrated in the mechanism to inherit, directly or indirectly, from the QObject root class. This solution required the use of virtual inheritance to break deadly diamonds which, apart from any discussion about its usefulness or its danger, prevented the correct resolution of polymorphic types as it was implemented. Now QObject is no more (as well as SQTypeExtensions) and macros are used in the base classes instead.
  • Small improvement of QDateTime's interface. So far it wasn't necessary to specify a time zone to create dates and times, using UTC by default, this means, setting the value of the argument corresponding to the pointer to QTimeZone to null. This caused the constructor overloads to overlap due to the implicit conversion of integer to pointer that takes place then the value is zero. To solve such ambiguity, the time zone will be mandatory always. It's a usability loss in exchange for avoiding potential serious problems in the client code.
  • Small usability improvement of QStringUnicode. Contrary to the previous case, some default values have been added in some methods for them to be easier to use. For example, in the past it was necessary to provide the comparison type to be used when calling Contains, and now it is understood to be binary and case-sensitive (the fastest and the most frequent) so we can, simply, write something like myString.Contains("myPattern").
  • Multi-process compilation in Visual Studio. Now it seems that all the projects compile in half the time
  • Exception handling and RTTI support deactivation. Quimera Engine uses its own RTTI system, lighter and faster, only in the parts where it provides some value. Regarding exception handling, the only exception thrown by the engine is the one thrown by the asserts when they are configured that way, uniquely for testing purposes. It's accepted as a rule of thumb that a videogame works perfectly or doesn't work. Unexpected errors (those not found during development) will be traced, when possible, and the program will terminate. All the reasons to make this decision are written in the Quimera Engine's Design Foundations (currently out of date, by the way). Boost libraries had to be recompiled with the corresponding options and some header files had to be fixed in situ in order to make them compile, until a better solution is found. As an important addition, the size of all libraries have been reduced considerably, as it can be seen in the comparative table below.
  • Improvement of machine endianness detection utilities. Some definitions for detecting whether the machine is either Little-endian or Big-endian at compile time have been added. Some functions to check the same at runtime were added too.
  • Migration of the Test Automation Tool from Code::Blocks to CodeLite and port to Linux and Mac OS X. So far, the test automation tool created ad-hoc for executing the Quimera Engine's unit tests was mounted on a Code::Blocks project and had never worked on Mac. Now it's mounted on CodeLite (Code::Blocks is not used anymore for anything) and works on all the supported operating systems.
  • Creation of a tool for converting CodeLite workspaces to makefiles automatically. Updating the makefiles of all the projects every time a source code file was added or a compiler option was changed was a pain in the neck and, most of all, a risk. In order to avoid that I created a small tool in C# (fast development, no portability needed) which transforms CodeLite projects and workspaces into ready-to-execute makefiles, with a pair of mouse clicks. The saving of time is considerable. More details can be found below.
  • General warnings cleaning. They are like bad weeds, they must be kept at bay. Warnings lose their usefulness if they are not kept at zero, when possible. Besides, it's necessary to prevent them from propagating to the library user's project. This happens, just recalling a recent example, with Boost, which forces you to include its libraries as if they were system libraries to avoid a torrent of third party warnings infesting the build log.
  • Support for SIMD instructions and types added to projects. Now when compiling with GCC/Clang too.
  • Partial optimization of QArrayFixed and QArrayDynamic. As I anticipated in the previous end of phase post, I've performed some optimizations in container classes. The most affected class is QArrayFixed, which has the same efficiency as std::vector when iterating (in the end it is boiled down to an integer increment and a dereferencing to obtain the actual value). I've not dedicated so much time to make performance experiments, which I would love to do, but I did some concrete tests and I can say that QList is faster than std::list, for example. I hope I have time to post a demo in the future.
  • Usability improvement of methods which return arrays. Some methods, like QStringUnicode::Split, return multiple results whose amount is unknown. The typical solution is to pass a pointer and an integer as output arguments. To improve both usability and readability, I decided to return a small structure which contains both the number of elements and the array, whose memory has been allocated inside the method. The structure is attached to the content so, when its destructor is called, it will free the memory occupied by the array so the user hasn't got to worry about that. In order to avoid creating something similar to unique_ptr (with the problems it would imply) it's been remarked that the usage of such structure is limited exclusively to the return of this kind of results and hence its name: QArrayResult. The structure can be detached from the content manually, calling Detach. The addition of overloads that receive a pointer (the same to be returned in the structure) to give the possibility of providing a preallocated memory block, increasing the performance, is not discarded.

Pending work

Classes

  • Threading: QConditionVariableExclusive.
  • Hashed strings: SQStringPool, QHashedString

Improvements

  • Mechanism to enable/disable assertions by type (error, warning or deprecation).
  • Memory allocation tracing.
  • Quit STL dependency in QLocalTimezone.
  • Conversion of some container's methods to templates, so they can interact with each other.
  • Add ForEach methods to containers to execute a function per element.
  • Add low priority methods to many of the existing classes.
  • Research on whether it is possible to replace time zone libraries of Boost with ICU's, which seem to be more complete.

 Comparative tables of libraries sizes when compiled with and without both RTTI and exceptions

The following 2 tables show the difference among the libraries before and after the changes. The explanation of every table appears under each one.

  Before After Reduction Total
QuimeraEngineCommon.dll 554 kb 447 kb 20%  
QuimeraEngineTools.dll 2068 kb 1741 kb 16%  
QuimeraEngineSystem.dll 1308 kb 1070 kb 18% 17%
QuimeraEngineCommon.lib 4635 kb 3660 kb 21%  
QuimeraEngineTools.lib 12100 kb 9756 kb 20%  
QuimeraEngineSystem.lib 14254 kb 9728 kb 32% 26%

Compiled with Visual Studio 2010, DebugWin32Sharedrt configuration. The results of this first table are not completely reliable due to, in order to make the libraries compile with the new configuration, some changes were made to the code, which may affect the final size. However, differences are significative enough to assure that both deactivations produced smaller binaries.

  Before After Reduction Total
QuimeraEngineCommon.dll 315 kb 259 kb 18%  
QuimeraEngineTools.dll 1838 kb 1558 kb 15%  
QuimeraEngineSystem.dll 1015 kb 842 kb 17% 16%
QuimeraEngineCommon.lib 2188 kb 1778 kb 19%  
QuimeraEngineTools.lib 9156 kb 7368 kb 20%  
QuimeraEngineSystem.lib 10700 kb 6971 kb 35% 27%


Compiled with Visual Studio 2010, DebugWin32Sharedrt configuration. This table shows, some time later, the reduction due only to the deactivation of exceptions, not modifying any code between both samples. Up to 35% depending on the file.

Since the moment I started working on the task until I finished it, the total size decreased around 25-30%.

CodeLite projects to Makefiles converter

This is a brief presentation of the little tool created to generate totally functional makefiles from CodeLite IDE 6.01 workspace and projects.

1. The workspace is selected:

2. When selected, all its content is loaded into the screen, allowing the visualization of every compilation configuration (using the combo box):

3. Makefiles are generated for every configuration of all the projects in the workspace:

 


Links to the project's Wiki for more information

API reference documentation

State of the project, detailed

Road map

Proposals table

P.D.: I've to learn to ration the information in several posts, otherwise it is impossible to read.

 

You have no rights to post comments

Twitter