0.2.x precompiler headers & build

[daniel.santos]

This is a follow up from the discussion here https://forum.megaglest.org/index.php?topic=4085.msg24333#msg24333 as I realized we were off-topic (and I'm trying to follow my own rules ).

So anyway, I've taken what you (bork) did with the Jamfile & such and I'm trying to improve upon it to make sure that pre-compiled headers are properly generated and used.  I've removed everything from the game/pch.h header and just have it include "../shared_lib/include/pch.h" as they are pretty much using the same stuff.  Then, any additional dependencies in game that aren't in shared_lib can be added to it.  I really appreciate the organization you did to pch.h!  Also, I've created a shared_lib/include/lang_features.h and moved the DELETE_FUNC definition there as it really didn't belong in the pch.h, but I didn't see any other reasonable place for it.  I'm considering moving the boost includes and the "#define foreach BOOST_FOREACH" there as well, although that's not precisely "language", though one can consider the functionality in boost to be similar to an extension of the C++ language.

Anyway, I'm going to try to setup the pre-compiled headers to be forced to build before anything else gets a chance so that -jx (with x > 1) doesn't cause problems.

[hailstone]

If you're going to be changing the build system it might be good to look at http://code.google.com/p/waf/ or http://www.scons.org/ or http://premake.sourceforge.net/about (CEGUI uses the last one)

I also read an article on using precompiled headers with gcc and it says there needs to be a file with the extension .gch

[bork]

This is a follow up from the discussion here https://forum.megaglest.org/index.php?topic=4085.msg24333#msg24333 as I realized we were off-topic (and I'm trying to follow my own rules ).

[snip]

I really appreciate the organization you did to pch.h!

Oh, it was just 'find . -name "*.cpp" -o -name "*.h" | xargs grep -oh '#include <.*>' | sort | uniq > p.h' actually.

[snip]

Anyway, I'm going to try to setup the pre-compiled headers to be forced to build before anything else gets a chance so that -jx (with x > 1) doesn't cause problems.

Well, looks like the patch was not so perfect Sorry, for error with -jx, I've used it as well, but I've used ccache also and it hided the problem with incorrect build order. I'm not any good with jam build system, so I have no good idea about fixing it right now. Main source of the problems is the usage of precompiled header from gletstadvlib by gletadv and testgae targets. I think that it might be possible to create a separate target for creating precompiled header for glestadvlib and use this target as a dependency in glestadvlib, glestadv and testgae. This looks quirky to me, although.

[bork]

If you're going to be changing the build system it might be good to look at http://code.google.com/p/waf/ or http://www.scons.org/ or http://premake.sourceforge.net/about (CEGUI uses the last one)

Hope he is not going to change the build toolchain, but only it's usage, I've just started to be acquainted with the jam

I also read an article on using precompiled headers with gcc and it says there needs to be a file with the extension .gch

Yes, this file is there actually, the problem is that jam thinks that it already created being in the middle of precompilation and starts building files dependent on the header. So it's a problem of targets organization in the Jamfile, not creation of precompiled headers itself.

[daniel.santos]

I spoke with some guys in the #boost chat room about this and they said it was one of the problems that they overcame in bjam, the inability to control dependencies.  I'm not sure what I did, but at some point, I started having massive linking errors (missing symbols) as if one of the libraries (glestlib and/or gamelib) wasn't being linked in and it was excruciatingly irritating   I finally found an alternate solution and I reverted the creation of the gamelib and just encapsulated the glestadv and testgae in an if/else statement.  I wish I had saved the intermediate state where I had it all working, but I got frustrated and have it working now and will probably leave it that way until a better solution comes along.

I do want to change the build system if Matthias wont be able to assist with issues like this, but I don't want to work on it now because I need to stay focused on completing  the networking re-write (it's most of the way there now!).  So for now, if you want to build the unit test, you run the following:

Code: [Select]

BUILD_UNIT_TESTS=yes jam
If you have previously built the game target (glestadv) then it will only compile the test code and then link.  What I still might do is try to make a build target that will only build the precompiled headers, then one could build that target and then, in a separate process build everything else with as many parallel jobs as you like.  Example:

Code: [Select]

jam pch && jam -j5Right now, I'm using two precompiled headers, one for shared_lib project and one for the game.  Currently, I have them including every header file in each of those as well as external dependencies.  Thus, if you change any header, it will cause them to be rebuilt, while at the same time, should contain most of the code that will need to be compiled.  So if they are working correctly, they should speed up compile times a good deal.

I should add the note that the Application rule accepts various options, one being "independent", this option causes the target not to be included in the default "all" target.  However, it still added the object files, go figure.

If you're going to be changing the build system it might be good to look at http://code.google.com/p/waf/ or http://www.scons.org/ or http://premake.sourceforge.net/about (CEGUI uses the last one)

Well, if we're going to change it, I want it to be to something that can do the job and is maintainable.  A standard autotools/make build is highly maintainable.  I'm not hard set on that, but I don't want to run into the types of problems that we have with this build system where it's hard to maintain.  Granted, Matthias' system has some incredible capabilities that it turns out aren't being exploited!  It has the ability to generate msvc project files, but I never knew that until after I already had one working on my system and I never tried to figure out how to work it.

[daniel.santos]

I have committed these changes as rev 306, along with a fix to GameSettings that was breaking saved games.  I've actually play tested this one!

[bork]

I was able to create correct dependencies for precompiled headers at last. Now all object files depend on the corresponding gch files. The only thing you need is to pass dependent libraries into Application or Library rule, e.g.

Code: [Select]

Application glestadv : $(GAME_SOURCES) : : $(GAME_PCH) : glestlib ;

I've tried this stuff with/without -jx and -sUSE_PCH=yes. There were linking errors when using precompiled headers due to the way of creating enumerations names. I've changed it so it's no more necessary to use GAME_CONSTANTS_DEF:

Code: [Select]

#define STRINGY_ENUM_NAMES(name, count, ...) \
       inline Shared::Util::EnumNames &name ## _() { \
               static Shared::Util::EnumNames instance(#__VA_ARGS__, count, true, false); \
               return instance; \
       } \
       static Shared::Util::EnumNames &name = name ## _();

The problem with previous approach is in inclusion of game_constants.h from pch.h, so it could be eliminated by removing glest headers from precompiled header, but I think, my fix is a cleaner way to achieve this goal

I was also able to use glestadv as library again, so no need for conditional selecting glestadv or testgae in Jamfile, hope it will work for you as well.

And now, without any further ado, is the patch itself.

Code: [Select]

diff -r 27c339fb2cda mk/linux/Jamfile
--- a/mk/linux/Jamfile  Sun Apr 19 09:47:40 2009 +0400
+++ b/mk/linux/Jamfile  Mon Apr 20 08:32:49 2009 +0400
@@ -6,14 +6,13 @@ if $(USE_PCH) = "yes" {
        CPPFLAGS += -DUSE_PCH=1 ;
        SHARED_PCH = shared_lib/include/pch.h ;
        SHARED_PCH_INCLUDE =  $(LOCATE_TARGET)/shared_lib/include ;
-       #CompilePchSingle $(SHARED_PCH) ;
        GAME_PCH = game/pch.h ;
        GAME_PCH_INCLUDE =  $(LOCATE_TARGET)/game ;
-       #CompilePchSingle $(GAME_PCH) ;
 }
 
 Package license.txt readme.txt ;
 
+local EXTERNAL_LIBS = SDL GL GLU XERCES VORBIS VORBISFILE OGG OPENAL Z ;
 
 #### Shared Library ####
 SubDir TOP ;
@@ -42,11 +41,10 @@ for i in shared_lib/include shared_lib/i
 }
 
 Library glestlib : $(SHARED_SOURCES) : : $(SHARED_PCH) ;
-ExternalLibs glestlib : SDL GL GLU XERCES VORBIS VORBISFILE OGG OPENAL Z ;
+ExternalLibs glestlib : $(EXTERNAL_LIBS) ;
 IncludeDir glestlib : $(SHARED_PCH_INCLUDE) $(SHARED_INCLUDE_DIRS) ;
 
-
-#### Game ####
+#### Game Library ####
 SubDir TOP ;
 
 GAME_DIRS =
@@ -67,41 +65,51 @@ GAME_DIRS =
 ;
 
 GAME_DIRS = game/$(GAME_DIRS) ;
-
-for i in $(GAME_DIRS) {
-       GAME_SOURCES += [ Wildcard $(i) : *.cpp *.h ] ;
+for dir in $(GAME_DIRS) {
+    GAME_FILES += [ Wildcard $(dir) : *.cpp *.h ] ;
 }
 
-if $(BUILD_UNIT_TESTS) != "yes" {
-       Application glestadv : $(GAME_SOURCES) : : $(GAME_PCH) ;
-       LinkWith glestadv : glestlib ;
-       ExternalLibs glestadv : SDL GL GLU XERCES VORBIS VORBISFILE OGG OPENAL Z ;
-       IncludeDir glestadv : $(GAME_PCH_INCLUDE) $(GAME_DIRS) $(SHARED_INCLUDE_DIRS) ;
-
-#### Test ####
-} else {
-       SubDir TOP ;
-
-       TEST_DIRS = test ;
-       for i in $(TEST_DIRS) {
-               TEST_SOURCES += [ Wildcard $(i) : *.cpp *.h ] ;
-       }
-       
-       for i in $(GAME_SOURCES) {
-               if ! [ MATCH (main.cpp) : $(i) ] {
-                       TEST_SOURCES += $(i) ;
-               }
-       }
-       
-       LDFLAGS += "-lcppunit" ;
-       
-       # Includes all game sources
-       Application testgae : $(TEST_SOURCES) : console : $(GAME_PCH) ;
-       LinkWith testgae : glestlib ;
-       ExternalLibs testgae : SDL GL GLU XERCES VORBIS VORBISFILE OGG OPENAL Z ;
-       IncludeDir testgae : $(GAME_PCH_INCLUDE) $(TEST_DIRS) $(SHARED_INCLUDE_DIRS) $(GAME_DIRS) ;
+for src in $(GAME_FILES) {
+    if ! [ MATCH (main.cpp) : $(src) ] {
+        GAMELIB_SOURCES += $(src) ;
+    }
 }
 
+Library glestadvlib : $(GAMELIB_SOURCES) : : $(GAME_PCH) : glestlib ;
+ExternalLibs glestadvlib : $(EXTERNAL_LIBS) ;
+IncludeDir glestadvlib : $(GAME_PCH_INCLUDE) $(GAME_DIRS) $(SHARED_INCLUDE_DIRS) ;
+
+#### Game ####
+SubDir TOP ;
+
+for src in $(GAME_FILES) {
+    if [ MATCH (main.cpp) : $(src) ] {
+        GAME_SOURCES += $(src) ;
+    }
+}
+
+LDFLAGS += -L$(LOCATE_TARGET) ;
+LIBS += -lglestlib -lglestadvlib ;
+
+Application glestadv : $(GAME_SOURCES) : : : glestlib glestadvlib ;
+LinkWith glestadv : glestlib glestadvlib ;
+ExternalLibs glestadv : $(EXTERNAL_LIBS) ;
+IncludeDir glestadv : $(GAME_PCH_INCLUDE) $(GAME_DIRS) $(SHARED_INCLUDE_DIRS) ;
+
+SubDir TOP ;
+
+TEST_DIRS = test ;
+for i in $(TEST_DIRS) {
+       TEST_SOURCES += [ Wildcard $(i) : *.cpp *.h ] ;
+}
+
+LIBS += -lcppunit -lglestlib -lglestadvlib ;
+
+# Includes all game sources
+Application testgae : $(TEST_SOURCES) : console : : glestlib glestadvlib ;
+LinkWith testgae : glestlib glestadvlib ;
+ExternalLibs testgae : $(EXTERNAL_LIBS) CPPUNIT ;
+IncludeDir testgae : $(GAME_PCH_INCLUDE) $(TEST_DIRS) $(SHARED_INCLUDE_DIRS) $(GAME_DIRS) ;
 
 #### Editor ####
 if $(WX_AVAILABLE) = "yes" {
diff -r 27c339fb2cda mk/linux/mk/jam/application.jam
--- a/mk/linux/mk/jam/application.jam   Sun Apr 19 09:47:40 2009 +0400
+++ b/mk/linux/mk/jam/application.jam   Mon Apr 20 08:32:49 2009 +0400
@@ -25,19 +25,17 @@ rule Application
 rule Application
 {
     # check options
-    CheckOptions noinstall console independent : $(3) : $(<) ;
+    local options = $(3) ;
+    CheckOptions noinstall console independent : $(options) : $(<) ;
 
-    local target = [ ConstructApplicationTarget $(<) : $(3) ] ;
+    local target = [ ConstructApplicationTarget $(<) : $(options) ] ;
     local pch = $(4) ;
+    local libs = $(5) ;
     local sources = [ SearchSource $(pch) $(>) ] ;
-    local objects ;
     local install_targets ;
 
-       if $(pch) {
-        objects = [ CompilePch $(pch) ] ;
-       }
-
-    objects += [ CompileObjects $(sources) : $(3) ] ;
+    local objects = [ CompileObjects $(sources) : $(options) ] ;
+    objects = [ CreatePch $(<) : $(objects) : $(pch) : $(libs) ] ;
     $(<)_TYPE = application ;
     $(<)_OBJECTS = $(objects) ;
@@ -60,13 +58,13 @@ rule Application
     }
 
     # make dependency on apps target
-    if ! [ IsElem independent : $(3) ]
+    if ! [ IsElem independent : $(options) ]
     {
         Depends apps : $(<) ;
     }
 
     # construct Install target
-    if ! [ IsElem noinstall : $(3) ]
+    if ! [ IsElem noinstall : $(options) ]
     {
         $(<)_INSTALLTARGET = [
             DoInstall $(target) : $(bindir) : $(INSTALL_PROGRAM) : nopackage
@@ -74,17 +72,11 @@ rule Application
         Depends install_bin : $($(<)_INSTALLTARGET) ;
     }
 
-       local app_objects ;
-
-       for i in $(objects) {
-        if [ MATCH (\.o) : $(i) ] {
-           app_objects += $(i) ;
-         }
-       }
+    objects = [ FilterObjects $(objects) ] ;
 
     # Link
     MakeLocate $(target) : $(LOCATE_TARGETS) ;
-    SystemLinkApplication $(<) : $(app_objects) : $(3) ;
+    SystemLinkApplication $(<) : $(objects) : $(options) ;
 
     # Import default flags
     CppFlags $(<) : $(APPLICTION_CPPFLAGS) ;
@@ -93,7 +85,7 @@ rule Application
     LFlags $(<) : $(APPLICATION_LIBS) ;
 
     # Sources are part of the package
-    if ! [ IsElem nopackage : $(3) ]
+    if ! [ IsElem nopackage : $(options) ]
     {
         Package $(sources) ;
     }
diff -r 27c339fb2cda mk/linux/mk/jam/library.jam
--- a/mk/linux/mk/jam/library.jam       Sun Apr 19 09:47:40 2009 +0400
+++ b/mk/linux/mk/jam/library.jam       Mon Apr 20 08:32:49 2009 +0400
@@ -18,26 +18,25 @@ rule Library
 rule Library
 {
     # check options
-    CheckOptions noinstall independent shared : $(3) : $(<) ;
+    local options = $(3) ;
+    CheckOptions noinstall independent shared : $(options) : $(<) ;
 
     local no_scan_archive = $(NOARSCAN) ;
-    local target = [ ConstructLibraryTarget $(<) : $(3) ] ;
+    local target = [ ConstructLibraryTarget $(<) : $(options) ] ;
     local pch = $(4) ;
+    local libs = $(5) ;
     local sources = [ SearchSource $(pch) $(>) ] ;
     local objects ;
     local install_targets ;
 
-       if $(pch) {
-        objects = [ CompilePch $(pch) ] ;
-       }
-
-    objects += [ CompileObjects $(sources) : $(3) ] ;
+    local objects = [ CompileObjects $(sources) : $(options) ] ;
+    objects = [ CreatePch $(<) : $(objects) : $(pch) : $(libs) ] ;
 
     $(<)_TYPE = library ;
     $(<)_OBJECTS = $(objects) ;
     $(<)_SOURCES = $(sources) ;
     $(<)_TARGET = $(target) ;
-    $(<)_OPTIONS = $(3) ;
+    $(<)_OPTIONS = $(options) ;
 
     # create target clean rule
     Always $(<)clean ;
@@ -51,7 +50,7 @@ rule Library
         NotFile $(<) ;                                                              }
 
     # library depends on its member objects
-    if ! [ IsElem independent : $(3) ]
+    if ! [ IsElem independent : $(options) ]
     {
         if $(KEEPOBJS)
         {
@@ -64,13 +63,13 @@ rule Library
     }
 
     # Generate install rules
-    if ! [ IsElem noinstall : $(3) ]
+    if ! [ IsElem noinstall : $(options) ]
     {
         install_targets = [ DoInstall $(target) : $(libdir) : $(INSTALL) : nopackage ] ;
         Depends install_lib : $(install_targets) ;
     }
 
-    if [ IsElem shared : $(3) ]
+    if [ IsElem shared : $(options) ]
     {
         if ! $(LIBTOOL) {
             exit "LIBTOOL not defined, can't create dynamic library." ;
@@ -111,7 +110,9 @@ rule Library
 
     if $(CRELIB) { CreLib $(target) : $(objects[1]) ; }
 
-    SystemLinkLibrary $(<) : $(objects) : $(3) ;
+    SystemLinkLibrary $(<) : $(objects) : $(options) ;
+
+    objects = [ FilterObjects $(objects) ] ;
 
     # If we can't scan the library, we have to leave the .o's around.
     if ! ( $(no_scan_archive) || $(NOARUPDATE) )
@@ -126,7 +127,7 @@ rule Library
     LFlags $(<) : $(LIBRARY_LIBS) ;
 
     # Sources are part of the package
-    if ! [ IsElem nopackage : $(3) ]
+    if ! [ IsElem nopackage : $(options) ]
     {
       Package $(sources) ;
     }
diff -r 27c339fb2cda mk/linux/mk/jam/objects.jam
--- a/mk/linux/mk/jam/objects.jam       Sun Apr 19 09:47:40 2009 +0400
+++ b/mk/linux/mk/jam/objects.jam       Mon Apr 20 08:32:49 2009 +0400
@@ -55,6 +55,48 @@ rule CompileObjects
     }
 
     return $(targets) ;
+}
+
+##  FilterObjects objects
+##    Filters out all non-object files from the specified list.
+rule FilterObjects
+{
+    local objects = $(1) ;
+
+    local actual_objects ;   
+    for obj in $(objects) {
+        if [ MATCH (\.o) : $(obj) ] {
+            actual_objects += $(obj) ;
+        }
+    }
+
+    return $(actual_objects) ;
+}
+
+## CreatePch target : objects : pch : libs
+##   Creates precompiled header and necessary dependencies for the specified target.
+rule CreatePch
+{
+    local target = $(1) ;
+    local objects = $(2) ;
+    local pch = $(3) ;
+    local libs = $(4) ;
+
+    local pch_object ;
+    if $(pch) {
+        pch_object = [ CompilePch $(pch) ] ;
+    }
+
+    Depends $(objects) : $(pch_object) ;
+    objects += $(pch_object) ;
+
+    for lib in $(libs) {
+        Depends $(objects) : $($(lib)_PCH) ;
+    }
+
+    $(target)_PCH = $(pch_object) ;
+
+    return $(objects) ;
 }
 
 # Begin pre-compiled header support, added by Daniel Santos
diff -r 27c339fb2cda source/game/game/game_constants.cpp
--- a/source/game/game/game_constants.cpp       Sun Apr 19 09:47:40 2009 +0400
+++ b/source/game/game/game_constants.cpp       Mon Apr 20 08:32:49 2009 +0400
@@ -10,8 +10,6 @@
 // ==============================================================
 
 #include "pch.h"
-
-#define GAME_CONSTANTS_DEF
 #include "game_constants.h"
 
 namespace Game {
diff -r 27c339fb2cda source/game/game/game_constants.h
--- a/source/game/game/game_constants.h Sun Apr 19 09:47:40 2009 +0400
+++ b/source/game/game/game_constants.h Mon Apr 20 08:32:49 2009 +0400
@@ -17,11 +17,12 @@
 
 using Shared::Util::EnumNames;
 
-#ifdef GAME_CONSTANTS_DEF
-#      define STRINGY_ENUM_NAMES(name, count, ...) EnumNames name(#__VA_ARGS__, count, true, false)
-#else
-#      define STRINGY_ENUM_NAMES(name, count, ...) extern Shared::Util::EnumNames name
-#endif
+#define STRINGY_ENUM_NAMES(name, count, ...) \
+       inline Shared::Util::EnumNames &name ## _() { \
+               static Shared::Util::EnumNames instance(#__VA_ARGS__, count, true, false); \
+               return instance; \
+       } \
+       static Shared::Util::EnumNames &name = name ## _();
 
 #define STRINGY_ENUM(name, countValue, ...)                                                            \
        enum name {__VA_ARGS__, countValue};                                                            \
diff -r 27c339fb2cda source/shared_lib/include/pch.h
--- a/source/shared_lib/include/pch.h   Sun Apr 19 09:47:40 2009 +0400
+++ b/source/shared_lib/include/pch.h   Mon Apr 20 08:32:49 2009 +0400
@@ -66,7 +66,6 @@
 #include <sstream>
 #include <stdexcept>
 #include <string>
-#include <string.h>
 #include <vector>
 
 // will this fly on windoze?


[daniel.santos]

very cool!  I'll have to try this out tomorrow.  My only comment on the STRINGY_ENUM_NAMES is that it looks like it will increase the total executable size (mostly in the data segments).  I'm a little less worried about that at this point however and there are ample mechanisms to solve this problem that don't have the same drawbacks as my previous approach.  For now, I'm going to go with the "first make it right, then optimize" approach.

Finally, if I get this to work, I'm going to remove my "CompilePch" and "CompilePchSingle" rules, as I barely understood what I was doing!