KJam - a new build tool for game development
by Rafael Baptista

Introduction

KJam is a build tool - like make. It is designed to support very large software projects, especially project which require building more than just code but data as well. It is also designed to be extremely fast.

I am a game developer. After my last game project I went looking for a build tool which was better than the tools I was using, Gnu Make and MSBuild (the build tool that comes with MSVC). The best tool that I could find was something called Jam, developed by the same people who develop Perforce. The great thing about Jam is that it is simple, powerful and extremely fast .

After working with Jam for some time, I started seeing things that I wished Jam could do. Specifically:

Batched Building: MSVC can compiled multiple source files into multiple object files in a single call to the compiler, much faster than it if you compile the files one at a time. But few build tools support that, and those that do are very slow.
Network Builds: Many game development projects require that data be compiled in addition to code. For example pre-computation of visibilty data, environmant maps, shadow maps and the like. I needed a build tool which could distribute these expensive build tasks over a distributed network of build machines. ( Like Incredibuild does with MSVC source files - but for data ).

Luckily the sources for Jam are available, and I started adding these features. But the more work I did to Jam the more I realized that things could be done so much better. I didn't intend to, but I ended up working on a completely new version of Jam for more than a year. I call it KJam.

KJam has several major benefits over Jam and other build tools:

Fast : KJam is the fastest build tool I have tested . KJam can scan a large codebase and build a dependency graph more than 20 times faster than MSBuild and Gnu Make. Through the use of parallel execution, and batched building it will build source files under MSVC significantly faster than other tools. In some cases KJam can reduce full build times by more than 3x. These accelerated build times are possible on multi-processor workstations
Easy: KJam's build language is easy to use, and designed to support projects which target multiple build environments. Build files are split into rules and actions . Rules describe how files are related to each other, what dependencies exist and where the files can be found. The rules can be completely platform independent. The actions describe exactly what build steps need to performed on a given target platform to build. Targeting a new platform normally means writing a new set of actions. The rules remain unchanged. This is a great idea take directly from Jam.
Robust: KJam is designed to support large projects. For extremely large projects, determining what targets need to be rebuilt can take minutes. Every compile becomes a chore. KJam can reduce this scanning time by a factor of 20 or more.
Distributed: KJam has a distributed build mode. KJam servers can be set up to run in the background on workstations all over the office, spreading the build load to multiple workstations, resulting in dramatic reductions in build times even above what is already possible in non-network mode.
Portable: By default KJam runs all build commands in its own internal sh-like shell. This means that developers have a single shell environment no matter what platform they are on. This is mostly to the benefit of Windows programmer who would otherwise be stuck using Window's extremely limited cmd.exe shell. With the built-in shell many build actions are completely portable across platforms.

KJam is currently still in development, but it is stable and useful enough that it is already used in production by several major game development studios for their next generation projects. A version of KJam is available for download free for use by non-commercial projects. Commercial developers are encouraged to contact us and join our beta program. Eventually, when KJam is perfect we plan to sell a commercial version. For right now we just want to make it easy for people to start using KJam and to give us feedback on how well it worked for them. Comments from our users currently drive most of the development of KJam.

Getting Started with KJam

So how do you get started using KJam? First download the binaries. They are currently available for Windows and Linux.

KJam comes configured with a standard set of build rules for c and c++ projects. For many simple projects it is possible to get started using KJam in just a few minutes by writing some very simple, yet surprisingly powerful Jamfiles. For larger projects, or to support custom tools you will need to write your own custom build rules. For now, the fastest way to try KJam is to use the default build rules.

Suppose you have a directory called src full of .cpp source files that you want to build into an application called myprog. You would create a file called jamfile with the following contents:

StaticExecutable myprog : src ;

To build myprog, run KJam in the same directory as the jamfile ( by typing kjam ). KJam will create an output directory, bin for all the generated files. It will search the directory src for source files. It will process any .l and .y files that it finds though flex and bison. It will scan all source files for #include statements and generate dependencies. It will compile all the .c and .cpp files it found or created. And finally it will link all the resulting object files into a binary.

Even though KJam does a full dependency analysis of your sources before every run, it will start building even very large projects almost immediately. KJam is extremely efficient.

This same jamfile should work without modification under Windows or Linux. The generated files will be stored in a directory under bin which specifies the platform and the debugging level.

If your machine has multiple processors, KJam will figure out which build steps can be safely done simultaneously and take advantage of them. If your compiler supports batched building (like MSVC does ) KJam will build sources in batches.

KJam will detect the width of your shell, and format all the output neatly for easy readbility. And even though many build steps will happen simultaneously, KJam will a not allow output from different steps to be interspersed.

Invoking Rules

Suppose you don't want to build every source file in a directory, but would rather name your source files specifically. You can write your jamfile like this:

StaticExecutable myprog : src1.cpp src2.cpp src3.cpp ;

To build a library you use the StaticLibrary rule:

StaticLibrary mylib : src1.cpp src2.cpp src3.cpp ;

To build a .dll or a shared library use the SharedLibrary rule:

SharedLibrary mylib : src1.cpp src2.cpp src3.cpp ;

Under Windows this will create a .dll and its associated export library. Under Linux this will create a .so file. All the details of how to do that on each platform are dealt with for you - portably. The target names do not have to deal with non-portable file extensions like .dll and .so.

To build a program that links with a library, just list the libraries at the end of the rule:

StaticLibrary lib1 : lib1/src ;
StaticLibrary lib2 : lib2/src ;
StaticExecutable myprog : myprog/src : lib1.lib lib2.lib ;

The .lib extension helps KJam to know that you want to link with a static library instead of a dynamic library. But the extensions are portable between Windows and Linux. Under Linux, KJam would link with lib1.a and lib2.a. And if you had passed in lib1.a then under Windows KJam would have automatically converted it to lib1.lib.

You can also easily link with a shared library:

SharedLibrary lib1 : lib1/src ; 
StaticLibrary lib2 : lib2/src ; 
StaticExecutable myprog : myprog/src : lib1.dll lib2.lib ;

Under Windows, KJam will figure out that it needs to look for an export library. Under Linux, it will look for a .so shared library. The user does not have to worry about these non-portable details. KJam will even figure out all the dependencies between these different targets automatically.

By default when you run KJam without any arguments it will try to build the target all, which will build all the targets that have been declared using the above rules. So in the case of the previous jamfile, it would build lib1, lib2 and myprog.

To build only a subset of the declared targets you can declare a Group target:

SharedLibrary lib1 : lib1/src ;
StaticLibrary lib2 : lib2/src ;
StaticExecutable myprog : myprog/src : lib1.dll lib2.lib ;
Group libs : lib1.lib lib2.lib ;

Now when you ask it to build libs, it will only build lib1 and lib2. To exclude a target from the defauilt all target, declare your own all group listing the targets you want. This will override the automatically created all target:

SharedLibrary lib1 : lib1/src ;
StaticLibrary lib2 : lib2/src ;
StaticLibrary special : lib2/src ;
StaticExecutable myprog : myprog/src : lib1.dll lib2.lib ;
Group all : lib1.lib lib2.lib myprog.exe ;

Now when you run KJam with no arguments, it will not build the library special. To get that you would have to ask for it specifically by running kjam special.

These simple jamfiles automatically create clean targets. So if you want to clear out all the generated files just run kjam clean.

Setting Variables

The operation of the built-in rules can be modified by setting variables.

By default KJam will search for header files wherever it finds source files. If you would like it to use headers found in other directories, say the headers for an external library, set the INCLUDE_DIRS variable:

INCLUDE_DIRS = ../otherlib/src ;
StaticExecutable myprog : src ;

By default KJam will look for libraries to link with in the default output directory, and in the default system library directories. To add more places to look for libraries to link with, use the LIB_DIRS variable:

INCLUDE_DIRS = ../otherlib/src ; 
LIB_DIRS = ../otherlib ; 
StaticExecutable myprog : src : otherlib.lib ;

KJam will automatically chose a reasonable set of build flags when running your c compiler. To add additional build flags use the CCOPTS variable.

To add extra link options use the LINKOPTS variable. The new options will be added to every compile or link:

CCOPTS = /DSPECIAL ; 
LINKOPTS = -lspecial ;
StaticExecutable myprog : src ;

To change the output directory set BIN_DIR. By default BIN_DIR is set to bin/$(PLATFORM)/$(CURFLAV). You can set this to anything you want, though it is highly recommended that BIN_DIR always include $(CURFLAV) or $(BUILD_CODE), to avoid problems with mixing generated files built with different debugging levels. So for example if you want to have all your generated files go to a special directory shared by all projects, instead of a local one, and you don't care about multiple platforms, you might set BIN_DIR this way:

BIN_DIR = ../shared_bin/$(CURFLAV) ;

These variables can also be set on a per target basis using the on keyword. For example to add special link options to just lib1:

LINKOPTS on lib1.a = -lspecial ; 
StaticLibrary lib1 : lib1/src ; 
StaticLibrary lib2 : lib2/src ; 
StaticLibrary lib3 : lib3/src ;

To build targets with special compilation options, you can use the StaticObject rule. For example in the following case only special.cpp is built with the extra flags:

StaticObject special : special.cpp : /DSPECIAL ;
StaticExecutable myprog : special.obj src1.cpp src2.cpp src3.cpp ;

Notice that when you need to, you can pass compiled object files as sources.

The built-in Jambase supports building targets at different optimization levels. You can do this by setting the CURFLAV variable. Set this variable in the environment. On Linux remember to export it. There are 4 levels, release, optimized, debug and extra_debug. At each setting the output files will go to their own directory, to avoid mixing output files built with different compiler flags.

KJam can also build projects with platform specific source files. Suppose you have a project with one or more source files which are different on Windows and Linux. Simply create linux and win32 subdirectories in the directory where source files would normally be found, and put the platform specific source files in each directory. KJam will build the appropriate sources for the platform.

Managing Sub-Projects

Jamfiles can call other Jamfiles that manage individual subprojects. Suppose you have 5 directories, where each directory has its own jamfile and can build a separate library or executable. There may be dependencies between some of these sub-projects. This is easy to do:

SubProject lib1 ; 
SubProject lib2 ; 
SubProject lib3 ; 
SubProject app1 : lib1 lib2 ;
SubProject app2 : lib1 lib3 ;

With the jamfile above you can now build all the libraries and applications by running KJam in the parent directory. You can build any individual sub-project and the projects it depends on by running KJam with the name of the sub-project as the target. KJam will build the sub-projects in the right order. You can also switch to any individual sub-project directory to build and test just that sub-project.

A clean target is also automatically created. Building clean will clean all the sub-projects.

When building a sub-project KJam will invoke a copy of itself in the subproject's directory and will build the default all target. To have KJam build other targets as well, list those as the third argument to the SubProject rule:

SubProject app1 : lib1 lib2 : all verify install ;

It is also possible to make multiple SubProject targets each of which builds a particular sub-project target:

SubProject app1 : lib1 lib2 ; 		// build the 'all' target 
SubProject app1 : lib1 lib2 : verify ; 	// build the 'verify' target 
SubProject app1 : lib1 lib2 : install ; // build the 'install' target

Now you can invoke each of those targets in the given subproject from the main build file. Here is a jamfile which builds all your sub-projects, and has an install target:

SubProject lib1 ; 
SubProject lib2 ; 
SubProject lib3 ; 
SubProject app1 : lib1 lib2 ; 
SubProject app2 : lib1 lib3 ; 


SubProject app1 : app1 : install ; 
SubProject app2 : app1 : install ; 

Group all : lib1 lib2 lib3 app1 app2 ; 
Group install : app1 app2 ;

Note that you override the automatically created all target to exclude the install sub-project targets. Also notice that the install targets depend on their respective applications being up-to-date.

Given the jamfile above, here are some commands you can run and what they would do:

kjam 			# re-build all the libraries and applications 
kjam lib2 		# re-build just lib2 
kjam app2 		# re-build app2 and its dependents lib1 and lib3 
kjam clean 		# clean all the sub-projects 
kjam install 		# re-build all the applications, and install them all 
kjam "app1" 		# re-build app1, lib1 and lib2, and then install just app1

Try KJam and let me know what you think. It will really help us to refine it.

Discuss this article in the forums

Date this article was posted to GameDev.net: 10/5/2006
(Note that this date does not necessarily correspond to the date the article was written)

See Also:
Programming