Automatically generating code for a new SPU

This page will describe the creation of a more complicated SPU -- the "seethrough" SPU. This SPU will force everything to draw transparently. The alpha setting for every color will be configurable, along with the blend mode. Unlike the invert SPU, this SPU will not omit any of the relevant functions. Instead, it will use a Python script to automatically generate the code.

Step 1: Creating the SPU directory

This step is the same as the first step in the invert SPU walkthrough. Simply create a copy of the template SPU and run the provided script:

1. cp -r template seethrough
2. cd seethrough
3. python gen_template.py seethrough

Step 2: List the functions you will be implementing

The provided helper libraries for auto-generating OpenGL code have efficient routines for accessing lists of functions in external files that end in "_special" (note the underscore). For this example, we'll create a file called seethrough_special that will list every function we plan to implement.

Create a new file cr/spu/seethrough/seethrough_special.  We're going to need to override all of the glColor functions, as well as the glMaterialfv and glMaterialiv functions. In addition, we will need our own implementation of glDisable (to prevent the user from turning off blending), glEnable (to prevent the user from enabling the depth test), and glBlendFunc (to prevent the user from messing with our chosen blend function).  Therefore, add the following lines to the new "special" file (they don't need to be in alphabetical order):

Step 3: Auto-generate seethroughspu.c

This step is fairly involved.  If you're having trouble following along, you can always check out the completed SeeThrough SPU (although see step 13 for one required edit to a file outside the cr/spu/seethrough directory).

For this example, we're going to automatically generate the named dispatch table contained in seethroughspu.c. Create a file called seethrough.py that will be used to generate the code.

At the top of the file, put the lines:

This code loads in the parsed representation of the OpenGL headers, which gives a structured interface to the entire API supported by Chromium. gl_mapping is an associative array whose keys are OpenGL function names (without the "gl" prefix). stub_common is a set of utility functions that we will use to generate code later.

Although we won't need to in this example, we can easily extract the list of all functions into a sorted list with the commands:

Many of the code generating scripts throughout Chromium will iterate over this list of keys.

Now that the parsed representation of the file has been loaded, we should print out the header of seethroughspu.c. in our case, this will be identical to the existing seethroughspu.c (created in step 1), so just pull the first 6 lines of that file into our script, wrapped in a Python print statement:

In Python, strings enclosed in triple quotes (""") can have embedded newlines in them.  Notice the include of "cr_glwrapper.h" -- this is a wrapper for the system's OpenGL header, required on Windows (since <windows.h> has to be included before <GL/gl.h> will work on Windows).  Also, we've added "cr_string.h", which will be used when building the named function table.

Because there are so many glColor calls, we'll auto-generate the code for those, and hand-code the few remaining functions.  We're going to want a typical color function to look like:

Notice the variable "seethrough_spu.opacityf" -- we'll assume that the configuration routine has pre-computed the user-specified opacity in all the various types that we will need -- float, byte, short, double, and their unsigned variants.  To generate the 32 glColor calls, we will loop over the legal types and legal component numbers (in this case, 3 and 4).

Add the following code to the python script:

This code will generate empty function bodies for all the glColor functions that we want to implement, alternating between the parameter-passing (e.g., glColor3f) and vector (e.g., glColor3fv) varieties.  Notice that the gl_mapping array has given us the return type of our function, as well as a list of argument names and argument types.  The stub_common library uses this information to build the argument list for the function definition.

The easiest way to implement these functions is to have the stub_common library build the SuperSPU call string for you, just like it built the argument declarations.  To do this, all we need to have is a Python array that contains the names of the arguments we want to pass.  For the parameter-passing glColor functions, we either append the name of the opacity variable to the arg_names array (if there are three components), or we modify the fourth parameter name (if there are four components).

So, for the parameter-passing code (i.e., between the first printed pair of curly braces), add the following code, taking extra care to make sure that the indentation matches with the surrounding code (i.e., this code should be indented two block levels):

Now we have half of our glColor functions working.  The implementation for the vector passing functions is even easier.  Just add this code:

Notice that in both cases, we have dispatched to the 4-parameter version of the glColor functions.  Also note that if we are implementing a glColor that itself has four parameters, we need to insert a bogus reference to the fourth (alpha) parameter to avoid "unused variable" warnings from the compiler. 

Now that all of our color  functions are implemented, let's declare the rest of the functions as "extern" and generate the named function table.  To do this, we're going to use a method of stub_common called "AllSpecials", which returns an the array of function names in a given "_special" file.  There is also a "FindSpecial" predicate function that tells whether a given function is contained in a "_special" file or not.

Append the following code to the end of seethrough.py (note that this code is not indented at all -- it's at the outermost scope):

Notice that we bother declaring the actual prototype for these functions even though they're going to be cast to SPUGenericFunction in the named function table anyway, because otherwise the Windows linker won't find them.

Finally, let's make the named function table.  Unfortunately, since we're not implementing all of our functions in seethroughspu.c, we can't just create a statically initialized table (since the four extern'ed function pointers aren't compile-time constants).  So we have to build the function programmatically, which is just slightly harder.  First, let's declare the table.  Since we're going to fill it ourselves, we know how big it is.  Don't forget to leave space at the end for the NULL terminator!

I like to use a helper function called "__fillin" for this table-building task.  We'll need to print it out at the bottom of this file:

Now, we're finally ready to build our named function table.

So that we can call this function from SPUInit, add a declaration for it to seethroughspu.h:

Step 4: Take a break

That was brutal.  Go have a good glass of wine.  When you get back, let's make sure that the initialization step for the SPU actually builds the function table before we return it.  Add a call to the function defined in step 3 at the bottom of the SPUInit function:

Step 5: Declare and initialize the opacity variables

Let's stop implementing functions for a moment and go deal with all of these opacity variables that are floating around.

Add the following declarations to the SeethroughSPU structure in seethroughspu.h:

Once you do this, the project should compile without warnings, although it won't link because we still haven't implemented all the functions.

In order to initialize these variables, we need to get some configuration information, so we need to talk to the mothership!

Open the file seethroughspu_config.c.  You should see a comment near the bottom that says "CONFIGURATION STUFF HERE".  This is where we will be asking the mothership questions.  If you read the code in this file, you'll see that the template SPU doesn't consider inability to contact the mothership fatal.  Some SPU's (such as the tilesort SPU) need to talk to the mothership, and can't run if they don't.  In our case, the template SPU's behavior is correct.

We'll make the design decision that opacity will be specified to the mothership as a floating point value between 0.0 and 1.0.  Let's add a function to convert such a floating point value to all the desired types.  Add the following function to the top of seethroughspu_config.c:

The GL_MAX* constants are defined in cr/include/state/cr_statetypes.h, so let's add an include line to the top of the file:

Before we go asking the mothership anything, let's set up some reasonable defaults.  In the "__setDefaults" function at the top of this file, add the following initialization code:

Now, we're ready to query the mothership.  There's almost nothing to it!  Replace the "CONFIGURATION STUFF HERE" comment (and, if you want, the cast of "response" immediately below it, since we're going to use response now) with this:

crMothershipSPUParam will return a positive value if the SPU has such a named parameter, and the value will be placed in the "response" buffer.

We're done with this for now, although we will return to configuration in a few steps to get the user-specified blend mode.

Step 6: Implement the remaining functions

Okay, back to seeing through things.  There are four functions left to do: glMaterialfv, glMaterialiv, glBlendFunc, and glDisable.  We'll present the material functions first.  They're totally straightforward, so they're presented without comment.  Add these functions to a new file called seethrough_misc.c:

Let's do glDisable next.  The idea here is to prevent the user from ever turning off blending.  We just look at the enum, and if it's GL_BLEND, we drop the command on the floor.  To be polite about it, we'll print a warning to the screen when this happens.

glEnable (for preventing the depth test from getting turned on) is almost identical:

Finally, the simplest one of them all: glBlendFunc.  If the user tries to change the blend function, he's out of luck -- our SPU is in charge of blending.  We just ignore all glBlendFunc calls, making sure not to get compiler warnings from unused variables.  Note that if you're following along in the completed implementation, this function actually doesn't exist, as explained in step 12.

Step 7: Configure and initialize the blend function

We're almost there.  We want to allow the user to set the blend function to be used.  Declare the following two variables in the SeethroughSPU structure in seethroughspu.h:

Now, let's return to seethroughspu_config.c and set a default value.  In __setDefaults, add the code:

Now that everything has reasonable defaults, let's get values from the mothership.  After the configuration step for the opacity, add this code:

Since the mothership is going to return us a string, we need to turn it back into a GLenum type, so implement the setBlendFuncFactor function at the top of this file:

There is a small robustness problem here -- the legal values for source and destination blending factors are slightly different, but no distinction is made between them here.

Step 8:  Set up and turn on blending at startup

When the SPU is loaded, we need to enable blending and set the blend function.  This is analogous to setting the default color in the Invert SPU.  At the very end of the SPUInit function in seethroughspu_init.c (after your previously-added call to build the function table), add the code:

Note that we do not have to disable the depth test at this point, as it is off by default, and our implementaton of glEnable will prevent it from ever getting turned on.

Step 9: Claim our inheritance

Don't forget, this SPU is a SubSPU of the PassThrough SPU.  Change the second line in SPULoad in seethroughspu_init.c to:

Step 10: Write a configuration file

It's hard to believe, but we're done coding!  Now all we need to do is update the crdemo.conf file described in the "Configuration scripts" section.  Add the creation of a new SPU to the SPU creation section:

In the SPU configuration section, configure this SPU any way you like:

And finally, add it before the client SPU:

Step 11: Enjoy

Step 12: Think about what you've done

Will this SPU work reasonably all the time?  A little thought reveals that it will not.  Although the "bluepony" demo works OK, Quake III doesn't look right at all, for two reasons:

1. Quake III uses vertex arrays to specify its colors, and our SPU doesn't handle vertex arrays.  So we've turned on blending, but we haven't tweaked the alpha of the colors
2. Quake III has its own transparency in many places, and a single blend function doesn't work.

The solution to problem #2 is easy.  We will allow the user to change the blend function to whatever they want.  If they ever try to disable blending, we will instead reset the blend function to our configured defaults.  This should have the effect of only modifying geometry that was intended to be opaque.

To do this, we just add one line immediately after our warning in seethroughDisable in seethroughspu_misc.c:

With this modification, we don't need to implement seethroughBlendFunc at all!  So simply remove it from the seethrough_special file and delete its implementation.

If you run Quake III now, the user interfaces will work again, and the game is playable.  Some things are transparent, and some things aren't, mainly because Quake III doesn't change any alpha values if it thinks that blending is turned off.  So if a wall is drawn after a transparent water surface, the wall will have the same transparency as the water.  Also, Quake III does extremely agressive visibility culling, so you can see elements appearing and disappearing all the time.

One other thing that seems wrong is that backface culling could be turned on.  We certainly don't want that, so we'll add it to the list of things that's prohibited in seethroughEnable in seethroughspu_misc.c:

And we disable it in SPUInit in seethroughspu_init.c:

In order to actually play what I call "GlassQuake", we still need to implement vertex arrays in our SPU.

Step 13: Bring in the state tracker

To get vertex arrays to work, we will need the assistance of the state tracker.  The state tracker will keep track of the location of all of the vertex array pointers, which ones are enabled, and what format they're in.

First, let's make sure that we are linking against the state tracker.  Because the state tracker has global state, we link against it in a special way to avoid sharing global variables with other SPU's that also track state.  To link against the state tracker, do the following.

Edit the file cr/spu/state_tracker/Makefile and add the string "seethroughspu" to the variable "LIB_COPIES".  The new setting should look like:

Once you've done this, type 'make' in the state_tracker directory to get a personalized copy of the state tracker built for the SeeThrough SPU. 

NOTE: This step won't have been done for you if you're just using the completed implementation, so you have to do it!

Now go back to cr/spu/seethrough/Makefile, and add the line:

Anywhere before the "include" of cr.mk..  I typically put the TRACKS_STATE setting immediately after the LIBRARIES variable.

While we're in the Makefile, let's create a new file for our array implementations.  Add a file called "seethroughspu_arrays" to the FILES variable.  The resulting list should look like:

Before we start implementing functions, let's figure out what it is we want to do.  We will need to use the state tracker's implementation of glEnableClientState, glDisableClientState, glVertexPointer, glColorPointer, glIndexPointer, glNormalPointer, glTexCoordPointer, glEdgeFlagPointer, and glInterleavedArrays.  For each of these functions, we will want to call the state tracker's implementation, and then also dispatch to our SuperSPU.  Because this is a very simple task and highly repetitive, we'll add it to the auto-generating code in seethrough.py.

First, let's add the 9 functions listed above to a new file called seethrough_state_special.  We'll update the auto-generating code so that anything found in this file will automatically dispatch to the state tracker as well as our SuperSPU.  The file should look like:

Now, let's go back to the seethrough.py script.  Right after the code for generating the seethroughColor functions, add the following code:

This will generate functions that look like:

Which is exactly what we want.  Because we're going to be using state tracking functions or data in multiple files, let's include the state tracker's header file at the top of seethroughspu.h:

To get the function declarations for all the state_tracking functions.  Now, we need to add these functions to the named function table at the end of seethroughspu.c.  First of all, this will make our named function table bigger.  Let's update the line in seethroughspu.py where the named function table declaration is printed.  The new line should read:

This will allow enough space for all the functions from both _special files.  Now, just add a second loop to add functions to the table.  This should come immediately after the final loop in the script, before the NULL terminator is printed:

Step 14: Initialize the state tracker

The state tracker needs a "context" into which to track all of the OpenGL state (or in our case, the subset we care about).  Let's add a "CRContext" structure to the SeethroughSPU structure in seethroughspu.h:

Now that the variable exists, we can initialize it in SPUInit in seethroughspu_init.c:

That's it!  You're tracking state!

Step 15:  Handle vertex array drawing calls

This is by far the most complex part of this SPU.  What we're going to do is take calls to glArrayElement, glDrawArrays, and glDrawElements, and pull them apart into individual calls to the non-vertex array equivalents, based on which arrays are enabled.  Although this is a complicated thing to get right, the logic to do it already exists in the Chromium packer!  We're going to copy functions out of cr/packer/pack_client.c and rework them for our own needs.  

Go ahead and add those three functions to the file seethrough_special. (RESULTS NOT SHOWN)

Before we tackle seethroughArrayElement (by far the most complex function), let's write the other two functions in terms of it.  Create the file seethroughspu_arrays.c (remember, this was added to the Makefile back in step 13), and write the following two functions.  They are shown here without much comment, except to point out that they do a bit of error checking.  Chromium tends to consider OpenGL errors to be fatal, rather than setting a flag to be checked later.  This is a design decision, and one of the ways in which using Chromium can be slightly different from using vanilla OpenGL.

The implementation of glDrawElements is similar:

Okay, now we're down to the final function: glArrayElement.  Basically what this function is going to do is use the CRClientState structure to figure out which arrays are enabled, and for each enabled array, generate the appropriate function calls corresponding to that array ( glColor, glTexCoord, etc).  This function is big, but it's not complicated.  Notice that we call seethroughColor instead of seethrough_spu.super.Color, because that's where the transparency transformation happens.  Since those functions aren't in any header file, we declare them extern here.

Step 16: Enjoy even more

GlassQuake, with BlendFunc( GL_SRC_ALPHA, GL_ONE )

Here are a few more screenshots of GlassQuake in various blending modes.