Caveats, Extensions and Conclusion
That's all there is to 3D decals - they look miles better than current 2D decals, and can create a much greater sense of realism. However, there are a few caveats to the above method:
- Vertices in the decal model must not extend further outwards than the leading edge of the model (the one that will be 'on' the wall) - if the models feature this, then you'll get holes in the wall beside it due to the method of generating the hole in the wall via flattening the vertices. This can be countered by finding the edge of the decal and flattening that instead. (A way to do this would be to use all edges that only have one triangle on them, as opposed to 2.)
- Decals can't extend across more than one surface. This would be extremely complex to do across, say, 3 surfaces (in a corner of a room or something similar) but across 2 surfaces shouldn't be too difficult - it would require a weighted rotation of the vertices about the edge between the first surface and the second (where a 'weighted rotation' is where vertices that are further from the edge are rotated more than those located nearer the edge).
- Large decals will stretch through to other surfaces if the other surface is near enough the decaled wall. It's possible to avoid this by capping the decal at the other wall, preventing it from being rendered once past it - this can be done automatically via OpenGL's clip planes, or it can be computed manually.
There are also some extensions to the algorithm outlined here that can be implemented:
- Texture of a wall extends across the decal. If you shoot brick across the line of mortar, you'd expect to still see the mortar with brick on either side of it in the chipped-out hole left by the bullet, rather than a flat-colored hole. The vertices of the decal can be projected onto the plane of the wall (or other surface) and then the texture coordinates can be calculated relative to the vertices of the wall.
- Lighting the decal. This may seem fairly obvious, but the decal needs to be lit - if there's a bright decal standing out in the middle of a fairly dark wall, it'll look totally out of place. Simple per-vertex lighting can be used to make it look more in-place.
- Scaling the decal. If you have one or two decals of the same size, you're going to want an RPG impact to create a larger decal than a revolver bullet (although significant extensions to the algorithm would be required to do very large scalings without artifacts cropping up). The decal can be scaled up or down to reflect the size of the impact, thus giving some variation in your decals.
- What to do if decals overlap. If you have two decals overlapping and you just render them both as normal, then you'll get butt-ugly artifacts - you'll be able to clearly see the models overlapping, which doesn't look good at all. What is needed is to clip the decals before rendering them if you detect an overlap (just project the bounding boxes of the decals onto the surface and check for a 2D box collision - see figure 2) and render some 'bridging' polygons between the two clipped edges (as is done in some terrain rendering LOD implementations) to close any gap that may have appeared (see figure 3).
Figure 2 - The bounding boxes of the overlapping decals projected onto the plane with the ideal clipping line marked in red. The dotted lines represent the parts of the decals to clip away.
Figure 3 - A 2D analogy (side-on) of rendering 'bridging' polygons. The red line represents a 'bridging' polygon, with the clipped decals either side.
For more information on the stencil buffer, I recommend the OpenGL Programming Guide (often called the 'red book') and NeHe. Any comments on the algorithm (problems or extensions you've done) can be emailed to me at firstname.lastname@example.org - I'd love to hear your comments. (Thanks should go to Ryan "Professional Victim" Desgroseilliers and Tim "Chips" Green for valuable input into this article.)