Tech feature: Scripting upgrade

Some links in this article have expired and have been removed.

For a couple of months now I have, on and off, worked on some basic tech aspects for the engine. Everytime I was done with one of these I thought it was among the hardest things I would do for the new engine, yet the next feature as always proved more challenging. Terrain geometry was harder to implement than sun shadowsterrain texturing harder than geometry, and so on.

Implementing the script system is no different. It is easily the hardest thing I have done so far for our new engine – HPL 3. It has had this “perfect”sort of challenge: Difficult problems to solve, much basic knowledge to wrap your head around and awfully boring and monotonous parts. I really hope this marks the end of this trend of increasing difficulty, as another proportionally large step might make my brain to melt and my fingers to crack. At least it can wait for a lil while…

Enough complaining. Now that the scripting is pretty much implemented (some engine stuff still needs to be added and some more problems remain to be solved, but it is really minor), I am extremely happy with it. I think it will help us build better games, and to finish them faster. Now let’s move on to how a boring scripting system accomplishes this.

Background

Before moving on the the actual scripting I need to explain what brought on the creation of the current system. It all started with our first commercial games, The Penumbra series.

When creating the Penumbra games our tools were primitive to say the least. All maps were made in a 3D modeling program, Maya, and then exported to Collada. The game engine loaded the Collada file and built the map from it. As a 3D modeling program is meant to create 3D models, it is not really meant to make levels in. With no ways of placing entities we had to use special naming conventions to tell the game where any non-static objects in the game were located. To be able to do this properly we had to make special “instance” versions of each model meant for the game, since without this you would not be able to see how an object was placed before the game started.

Lighting was equally annoying since Maya has no support for radius on lights. This mean that you could not visually see how far a light reached, but simply entered a numerical value and hoped for the best. As this was not enough, you also needed to place portals and group meshes in order for the engine to provide occlusion culling. This could be quite tricky at times, and often you could sit a day or two simply tweaking portal setup. Added to this was also the problem that Maya often failed to show any textures, and most editing was done on grayish levels. For more info, you can just check the wiki.

The problems do not stop here though. Everytime you meant a change to the game, you had to do a complete reload. So setting up lighting in the game could be quite the effort: change light position for two seconds, load map for 2 minutes, notice it is not good, repeat. As you might figure, we got quite good at batches tasks and the phrase “it will have to do” was uttered more often than not.

For scripting it was just a grueling. Every change in the script required a full restart of the game, creating the same sort of frustration present when modeling maps. And to make scripting even more frustrating, there was no syntax checking until the entire map was loaded! This meant you could wait two minutes of loading only to find out you had forgotten a semicolon or something else trivial.

As I write this, I actually have a hard time understanding how we could have gotten anything done at all. And unsurprisingly, even though we released mod tools and documentation, not a single user map for Penumbra was ever released.

For Amnesia we knew we wanted to fix this somehow. The first step we took was to simply make our own editor where all the maps are built. Since it rendered with the same engine as the game, it made is much easier and faster to tweak entity and light placement. We instantly saw that productivity rise with this change. For scripting it was pretty much the same, but we added the extremely simple fix of compiling the script before loading the game. This removed some of the time previously spent on, in vain, looking at loading screen.

Although we had new tools all was not good. You still had to reload all level data every time you made a change to the map or script. We did not think much of this though as we were so used to doing it this way, and happy that we had all the other improvements. However, a year and a half into the development we discussed if we really needed to reload the level. I cannot recall what sparked this idea, but anyhow we figured that we did not and I added a menu with a Quick Reload button. This cached all textures, models, etc and reloaded map very quickly (usually taking but a few seconds). This increased productivity and creativity tremendously and was one of the better decisions we made during the development of Amnesia. Another sign of how much these changes improved work flow are the over a hundred of user maps created as of today.

What is so strange about the reload-feature is that is something that we could have added during the development of the first Penumbra, but for some reason we did not. It is quite frightening how often you convinces yourself that there is no better way of doing a task, and never try to improve it. We did not want to make this mistake again and started thinking of what more we could do.

Taking script to the next level

In Amnesia and Penumbra, scripting is only used to control logic flow in the levels. How enemies spawn, how puzzles work and so on. All other gameplay is hard-coded into the exe file and written in C++. Normally when I write this kind of code, rendering for instance, I can do large chunks at a time and then simply see if it works as intended. This often in small projects that are fast to reload. However, when writing gameplay and UI code this is almost never the case. Instead you constantly need to fine tune algorithms and variables until you get the expected behavior and work in large projects. Not only does this mean a level restart (with full resource reload), but the exe itself also needs to be built from code, a process that can easily take half a minute, even if the changes are minor. This means that coding gameplay can be quite a hassle at times, on par with how map building was in the Penumbra days. With the lessons learned from Amnesia fresh in mind, this felt like the obvious area of improvement.

In order to make this happen we had to move as much gameplay code as possible into the scripting. What this meant was that we needed to do some large upgrades to our current script implementations. For example, right now we only supported the most basic types (bool, int and float) together with strings in script. This already caused some issues when exposing game/engine functions and when writing scripts, for example instead having a single argument for color, you had to have four floats (one for each color and one of alpha), making code ugly and writing it more cumbersome. So just this upgrade was worth doing.

We also needed expose all engine classes, so that the script could be used pretty much as if it was normal C++ code, and achieve pretty much the same things. I was not sure exactly how much to expose but knew that the more the better.

Finally, the most important feature was to be able to reload script at any point, so it would be easy to just change a line, click a reload button and then a few seconds (or less) later see the change in-game.  To get this working was the by far the most important goal with the entire upgrade. This would not be as easy as the level script was in Amnesia though, since the script system would not only take care of the code, but of part of the data as well. This meant I needed to save the state somehow, a feat I was not sure yet how to accomplish.

Implementing Classes

Before tackling the problem of script reload, I first had to make sure to add engine types to the engine. And even before doing that I needed to be sure our current scripting middleware was up to the task.

The scripting system that we were using, and have been for a long time, is a library called Angel Script. It is actually the middleware that we have used the longest, ever since end of 2004 and the Energetic project. Even though we have used it for such a long time we never really used to anywhere near its full extent and now was finally the time to make up for that. I took a day to look through the documentation and found that AngelScript could support everything that we needed, and what was even better was that it was not that difficult to add.

AngelScript is a bit different from other popular script languages (like Lua) in that it is strongly typed and very closely connected to the underlying C++ code. For one thing this makes the script quite fast (although not faster, see end of post for more info). It also meant that AngelScript could link to the classes almost directly and I only had to declare the class and link to its different parts (whatever member variables and methods that I wanted to be exposed). It also supports pointers quite easily, but makes them more secure with a script specific handle type. This requires you to keep track of some memory management for the data, but you do not need to do it, and I could very easily and quickly add support for a vast number of engine resources (textures, meshes, lights, etc).

The only thing that was a little bit trickier was supporting inheritance (when a class can build upon another class). Basically you have to redeclare methods every time you add new class that inherits from something. You also need to specify to what other classes this class can be cast to. This might sound a bit of a hassle, but the result is that you can control so the script always has a very close mapping to the code it exposes, something that I was extremely thankful for when implementing the state saving (more on that below). Also, through some use of macros, adding the implemented classes become quite easy.

Basic script layout

The next thing to figure out was the basic structure of the script code. In Penumbra and Amnesia we simply added the functions directly in a script file and then allowed that script file to represent an object. I first thought that this would be a valid design this time again, until I started thinking about how to store the data (meaning any data that should be kept between executions of the script).

In Amnesia and Penumbra the only data that is saved is simple variables like an integer for the number of times the player had stepped on button or similar. This is done through using special functions for all these variables, for example:


SetLocalVarInt(“ButtonPushNum”, 1);

This function saves the data to the game, and when the script is reloaded (meaning destroyed and recreated) the script can easily reference the data again by doing:


int lX = GetLocalVarInt(“ButtonPushNum”);

However, this time the game had to save a lot more complex data, like pointers to resources, matrices and whatnot. At first I actually considered using system with functions like this, but I figured that it would just mean a lot of extra work, and just make things more difficult. Again I thought about the lessons from Amnesia’s reload feature, and thought it was worth trying to find a better solution. What I ended up doing was to force each object to be contained in class, and then let all data to be members of that class. This allowed AngelScript to make copies (needed for enemies and whatever there will be more than one instance of) and also made it was easy to keep track of the members (you simply create an object in the C++ code and can then iterate all its data).

A problem that I realized now was that I needed to have C++ functions that only worked in certain types of classes and only on data for a certain copy. For instance, an enemy class might want a function like IsInLineOfSight(…) to see if it has visual contact with something. However, there were not any functionality in AngelScript for doing this. I could give the script class a template class that forces it to implement certain functions, but I could not let the C++ code act as a base and expose specific functions from it. To solve this I had do some hacking. I ended up using global functions, and to keep track of the currently active object. (Again with macros to the rescue.) The resulting solution was not perfect though, as the global function can be used in any class and not just the one it was meant for. I am still looking into some fix for this.

Saving the state

It was now time to save the state of the script. To start this off I took the naive approach and simply saved the script data directly by copying it. This works very well for stuff like vectors, matrices, etc where it is just a matter to make a copy of the data. But it does not work that well with resources like meshes, texture or even objects like physics bodies, billboards and lights. There is way too much data in those to copy. And if I simply save the pointer I need to make sure that no data has changed when the state is loaded again, or else the saved state will not work.

At the time I was only building the system to work with a script reload only, so these issues did not pose a problem. But a major obstacle popped up when I wanted to save classes defined in scripts code. These where not possible to simply save by copying, because when you rewrite a script they can change entirely. For instance, if the class when saved consisted of two integers, and when reloaded has one string and a matrix instead the data you saved is invalid. It gets even worse if a script class has another script class as member and even more so if script classes are saved in arrays.

I figured I needed to do some kind of drastic change if this was to work. I was sketching on a few systems that could save variables in a separate structure, when it hit me. The system I was working on could not only be used to save the state, but if I implemented it correctly I could also use it for saving. For Amnesia and Penumbra I use a special serialize system that can save classes to file if initialized correctly. It is quite cumbersome, but way better than writing load / save for every single variable. However, as I was to do most gameplay code through script (the code that pretty much contain everything that needs saving), I could actually get rid of rewriting the save code for every single gameplay update. Another huge benefit of using scripting: automatic saving. This is bound to save tons of work. (It did mean a lot more work though…)

Up to that point I had looked at the scripting as separate part of in the engine. A module that all other modules could work without if removed, exactly like how most other modules work. Thinking like this had colored a lot of the design decisions I had made. Now, I instead decided that since scripting will be basically what controls the engine, I started of thinking as something that was part of all other modules. This led me to do a major rewrite of the state saving.

There is quite a lot to say about this system (and serializing in general), but I do not really have the space right now, so I will just do a quick overview. First of all, the system defines a few basic types that all other structures are built up from. These are bool, int, float, vector, matrix, color, etc and I implement very specific create/save/load methods for each of these. In AngelScript all of these are also implemented as primitives (built-in types) or data objects (which is a type where AngelScript handle all memory management). When saved the actual binary data is saved.

All other classes are implemented as references, meaning AngelScript only manages their pointers. When saving the state of these classes, each class must implement a a special method that adds all of the member variables (either basic types or other reference classes) to a list. (Basically the explicit method described here). This list is then used when transferring data to a special save buffer that contains the data of all the basic types and/or sub buffers containing data for other classes.

For certain classes, its pointer is backed up in the C++ code. So when the save data for it is loaded, the pointer to the class is first searched for and if found it is updated with the saved member data. This makes it possible to reload the script code without having to recreate all the data.

Of course, all sorts of complications arise during this, and again it would take too long to go through them all. But as I hinted before one thing that saved me is that script objects are so closely connected to the engine data. When writing code that deals with serialization one of the biggest annoyances is that that class pointers with multiple inheritance can change their address when cast. Because the script language only returns void pointers (memory address with no type specified), you must be sure of the type it returns, something AngelScript allowed me to be.

Another fun thing with serialization are all the strange macros that you have to make. For example, this one was pretty fun:


#define ClassMemberClassOffset(aClass,aMember)

( size_t(static_cast(&(( (aClass*)1)->aMember)) ) -1 )

This one is used to make sure that class member offset address points correctly. It is one of the many things that make sure multiple inheritance problems, as explained above, does not screw things up. I got an entire header file just filled with fun stuff like this!

Closing notes

Once I had the system working it was quite an effort to add all the classes. There are countless classes that all need to be set up in specific way and everything to be saved needs to be initialized. Sitting several days in a row just coding stuff like this can really tear on the psyche. Fortunately, almost everything is added now, so the worst part should be over.

Another nice addition to the new script system is that it can auto generate the API function file for NotePad++. This allows NP++ to autocomplete any text that you write and you no longer need to keep function names or arguments list in your head (and skip the manual look-ups). What is annoying though is that it only works with global functions and NP++ cannot recognice what class a certain variable is in and show all its members (like visual assist can for C++). Does anybody know an editor that can manage this?

One thing that I was really eager to get working was threading. In Amnesia we use tons of special timer callbacks to set up events, and it would have been so nice to do something like this instead:


PlaySound(“x”);
Sleep(10)
PlaySound(“y”);
WaitUntilTriggered(“a”)


AngelScript does support this sort of thing, but there is currently no way of saving the state. So if the player where to save during a sleep, the game would be different when loaded. This is unfortunate, but I am looking into some other solutions instead. If anyone has ideas on this please share!

Right now I have only tried this system in some tests, but it still feels really good and works exactly as I want. It is just so great to be able to reload any code changes in a fraction of second. It allows for rapid iteration, makes one more productive and generally just makes it so much more enjoyable to code. It is also so gratifying to have an idea and then successfully implement it to the sort of quality you hoped for.


Addendum:

I earlier wrote that AngelScript was faster than Lua, which is not the case when it comes to code solely executed in the script (see here). So obviously my initial statement was not correct. However, I still think AngelScript must be faster in calling implemented c++ functions/types as there is pretty much no overlay involved. I do not have any data to back this up though, so do not take my word for it 🙂

Penumbra: Overture goes Open Source!

Some links in this article have expired and have been removed.

It has been in our minds for quite some time and now finally it has become reality: Penumbra:Overture and the HPL1 Engine are now open source! In case that is all you need to know, then head straight to: http://GitHhub.com/FrictionalGames to get your hands on it!

For more information just keep on reading.

Penumbra: Overture

First of all I would like to stress the fact that this open source release does not make Penumbra: Overture free in anyway. All assets (except a few that are part of the engine) are still under the same copyright as before. The thing that is free is the source code for the executable which is now released under the GPL version 3 licence.

The code for Penumbra: Overture is a continuation of the one used for the tech demo + some addition for the not so long lived Robo Hatch project. It also contains some code from Unbirth, giving it quite some history. This history means that the code is far from clean and as expected quite hackish in places. That said, it should have a few interesting bits, the major probably being the physical interaction system. This system is not the latest version in the Penumbra series and misses something like rotation. These features should be fairly easily to add though.

It is also important to note that Penumbra: Oveture source will not run Black Plague or Requiem. AI for the infected, GUI elements, etc are all missing, but all needed to implement them is present in the engine code (in case anybody is up for the challenge).

HPL1 Engine

For my part, the biggest part of this release is the engine itself. This is engine that has powered all of the Penumbra games and it even includes the stuff used to create the 2D platformer Energetic. The engine code was started in December 2004 and was actively developed until early 2008. After that only smaller fixes where made to it.

The transition from 2D to 3D and the fact that it was my first stab at a full 3D engine, makes the code quite patchy (and downright horrible) in places. This is especially true for some old and low level parts like the sound and input handlers which have evolved anything but gracefully. Later parts are often cleaner and nicer, but the code is not without its share of quick-and-dirty-hacks.

When it comes to interesting features, I think the following are the most prominent:

  • Physics sound system. This is all the code that is used to play and tweak the sounds heard during physics interactions (bumping, sliding, etc). A lot of work has gone into the system and is the result of a combined effort between me and Jens (who does all sounds) for several years.
  • Stencil shadow volumes. The shadow system in the engine is quite robust and can work on pretty much any mesh, something that shadow volumes usually don’t. It is far from state of the art these days (when shadow maps rules), but should provide nice info for the curious.
  • Serialize Class system. This is code used to easily save and load classes to disk. It is very useful when creating a save system for games.

The engine contains tons of more stuff (almost 100k lines of code), but these are the most interesting stuff I could think of right now. I am sure there is more for awaiting to those brave enough to explore its dark depths!

Finally, it is also worth noting that this is the latest and final version of the HPL1 engine. It is the beast that powers all games of the Penumbra series.

OALWrapper

This library is made almost entirely by our tool programmer Luis, who actually started out doing work for us on the OpenAL sound implementation. This library is simple and easy way of accessing the OpenAL functionality and was made since there is a strange lack of free sound libraries. This is also the only part of the open source release that we will continue to update and we hope that others will find it useful and contribute themselves!

Questions and Support

As we are a small company and already swamped with work, do not expect us to do full time support on this. We will try and help as much as possible, but we also hope that a community will form helping each other out.

For more discussion on the source code please go here:

This is the place to go for more technical information and please ask any lengthy / technical question there instead this blog.

Credits

I must also stress that I am not very experience managing an Open Source project and most work in making it happen is due to Edward Rudd, who is also responsible for the Linux and Mac ports of Penumbra. So big thanks to him!

End notes

We are extremely interested in seeing what people will be able to do with this! Open source releases of other games have spawned very exciting stuff and we hope this release will do that too. This means we are very interested in what people are doing with it. So whether you plan to do a full blown mod, or just check the code for learning purposes be sure to tell us about it!

Maps with s-tile

Originally posted by Luis.

The tool guy is back with some more dirty inside secrets on the development. This post was meant to talk a bit about the Level Editor, but first I need to tell you about something you might not know… and it’s called tilemapping.

The term tilemapping refers to a technique born in the mid 80’s or so, back when videogames were pretty much down to 2D. These games used 2D images to represent the game world and entities in it. A 2D image is, in a nutshell, a grid of color values or pixels, with the following parameters:

  • Size in pixels (width x height): these values tell how big our image is – 64×64, 320×200, 1280×800…
  • Color depth/bits per pixel(bpp): precision used to store an individual pixel in the image – 8 bpp, 16 bpp, 24 bpp, 32 bpp. This parameter pretty much depends on the format we are using to display our image.

To understand the need for such a technique, we have to think in terms of the hardware available back in the day. We are talking about machines with veeery limited resources, i.e. real slow CPU’s and quite low on memory (far from a single megabyte), so one had to be really careful and always keep an eye on those limits when developing… and even more if developing a game.

Many older 2D games out there display a quite big and detailed world where the game action takes place, and with big here I mean many times the screen (Turrican‘s huge maps being a real good example of this). So, let’s do a simple example:

  1. Figure our system has a graphic mode with a screen size of 320×200 pixels and 8 bpp for color depth (and we are talking high tech here).
  2. Now, we want our game to have maps with a moderate size of 960×200 (that’s only three times the screen width). The first solution that comes to mind is making a 960×200 drawing that represents the map. Doing the math, that is 960*200*8 bits = 960×200 bytes = 187.5 KB in memory at runtime. Not that much, I agree, but back in the day we could have limits like 256 KB in our main memory, so in this case we would have little room left for the rest of the game data (not to mention the worst and most common case would actually be having even lower limits and much higher needs)

So here’s where the tilemapping technique comes in. The technique itself is rather simple: we have an array of small (e.g. 16×16, 24×24,…) images called tiles, containing all the graphic details we need in our level, plus a 2D collection of integer values (almost like an image), which would act as the actual map, indicating which tile goes where. In our example, using a moderate set of 20 different 24×24 tiles and a 20×9 sized int array, we manage to build a big enough map to fit our needs, with a cost of 20*24*24 + 20*9*2 = 11880 bytes = 11.6 KB. Now that’s saving memory, don’t you think? ;).

Although nowadays we have tech advanced enough to kiss tilemapping goodbye, it’s still used, specially in games for small and limited devices such as cellphones and hand-held consoles. Actually, Fiend and Energetic use tilemapping for levels 🙂

Little 2D horror gem
The Energetic map editor, in all its glory

After having bored you all with all this seemingly pointless babble, I’ll tell you how most 3D games do for levels. There are many ways of having a 3D world. Most common is creating a huge 3D model in the modeling program of choice, that is all the geometry and texture mapping done there. While this is cool, any 3D artist out there would say it is a pretty time-consuming task, and doesn’t allow much reuse of stuff.

I know it looks untextured and that… but does this ring a bell?

What we are doing in Unknown is to create several sets of 3D pieces and make maps using them as building blocks. One can have a production-quality map in less than four days with this method, and thanks to the excellent job from our artists, the results are really nice to the eye as well.

Quite impressive what you can do with such simple pieces I must say… kudos to Jens

As you might have noticed, there’s quite a similarity in both cases. We can make a close analogy between the big 2D drawing and the huge 3D model map, and our current mapping method to the ancient tilemapping technique, although our point is not related to saving memory really, but to save time and sanity. Still, our technique allows for more flexibility, such as going back to the “huge 3D model” totally (by creating a piece containing the whole map geometry), or partially (just a room). Also, we have a grid, but only to ease the task of aligning pieces, so we are not constrained by it.

There’s little left to add really. I hope this served at the very least to give you guys a little overview on one of the most famous game development techniques that has been around for quite a lot of time, and how its basic principle can still apply to today’s methods.

Just couldn’t help myself

Happy Holidays!

Originally posted by Jens.
Links in this article have expired and have been removed.

Another year soon at an end and here at Frictional we are closing to take a nice vacation until the beginning of next year. Time really goes by fast as on Thursday next week Frictional Games will turn two years old! That is based on the date the company was registered, another view at it could be when we first started to create a game with an early version of our current game engine. In that case FG is now over four years old as we begun work on Energetic in December 2004!

Looking back at the year we have to mention IGN’s 13th Scariest games of this generation, a list were Penumbra: Black Plague ends up as number 4! We also would like to point towards the nice honourable mention of Penumbra: Black Plague by Gamasutra in their Best of 2008: Top 5 PC games.

Happy holidays from us here to everyone out there!

p.s as you can see the work on the new game is progressing well, latest added feature is high quality rendered particle snow flake effects! 😉

Texture Atlas

Originally posted by Thomas.

The texture atlas was one of the first thing that I implemented in the engine. It was actually something I started working on the day work started for Energetic, which also means the code is a bit messy 😀

 So I decided an update was needed.

So what is a texture atlas? Basically it is something that has been made to solve two major problems when rendering 2D GUI stuff (fonts, HUD, menus,etc):
1) One usually want this kind of graphics in non-power-of-two sizes like 23×43, 3×19, etc. But these kinds of sizes are not supported by most graphics hardware.
2) One wants to draw alot of different images on screen at once. However, 3d cards work best when a lot of things can be drawn at once and having 100 images using different texture is going to be slow.
The solution is to many images sharing the same texture. That way you can have any size you like and many images can be drawn at the same time!

One of the biggest problem with the current code was that a deleted image continued to use memory until all of the images sharing the same texture was deleted. To fix this I decided to add all images on the same texture again once an image was deleted and skipping the deleted image. While working on this code, it hit me that I could sort the images before adding them again, making them fit better, which would save quite a bit of memory. Since I could not do this every time an image was added, I decided on only sorting the images on a texture once an new image did not fit and then see if it fitted after the reorganisation. This way turned out to work great.

So now the engine has a system that can delete images without wasting memory and also saves alot of memory when saving the bitmaps 😀

Attached are pictures of one texture with unsorted images and one where the images are sorted.