Game Development Needs Data Pipeline Middleware

In 15 years I’ve worked at 7 different game studios, ranging from small to large, working on many different kinds of projects in a variety of roles.

At almost every studio, there was some way for the game to load data at runtime that controlled how it behaved – such as the damage a weapon would do or the cost of an item upgrade.

The studios that didn’t have this setup could definitely have benefited from having it. After all, this is how game designers do their job!

Sometimes though, this data was maintained via excel spreadsheets (export as csv for instance and have the game read that). That is nearly the worst case scenario for data management. Better though is to have an editor which can edit that data, preferably able to edit data described by schemas, which the game also uses to generate code to load that data.

Each studio I’ve worked at that did have game data each had their own solution for their data pipeline, and while they are all of varying qualities, I have yet to see something that is both fast and has most of the features you’d reasonably want or expect.

We really need some middleware to tackle this “solved problem” and offer it to us at a reasonable price so we can stop dealing with it. Open sourced would be fine too. Everyone from engineers to production to content people will be much happier and more productive!

Required Features

Here are the features I believe are required to satisfy most folks:

  1. Be able to define the structure of your data in some format (define data schema).
  2. Have an editor that is able to quickly launch, quickly load up data in the data schema and allow a nice interface to editing the data as well as searching the data.
  3. This edited data should be in some format that merges well (for dealing with branching), and preferably is standardized so you can use common tools on the data – such as XSLT if storing data as xml. XML isn’t commonly very mergable so not sure the solution there other than perhaps a custom merge utility perhaps?
  4. The “data solution” / project file should store your preferences about how you want the final data format to be: xml, json, binary, other? Checkboxes for compression and encryption, etc. Switching the data format should take moments.
  5. There should be a cooking process that can be run from the data editor or via command line which transforms the edited data into whatever format the destination data should be in. AKA turn the human friendly XML into machine friendly binary files which you load in with a single read and then do pointer fixup on.
  6. This pipeline should generate the code that loads and interacts with the data as described in the data schema. For instance you say “load my data” and it does all the decompression, decryption, parsing, etc giving you back a root data structure which contains compile time defined strongly typed structures. This is important because when you change the format of the data that the game uses, no game code actually has to know or care. Whatever it takes to load your data happens when you call the function.

Bonus Points

Here are some bonus point features that would be great to have:

  1. Handle live editing of data. When the game and editor is both open, and data is edited, have it change the data on the game side in real time, and perhaps allow a callback to be intercepted in case the game needs to clear out any cached values or anything. This helps iteration time by letting people make data changes without having to relaunch the game. Also needs to be able to connect to a game over tcp/ip and handle endian correction as needed as well as 32 vs 64 bit processes using the same data.
  2. Handle the usual problems associated with DLC and versioning in an intelligent way. Many data systems that support DLC / Patching / Schema Updates post ship have strange rules about what data you can and can’t change. Often times if you get it wrong, you make a bug that isn’t always obvious. If support for this was built in, and people didnt have to concern themselves with it, it’d be great.
  3. On some development environments, data must be both forwards and backwards compatible. Handling that under the covers in an intelligent way would be awesome.
  4. The editor should be extensible with custom types and plugins for visualizations of data, as well as interactive editing of data. This same code path could be used to integrate parts of the game engine with the editor for instance (slippery slope to making the editor slow, however).
  5. Being able to craft custom curves, and being able to query them simply and efficiently from the game side at runtime would be awesome.
  6. Support “cook time computations”. The data the user works with isn’t always set up the way that would be best for the machine. It’d be great to be able to do custom calculations and computations at runtime. Also great for building acceleration data structures.
  7. You should be able to run queries against the data or custom scripts. To answer questions like “Is anyone using this feature?” and “I need to export data from our game into a format that this other program can read”
  8. Being able to export data as c++ literal data structures, for people who want to embed (at least some of) their data in the exe to reduce complexity, loading times, etc.

It should also be as fast and lightweight as possible. It should allow games to specify memory and file i/o overrides.

Localized text is also a “solved problem” that needs an available solution. It could perhaps be rolled into this, or maybe it would make most sense for it to be separate.

As another example of how having something like this would be useful, on multiple occasions at previous studios, people have suggested we change the format of the data that the game uses at runtime. For instance, from json to a binary format. In each case this has come up so far, people have said it would take too long and it got backlogged (ie killed). With data pipeline middleware that works as i describe it, you would click a few checkboxes, recook your data and test it to have your runtime results. That’s as hard as it SHOULD be, but in practice it’s much harder because everyone rolls their own and the cobbler never has time to fix his shoes (;

Anyone out there want to make this happen? (:

How and Why Cleaning Up Code or Processes Gives Multiplicative Benefits

The engineering manager of my team Paul Haban (@XpresoAdct) mentioned to me once in passing that when you fix a problem, you often get multiplicative returns beyond the initial problem you intended to fix.

This is an idea from Kanban, and while it was in my best interest to believe this to be true, since it allowed to refactor personally painful inherited systems and code, it felt like a sort of mysterious voodoo and I wasn’t really a believer.

I recently experienced it first hand though. I refactored something and the benefits started multiplying. People from distant sub teams came out of the woodwork very excited to hear about my changes. Of course, this happens from time to time, and it’s a lucky break to get benefits beyond what you were planning, but looking at it in hindsight, there are some really good reasons why this happened.

This applies to source code, processes, etc, but for simpler language, we’ll focus on this being about code.

What is Bad Code?

Firstly, engineering is often about trade offs. You might see that solving a problem one way gives you certain benefits, while solving a problem a different way gives you other benefits. You weigh those things, talk to those affected to get their opinions in case you are missing information, and then you make the best decision you can with the information you have.

Sometimes you make a decision based on the current state of things, but then the situation changes, and the choices you make turn out to be bad choices for the new direction that things have taken. Now your code has turned bad.

Also of course, people sometimes people just make bad choices. We are human, we are learning, it’s how it goes. Sometimes people just make bad code to begin with.

A deeper chat of this sort of thing can be found here: No Bad Code, Creeping Normality and Social Structure Code Organization

But ultimately, here is what makes code “bad”: If it works less than ideally for someone or some thing that has to interact with it, it is on the spectrum of “bad code”, ranging from terrible code, to code that could be cleaned up, but doesn’t really matter enough to fix.

It also may be that code is bad for one set of interactions, while it is perfectly ideal for another set of interactions. This is the result of the trade offs weighed when solving the problem. That may just be a fact of life that you either can not really do anything about, or that in practical terms, you cannot do anything about due to cost versus reward analysis or whatever else.

Lastly, like in my case, you may have inherited some bad code from someone else. In this case, it could just be that you have different goals, or that you prefer a different trade off (pain flavor if you will) than the previous maintainer.

How Does Bad Code Affect Others

By definition, bad code is code that is less than ideal for a person, or code that has to interact with it.

That means that when they interact with the code two things happen:

  1. There are work arounds that have to be done to be able to get what is needed from the system.
  2. There may be perfectly reasonable things that interactions with the system may want to do that are not possible, or not practically possible with real world constraints.

The more central this bad code is, and the more people that interact with it, the more that there is both workarounds, and desired functionality that can’t be realized.


If you can legitimately refactor some code such that the result is decided to be better than where things are at now – say, there is less pain overall, or perhaps the pain is more concentrated on a group that nobody likes (hehe) – making that happen will make the code less bad.

Again, bad code is a spectrum, so it’s likely you’ll hit situations where the code will never be perfectly good code, but making it less bad is a good thing.

When you make code less bad, however you measure that, it means that the workarounds that needed to go up can start being taken down (simpler code, less maintenance, fewer things that can go wrong), and also, you open up the doorway for the improved functionality that was not previously practical.

Another way to think of it is that the optimist will say that fixing things gives multiplicative benefits. The cynic (realist?) on the other hand says that the less than ideal code has already incurred both a one time cost to the people that have interfaced with it, as well as a continual maintenance cost that is incurred by it existing, and that those costs are or were avoidable.

To me, this explains in a very down to earth way how the “voodoo” of multiplicative benefits actually comes about. It also shows a bit of how continual minor improvement really does add up (the main idea of Kanban), even when not taking into account things like the compound interest model (ie, saving you effort now allows you to save future effort sooner).

Go clean up some code, or fix a broken process. You will likely be surprised at how much benefit you get out of it!

I miss anything or you have a differing view? Let me know!

No Bad Code, Creeping Normality and Social Structure Code Organization

In the last post I said that another post was coming regarding using bilinear texture sampling for something interesting. Hopefully without sounding too pretentious, that interesting thing I wanted to write up has shown quite a bit of fruit, so I’m going to try to write it up and see about getting it put into The Journal of Computer Graphics Techniques. We’ll see how that pans out, and I’ll make sure and share once it’s published (or if I can’t get it published hehe) as well as a post on the process of trying to get something published, in case other people out there are interested in pursuing that sort of thing.

Today’s post is a bit different than usual. There are some interesting concepts that I’ve been exposed to that I think are worth sharing, and that I’m hoping you will also find interesting.

No Bad Code

One idea presented to me recently was the concept that there is no bad code. The idea being that people make decisions in code based on the situations at the time, and that as situations change, code that once was totally reasonable, and most engineers would likely have made the same choices, no longer is seen as a good choice in the new sets of circumstances.

I’m going to be a bit cynical here and mention that I do believe there is bad code, and that it comes around due to bugs, as well as lack of knowledge, lack of experience, and lack of testing before deployment. However, the idea of code that is now seen as bad, was once perfectly reasonable does make sense to me and is kind of interesting.

I know we’ve all hit code that we or others wrote that we despise, causes us grief, and gets in the way of us working what we ought to be working on. This concept definitely does absolve SOME such code I’ve personally had to deal with. But, there is definitely plenty of code left that it doesn’t. In my personal examples, this mostly comes from shoddy third party middleware software, and also, just a lack of knowledge or experience on the part of the implementer. I’m guilty of the second point, but I think we all are, and that is a sign we are learning and growing.

It’s probably good to keep in mind that lousy code of the present may have been perfectly reasonable code of the past, and before deciding that code was lousy, or that a specific engineer is terrible, that you have to judge the code in the light that it was put in.

… Then refactor it.

Creeping Normality

Creeping normality is an interesting concept. Basically, just like that old tale of a frog sitting in progressively hotter water til it boils (which by the way is debunked by snopes), it’s easy for individuals or companies to find themselves in situations that would NEVER be seen as ideal situations, but were arrived at by incremental mis-steps – or also just a changing landscape.

This relates to the last point a bit, because it can show how the needs of a piece of code can change over time, such that looking at it as a static snapshot in time, you might wonder why it’s so complex and trying to do so many different things needlessly.

This can also happen to a company on a more global scale, and can explain some really odd, less than ideal behaviors a company might be doing, where you are pretty sure the people involved know better.

How do you fight creeping normality? Good question… but when you are able to identify some oddness or problems that come up due to it, hopefully it’s as early as possible, and hopefully the people with the power to make things right listen to you (:

Social Structure Code Organization

The last topic is kind of bizarre, but totally makes sense once you think about it. The idea is that code will match the communication structure of the teams making the code.

This is Conway’s law which says: “organizations which design systems … are constrained to produce designs which are copies of the communication structures of these organizations”. Another good quote from the wikipedia page for Conway’s law states that (paraphrased) “if you have four teams working on a compiler, you’ll end up with a four pass compiler”.

This can be seen a lot in situations where you have a game engineering team and an engine engineering team. There will be a distinct line between the code bases, based on the responsibility of the teams. If instead, you have some code where the engine and game team are the same, there will be no such line, and if you then try to split that team into being a game and engine team, it’ll take some time to draw the line between responsibilities, separate the code along those lines, and possibly do something like get the code into separate projects and possibly code repositories, which all seems like a logical choice to do when you have separate teams.

I think this also relates back to the first point about “No Bad Code”, because were someone to come into a team where the organization had recently changed, they are going to wonder why the code doesn’t have nice abstracted API layers at the boundaries, like most folks would consider logical and good practice. Perhaps too, this would be a case of creeping normality, or at least, if the project organization changed, it could be the result of a reaction AGAINST creeping normality.

In short, there are a lot of ways in which perfectly good code can go bad, and it’s probably a good idea to think about that a bit before condemning code as inherently rotten.

HOWEVER, rotten code does exist. I’d name some names, but it would probably be classified as slander, so i’ll bite my tongue. I’m sure you have plenty of examples of your own (;

Lastly, if you are responsible for crimes against code-manity in days past, either due to any of the reasons above, or because you didn’t know things you know now, or even if you were just lazy or misguidedly purposefully malicious (?!), remember this: You are not your mistakes!

OK, enough of that, next post will be on some cool programming technique, I promise 😛

Situational Leadership

This is a little bit different than the normal kinds of topics I write about, but I found it pretty interesting so wanted to share with you guys. (Thanks for sharing the idea with me Paul!)

This is a “theory of management” type of thing that helps explain why people act certain ways in specific situations, and what sort of management style they need to be successful. I might butcher the details a little bit, but if you want to read about it in more depth, you might start with this wikipedia page: Wikipedia: Situation Leadership.

Let’s kick things off with a diagram:

The idea is that basically, when someone is put into a new situation, they will generally progress from 1 to 4(*). This can be either hiring a programmer fresh out of college, or taking an experienced programmer from one team and putting him onto another team, or presumably could also refer to an entire team tackling a new type of problem.

(*) I sort of hate generalizing about how people act, and cookie-cutter-ing people and situations, but I take this as a sort of general guideline, instead of a hard and fast rule for every person and every situation.

Step 1

At step 1, a person is overly confident and think that they know what needs to be done, but in reality they don’t have the skills and/or information needed to actually do what is needed – but they don’t know it yet. A person at this stage needs guidance or pairing with another person to keep from doing reckless things and to keep them on the path towards success.

Step 2

At step 2, a person has learned a bit more about things and realizes it’s a bit more challenging than they thought. There is some drop in morale at this stage, and is where people might contemplate giving up, switching teams, switching companies, or switching careers. At this stage, a person needs “beer and hugs” I’m told. Maybe they also need some smaller, isolated tasks, to give them a sense of accomplishment. Maybe tasks drawing on their passions or previous experience to give them some victories to help them get over the hump.

How often have you found yourself in the mindset? Maybe on a new team, at a new company, working with new technology, and feeling completely overwhelmed, thinking things are too difficult and maybe even wanting to give up, or feeling like you are not preforming well enough?

Chances are, everyone’s felt that way. I know I have! I’ve worked at something like 7 companies in 13 years and early in my career i felt that way A LOT, and OFTEN. It happens less as the years go on, and I find that more skills carry over than in previous days, but it still happens from time to time. That’s a good thing though, because if it didn’t, it would mean I wasn’t learning and growing, and stagnation is no good.

The good news is that this feeling is normal, it’s typical, and if you have a good manager, they’ve seen it many times before and expect it. No, you aren’t under-performing, you aren’t under-qualified, and you aren’t a slacker. But keep working hard anyways so you can get out of this stage!

Step 3

At step 3, a person has begun learning more and is getting more proficient, and starting to feel better about things. Keep on trucking!

Step 4

At step 4, a person has mastery over the subject matter and is feeling good about things. At this point they know what they are doing and are confident, and need a bit of a longer leash to be able to go out into the weeds a little bit to feel good about their work, even if it seems a bit silly. People who are truly passionate about what they do want to be trusted to do the right thing, and they want the freedom to pursue their interests. They’ve worked hard, through challenges both with the work and psychological, and now they are effective and hopefully pleasant – so hopefully they’ve deserved a little bit of diversion time hehe.

Plus! if you’ve done any programming in the areas of genetic algorithms, or training neural networks, you may remember that if you only let the winners reproduce (in genetic algorithms), or you only activate the winning neurons while training (for neural networks), you will likely end up in a local minima, instead of the global minimum. That “longer leash” time of letting people wander into the woods a little bit is kind of like mutation, or letting some of the losers reproduce. They may very well come back with something way better than you ever considered.

Super tangent – someone recently told me that path finding has the same “local minima” problem, and that if you pursue some of the “not so great” paths while searching a pathing space, you can get better results.

Outside of the Box


While this may be a useful tool for helping to understand people’s motivations, and helping to give them what they need to succeed and be happy and effective, this isn’t the whole story of course. Just like there is no such thing as a straight line in nature (AFAIK, but not sure what happens below the plank scale!), this doesn’t exactly match reality. It’s just a useful guideline.

You might also find this interesting, as a different take on the transition from step 1 to step 2: Wikipedia: Dunning-Kruger Effect

Recently I’ve been thinking about a few topics like this that are game dev related, but not about specific algorithms. You’ll probably see some more of this sort of thing smattered in amongst the cool algorithms I stumble on going forward (: