I got carried away while deleting some code that didn’t do anything and accidentally removed some parts that did do things — function calls with side effects that I didn’t fully understand. I make the same kind of change later to a different part of the codebase and got it right there. I didn’t think to go back and look for the possibility of this problem in my earlier change.

This one got checked in and caused problems for others later. It would have been caught by better testing on my part. (I’m convinced that I did test this and saw no problems, so either I didn’t actually test a build with the change or I my testing wasn’t really testing of the code affected)

Unravel the whole garment

In a related deletion, I managed to break another piece of code connected to one that I was modifying. I broke one of the assumptions of the other piece of code, which was far less obvious to me because they interacted via a mechanism that I was not familiar with.

Again, this would have been caught by better testing on my part.

My apologies to those affected. Clearly, I suck.

(On the upside, I’ve now learned a little bit more about another part of the codebase. This is a good thing.)

Summary of the weeks

Wednesday 2013-03-27

An optimistic beginning to the new week: no bugs to write about, so I posted a photograph.

Thursday 2013-03-28

An example of using the compiler to help identify particular code patterns that I wanted to eliminate.

Friday 2013-03-29

An actual bug, and worse than a runtime bug: checked-in build breaker. Shameful.

Tuesday 2013-04-02

• Implicitly finding bugs in other people’s code

Sunday 2013-04-07

Not at all bug related, but a point of curiosity that (thanks to a lot of input from others) turned into a interesting list of studios that have been around for more than twenty years.

Trends

Not a many bugs in the list this time, so doesn’t seem like there’s a lot to say about these two weeks — except that the rate of bugs does go down when you’re not writing any code…

• HAL Laboratory (1980)
• Falcom (1981)
• EA Canada (1983)
• Nintendo Entertainment Analysis and Development (1983)
• Naughty Dog (1984)
• Ubisoft Reflections (1984)
• Rockstar San Diego (1984)
• Rare (1985)
• Bethesda Softworks (1986)
• Codemasters (1986)
• The Bitmap Brothers (1987)
• Creative Assembly (1987)
• Eutechnyx (1987)
• Maxis (1987)
• Rockstar North (1988)
• Ubisoft Blue Byte (1988)
• Climax Group (1988)
• Ubisoft Montpellier (1989)
• Toys For Bob (1989)
• Vicarious Visions (1990)
• Team 17 (1990)
• Blitz Game Studios (1990)
• Edge Games (1990)
• Raven Software (1990)
• Blizzard Entertainment (1991)
• Epic Games (1991)
• id Software (1991)
• Bungie (1991)
• Radical Entertainment (1991)
• DICE (1992)
• Rebellion (1992)
• Crystal Dynamics (1992)
• Treasure (1992)
• Digital Extremes (1993)
• Volition (1993)
• Funcom (1993)
• Stainless Games (1993)
• Criterion Games (1993)
• High Voltage Software (1993)

(Current count: 39)

There are other game companies (including publishers) such as Square Enix (2003), Ubisoft (1986), Capcom (1983), EA (1982), Activision (1979), Taito (1973), Konami (1969), NAMCO (1955), and Nintendo (1889) that have been around for more than twenty years, but I don’t know if they have [other] studios that have been running that long.

What is the significance of a short list of game studios?

A couple of things happened recently that got me thinking:

1. I watched Hackers. The movie features a scene with footage from a prototype of Wipeout, and the movie credits Sony Psygnosis. Psygnosis was founded in 1984, acquired by SCE in 1993, became Studio Liverpool in 1999, and was closed in 2012. (and plenty of other things — go read the linked article)
2. LucasArts (Lucasfilm Games), formed in 1982, was closed down.

Were they to be continuing, these two names would appear near the top of the list above. LucasArts was one of the longest running game development studios in existence at the time of its closure, behind only to HAL Laboratory and Falcom.

The closure of a long-lived studio is a significant thing. Foremost, it has a huge impact on the lives of its employees. But it is a loss, I believe, to the industry as a whole.

(I’m not looking to make a judgement about whether LucasArts should have been closed or not. Whatever the reasons for the closure — and I know nothing of them — it has happened.)

The passage of time

There are clearly some benefits to a little bit of age.

Top of my list is experience. The ability to reflect on a long sequence of games — as a team. To learn from that experience, and to try to make better decisions for future ones. Having a group of people who have worked together over an extended period of time, who may not always agree (or who may rarely agree) but who continue in their commitment to working together to make the best games they can make together. The evidence of investment.

Experience making games is one thing. Experience making games with largely the same group of people over many years is another. To know who to ask about a particular ancient system. To know some of the history that has led to otherwise inexplicable artifacts in the technology, tools, and processes of a studio. To have a shared understanding and investment in the company’s accrued building blocks of game development.

And it’s not all shiny pony rainbows — not only good things accrue over time. Investments are made based on decisions that once may have made sense but now may not. Finding agreement and goodwill on all things across any company is nigh impossible. And so there is much effort required to continually move an organization forward.

The list of studios older than twenty years seems pretty short to me, but they’re all evidence of companies that have been able to grow, change, adapt (in some cases shrink), to move forward in response to outside pressures: changes in technology, changes to the market, to distribution models, and more. There is the possibility of endurance within the industry.

But does this endurance matter? Is it important that we have companies that can endure, and change over time? Or is it enough that teams can come together for a time, make a game and disband again only to reform for another project?

One consequence of persistence is the development of a shared identity — a team culture. The establishment of trajectories, good and bad, that are hard to break. These identities become crystallized in the games we make: the passions, the neuroses, the striving for technical excellence, the pursuit of new experiences, the desire to improve upon the existing, the decisions in the face of so many limitations, these things are reflected in the character of the games produced by a team.

Another consequence is an often implicit snapshot of shared history. Few game developers and publishers seem interested in preserving or enhancing the games they’ve made beyond a short period of profitability. Archiving — particularly, providing a public archive of games — is almost unheard of. A small number of companies release source code. Some interested parties attempt to archive older games, to preserve some part of our gaming history. But there are many more games that are created for a time and live only at the whim of their operators, with essential servers lasting a year or less. And just today I see a case where publishers are unable to identify the owner of a game.

In the absence of a widespread commitment to preserving our developing history, longer lived studios are at least some visible evidence that we have that history captured in some form, good or bad.

And there is financial benefit in being a studio that has persisted. Especially one that has made popular products :) An established identity is a valuable thing — no small part of my motivation to write about this comes from the fact that I have a lot of good memories associated with both LucasArts and Psygnosis.

Is it important?

I don’t know. Maybe it doesn’t matter if there are many “older” game studios.

What I do know is that I want studios thrive and live long, productive lives. I want the existence of older game studios to matter. I want games to be more than just one-time projects worked by a group that has come together for a short time only. I want teams of game developers persisting, struggling, wrestling to go beyond what they can do on their own. To inspire one another and to achieve unimagined heights.

I want game development to take itself seriously. To record and preserve its history. To provide a the shoulders for later developers to stand upon. I want there to be companies that persist, to be a foundation for the industry. To invest in talent and technology, and to make it available for the benefit of the industry (one way or another).

I want an industry with substance and depth, with studios capable of investing in long-haul projects, able and motivated to be looking long years into the future to consider what they might yet achieve and how to get there.

Selfishly, I want to make games, and I want a secure, reliable job to be able to work for the well-being of my family. I want there to be workplaces that I can commit to, invest in, and that I can trust to be around for years to come. I want to work with experienced, exceptional people. I want to be in that struggle to make great games, with others, for a long time.

I am convinced that the presence of long-lived studios is a good thing for this industry. It is my hope that the list of twenty-and-overs will grow.

—
While not a gamedev, SN Systems has been providing tools for game development since 1988 :)

—
Please let me know of any corrections or additions to the list — it’s based on wikipedia entries and tweets from people I do and don’t know.

My thanks to twitterers 666uille, JanDavidHassel, SamuelePanzeri, Mimus_, brett_douville, girayozil, Nshk, Allagash, pthiben, RonPieket, and michaelellerman for their suggestions, and to my dear wife for listening to me ramble through some of these thoughts.

Update 2013-04-07-6: Added Namco, Falcom and Treasure. Thanks to @janvanvalburg

Update 2013-04-07-5: Added Radical and High Voltage Software. Thanks to @TheJare

Update 2013-04-07-4: Added Criterion, Eutechnyx, Edge, Raven, Maxis. Thanks to @SebHillaire, @choddlander, @ChristerEricson, and @BrianKaris.

Update 2013-04-07-3: Added Blitz Game Studios, Rebellion, Climax, Stainless, Crystal Dynamix, Taito. Thanks to @zebrabox, @ggatheral, @HazelMcKendrick, @wuffles, @thinkinggamer, and @jasperbekkers.

Not sure about Square Enix – merger of Square (1983) and Enix (making games since 1982) in 2003.

Update 2013-04-07-2: Added Codemasters. Thanks to @Poita_.

Update 2013-04-07: Added Funcom. Thanks to fred r.

Update 2013-04-06-2: Added EA Canada, Ubisoft Blue Byte. Thanks to @h4kr and @p1xelcoder.

Update 2013-04-06: Added Bungie, Digital Extremes, HAL Laboratory, Nintendo EAD, Rockstar North, Rockstar San Diego, Team 17 and Volition to The List. Thanks to @alexr, @msinilo, @chrisjrn, @flawe, @morten_skaaning and @fahickman.

Added SN Systems as an addendum, thanks @gregbedwell.

A crash was reported with a call stack leading to some code I’d been making changes to a week ago (code that previously featured in Unexpected use of class and Initialized, but not in the right order).

The code I’d changed now looks something like this:

struct Foo : public Baz {
  Bar* m_Bar;
  Foo() : m_Bar(NULL) {}
  void SetBar(Bar* bar) { m_Bar = bar; }

  virtual void Destroy() override;
};

The cause of the crash was something trying to make use of a null m_Bar member of a Foo object.

With regard to the previous crashes I’d encountered, my first suspicion that this was an object that was somehow being created in a way that had not correctly initialized the m_Bar. Checking the circumstances of the crash (the reporter was able to find a specific cause, which was a great help), I was unable to see a way that this object could have been created with m_Bar remaining null.

Sidebar: A way in which my debugging skills have improved in the last year

I have identified the following tendency in my thinking: see a bug, take a guess at the cause, and then look for that cause in the following way:

1. Examine some part of the codebase.
2. If expected cause not found, goto 1.

So if I guessed the wrong cause, I get stuck in a loop: “It must be X, but I haven’t yet found how X could happen — I must look harder!”

Part of this came, I think, from a degree of insecurity about my own ability to read and understand code: “the bug is in here, but I must have missed something”

What I’ve realized in the past year is that my code reading skills aren’t that bad — if I haven’t found the cause I’m looking for, it’s more likely that I’m looking for the wrong cause. Particularly, I am now able to approach a problem with a hypothesis about the cause, and can more clearly identify when I’ve found some information that invalidates that hypothesis and I should be looking for a different cause.

In the context of this bug, validating how the object was being created and seeing that the codepath would always validly initialize m_Bar meant that the bug was not in the initialization — it must be somewhere else.

The other nuller

So something else was setting m_Bar to null after the initialization. There is one other place where that happens…

  virtual void Destroy() {
    if( m_Bar != NULL) {
      m_Bar->Destroy();
      m_Bar = NULL;
    } else
      HARD_TO_MISS_NOTIFICATION( USEFUL_WARNING_37 );
  }

So, it looks like there’s a good chance that this object has had its Destroy() called, and looking through the code, it becomes clear that this is indeed what has happened.

• There was a Validate() function called on our object that checked certain things about that object’s state, calling Destroy() if it did not meet certain conditions.
• Validate() was called before attempting to make use of m_Bar.
• The bool returned by Validate() is ignored by the calling code.
• The circumstances that lead to the crash are such that Validate() will always call Destroy().

One if(!Validate()) return; later, the crash no longer occurs.

This bug came from a combination of circumstances:

• existing code that ignored the logical (? I guess that’s the right word…) validity of the object — it had been destroyed, but still contained valid data, and
• my change with caused the logical state to be reflected in actual state.

I did examine all the other callers of the Validate() function — none of these exhibited the same type of problem.

Not all code is compiled with the same warning levels everywhere, so something that compiles without warning in one application with a lower warning level may not compile in another with a higher warning level (assuming warnings-as-errors).

After finding no-longer-needed uses of a particular macro, I did not delete my finder code. But taking the address of a class’ member function and not doing anything with it can be warning-inducing. Bah.

And one of the former users of the macro was left with a no-longer-used variable after my changes,  This was code I had failed to delete on the first pass. I am disappointed with myself to have not deleted more code when I first had the chance.

How should I avoid such problems in the future? Compile code I change for all targets/configurations/applications that matter.

Sometimes one can be too lazy.

class Foo {};

// Bar is one of many classes that inherit from Foo
class Bar : public Foo {};

// Baz is one of many classes that inherit from Bar
class Baz : public Bar {};

DECLARE_EXTRA_STUFF_FOR_CLASS(Baz);

So we have something like the above. The DECLARE_EXTRA_STUFF_FOR_CLASS() macro is used for some but not all classes that inherit directly or indirectly from Foo.

There have recently been changes within a system that means that DECLARE_EXTRA_STUFF_FOR_CLASS() is now never needed for any class that inherits from Bar — but it is not invalid for it to remain used in the codebase. Anything that makes use of the EXTRA_STUFF that the macro provides will still work, it’s just extra unneeded code which I’d like to clean up.

I could audit this by hand — find every class inheriting from Bar, check to see if there’s a macro definition for that class somewhere and perform the subsequent clean up, but I’m lazier than that.

I could write a script that finds classes that inherit from Bar, and then search for that class name with the macro, but that also seems like to much work. (A simple regex sweep through the codebase won’t trivially identify class Asd : public Baz {} as inheriting from Bar). I’m lazier than this.

What I can do is break the code expanded by the macro if it is passed a subclass of Bar. To do that, I can do something like the following:

class Foo              { public:  static void Foo(){} };
class Bar : public Foo { private: static void Foo(); };
// Add a line to the macro...
DECLARE_EXTRA_STUFF_FOR_CLASS(class_name) \
    extra_stuff; \
    class_name::Foo(); \
    more_extra_stuff

Done. Anywhere where the macro is invoked for a class inheriting from Bar will trigger a compile error. Walk through the errors, delete calls to that macro, rewrite (i.e. delete) code as necessary.

End result: less code. I count that as a win.

(In my specific case, I made an already existing public method declared in Foo private in Bar because there is no need to call it from or for classes in that part of the hierarchy. This has the added advantage of not polluting the list of Foo‘s members with no functionality of the class. Additionally, I didn’t call the function within the macro but instead took it’s address.)

struct A { void f(); };
struct B { virtual void f(); };
struct C { virtual void f() final; };
void a(A& a) { a.f(); }
void b(B& b) { b.f(); }
void c(C& c) { c.f(); }

like this

g++ call.cpp -std=c++11 -O3 -S -o- -masm=intel # gcc-4.7.2 i686-pc-cygwin

which produced [output that included] this

__Z1aR1A:
    jmp __ZN1A1fEv

__Z1bR1B:
    mov eax, DWORD PTR [esp+4]
    mov edx, DWORD PTR [eax]
    mov eax, DWORD PTR [edx]
    jmp eax

__Z1cR1C:
    jmp __ZN1C1fEv

What does it mean? Because C::f() is marked final, calls to it can be devirtualized by the compiler, so there is no extra indirection when calling it in the above case.

(Trying the same code with VS2012′s cl.exe, the call to C::f() does not appear to be devirtualized)

struct C does still appear to have a virtual function table pointer: (sizeof(C) == sizeof(B) == 4), whereas (sizeof(A) == 1). It seems to me that the vtable pointer could be omitted in this case…

Summary of the week

Tuesday

• Changed virtual function prototype results in correct function not being called
• memcpy() into an uninitialized pointer variable
• Bad loop counting
• Failing to store updated state

Wednesday

• Uninitialized class member
• Sliced inheritance
• Undesired printf() output

Thursday

• Failed to initialize instance-specific member variable correctly because I was calling the wrong constructor
• Use of uninitialized system
• Yet another uninitialized member variable crash

Friday

• Unexpected use of class
• Hard-coded array size, no checking
• General disaster

Monday

• Initialized, but not in the right order (fallout from Unexpected use of class)

Tuesday

• Local static state is local, static
• Redux: Changed virtual function prototype results in correct function not being called

Trends

Looking back over the week, failure to initialize, failure to init at the right time or in the right order stands out as a primary source of bugs. Maybe I need to be asking myself questions like “where did this variable’s state come from” to try and prompt more consideration of this.

Broadly, there is also what appears to be insufficient familiarity with various of C++’s OO features, resulting in silly mistakes. I don’t generally write much OO-style code — this week has been unusual in that way. It is a problem if I can’t use them reliably.

I’m pretty happy with the solution to the “Changed version prototype“ situation. Finding ways to prevent possible runtime bugs at compile-time always feels like a worthy achievement.

Meta

Thus far, I’ve found the process of logging these bugs to be worthwhile — attempting to explain the bugs helps me to understand them more clearly. I have found myself thinking about the bugs: the solutions I’ve attempted, and how to try to avoid writing the same bugs in the future.

My plan for now is to continue to write these posts. Hopefully, it leads to fewer bugs.

Local static state is local, static

int Foo() {
  static int timeout = kTimeoutLimit;
  bool polled_condition = CheckExternalCondition();
  if( !polled_condition )
    --timeout;
  if( timeout == 0 ) { 
    printf("condition not set before timeout");
    return kTaskFailed;
  }
  if( polled_condition )
    return kTaskComplete;
  else
    return kTaskIncomplete;
}
void Bar() { 
  while( Foo() == kTaskIncomplete );
}

Sometimes a local static variable is a useful thing — conveniently quick to add to any piece of code, and localized to a particular scope. You know everything about what can happen to it during its entire lifetime — assuming you know about how the scope it is contained within operates. (and I’m disregarding the returning of references to local static variables from a function. That’s a whole different bag of fun…)

Quickly rewriting some code today, I added a local static counter — counting down each time a desired condition was not met, and triggering a printf() should the count ever reach zero. I was using this to diagnose that something unexpected had happened, and that some event had not taken place.

The above code should work adequately well — if Bar() is only ever run once (and only from one thread at a time — but that’s not the issue here). timeout is never reset, so unless CheckExternalCondition() returns true, first time, every time, the value in timeout will continue to decrease with each run of Bar(). If it drops below zero, there will no longer be a limit on the on how long Foo() can loop.

A fix here is to make sure timeout is reset each time Foo() “completes” i.e. when it returns kTaskComplete or kTaskFailed.

More generally, this does not seem like a good use of a static local variable — relevant context for the function is not readily apparent within the function, so it seems better to store the timeout value outside of Foo().

(The actual code is more convoluted than the example above, so it’s not quite as simple as adding a local timeout var in Bar(), but only just)

Redux: Changed virtual function prototype results in correct function not being called

This was a bug where I changed the prototype for a virtual function, but that classes with the old prototype would not be detected at compile time and would silently do nothing should the virtual function be called. Read the details via the link above.

I had added an assert to catch any classes that didn’t have the new version of the function, but that only worked at runtime, and only when the function was actually called. What I really wanted for this was a compile-time check that was more reliable than the inheriting programmer remembering to use override. I discovered that there is a way to do this with C++11: final.

class Base {
public:
  virtual void Foo(int some_param, WhatsitType some_other_param)  {  }
private:
  virtual void Foo(int) final { }
};

With this, anything that inherits from Base and attempts to define its own void Foo(int) method will result in a compile error. I’m much happier with this, and making this change revealed some extra instances of the old function that I’d missed in my previous sweep. Huzzah.