December 12, 2004
Reinventing the wheel
In thinking about using other languages in games, I can't help but consider inventing a new language tailor made for the higher level aspects of game development. This is exactly what Tim Sweeney did with UnrealScript for the Unreal engine. I have worked on a title that used the Unreal engine and it certainly improved programmer productivity for that project. But what worked for Epic might not work out so well for the rest of us.
Lets way up the pros and cons. The obvious advantage is that you get a language that is exactly what you need, or at least exactly what you think you need as you develop the language. Only the test of time will tell you whether it really is exactly what you need. And needs always change. To my mind an advantage of using an existing language is that, although it is not exactly what you need, at least it has been proven for a lot of other projects in the broader domain of IT. So although it might not be perfect, at least you are less likely to discover that it is really bad for anything when you get further into game development.
It is also worth considering what happened to Naughty Dog. They developed their own Lisp variant and used it for a number of projects. But when their Lisp guru moved on to other things they were no longer able to continue using that language. Like me, they are currently looking into alternatives to C++. Depending so heavily on one programmer who really understands the language is not a good idea.
An obvious disadvantage of developing a language in house is that it is a lot of effort. Not only do you need to implement a new compiler or interpreter, you potentially need to implement a runtime environment, a debugger and IDE support. You also need to document the language. By using an existing language, you are reusing not only the language itself but all the supporting tools and documentation.
You will also find that you are debugging your new language throughout development. If there is a bug, is it a bug in the game code or the language?
If you have succumbed to the temptation to write a new language, you may be tempted to add new features to the language throughout development of the game. That is both good and bad.
All of the other programmers on the team will need to learn the new language. If you use an existing language then some of them might know it already and you might be able to hire programmers who already know it.
There will be no third party libraries available for your new language. If you want an XML parser then you have to write a new one. If you used Python or Java you would have 100s (slight exaggeration) of options.
An advantage of rolling your own is that there are absolutely no licensing issues. On the other hand, there are plenty of existing languages with unrestrictive licenses.
There is another option. I am still on the fence on this one. I have written a prototype that translates Java byte code (.class files) into C++ source code. Now although I have written a compiler, I haven't invented a new language. I can reuse existing IDEs, debuggers, 3rd party libraries and language documentation.
Lets way up the pros and cons. The obvious advantage is that you get a language that is exactly what you need, or at least exactly what you think you need as you develop the language. Only the test of time will tell you whether it really is exactly what you need. And needs always change. To my mind an advantage of using an existing language is that, although it is not exactly what you need, at least it has been proven for a lot of other projects in the broader domain of IT. So although it might not be perfect, at least you are less likely to discover that it is really bad for anything when you get further into game development.
It is also worth considering what happened to Naughty Dog. They developed their own Lisp variant and used it for a number of projects. But when their Lisp guru moved on to other things they were no longer able to continue using that language. Like me, they are currently looking into alternatives to C++. Depending so heavily on one programmer who really understands the language is not a good idea.
An obvious disadvantage of developing a language in house is that it is a lot of effort. Not only do you need to implement a new compiler or interpreter, you potentially need to implement a runtime environment, a debugger and IDE support. You also need to document the language. By using an existing language, you are reusing not only the language itself but all the supporting tools and documentation.
You will also find that you are debugging your new language throughout development. If there is a bug, is it a bug in the game code or the language?
If you have succumbed to the temptation to write a new language, you may be tempted to add new features to the language throughout development of the game. That is both good and bad.
All of the other programmers on the team will need to learn the new language. If you use an existing language then some of them might know it already and you might be able to hire programmers who already know it.
There will be no third party libraries available for your new language. If you want an XML parser then you have to write a new one. If you used Python or Java you would have 100s (slight exaggeration) of options.
An advantage of rolling your own is that there are absolutely no licensing issues. On the other hand, there are plenty of existing languages with unrestrictive licenses.
There is another option. I am still on the fence on this one. I have written a prototype that translates Java byte code (.class files) into C++ source code. Now although I have written a compiler, I haven't invented a new language. I can reuse existing IDEs, debuggers, 3rd party libraries and language documentation.
# posted by Alastair Patrick : 6:22 PM
Comments:
<< Home
I agree! I'm a non-programmer but in all the projects I've ever worked on relying on the ideas/knowledge/structure/language of one programmer is a very bad idea. Even if it's in a commonly used language, everyone has their own tricks, their own way of structuring and their own knowledge of the problems with the program, code etc. When they leave that knowledge goes too and it either takes ages to regain or is lost forever...
My team has just made this mistake - why do managers never listen...?!
Charlie (Al's friend)- having an early morning rant!
My team has just made this mistake - why do managers never listen...?!
Charlie (Al's friend)- having an early morning rant!
# posted by 
: 12:09 AM
: 12:09 AM
I swing the other way on this one lately. I've worked on very successful games that had home-rolled scripting languages, but the tack that we took was to really focus on the domain problems we really wanted to solve. We weren't trying to develop new general-purpose languages (as in Epic's or Naughty Dog's cases), we specifically restricted ourselves to a very focused set of goals that we had.
What we focused on was gameplay setups. We could trigger explosions, we could send enemies to pull switches, and so forth. We didn't have any math support at all if I recall correctly, because we didn't need it. We had lists and iterators for those lists, but those lists were only ever used for (and restricted to being) lists of objects appearing in the game world.
Scripting engines like this allow you to do very high-level things without worrying about details. In our case, you are actually prevented from knowing about the details, because only very high-level operations were supported. It allowed us to iterate very quickly on our various setups, and combined with an in-game console within which we could run script commands, we could do a lot in a very short time.
Contrast this to using C or C++ for gameplay and the benefits are obvious. Versus integrating Python or Lua or anything else, the benefits are less obvious, but I still think that they weigh toward a tightly-focused domain-specific language due simply to the fact that they force the users to think about their problems at an appropriately high level.
So, we are left with two unrelated arguments. First, "is it smart to roll your own general-purpose language?" (I agree with Al on this point) and second, "is it smart to roll your own domain-specific language?" (Where clearly, I feel that this can be a wise choice).
-tom!
What we focused on was gameplay setups. We could trigger explosions, we could send enemies to pull switches, and so forth. We didn't have any math support at all if I recall correctly, because we didn't need it. We had lists and iterators for those lists, but those lists were only ever used for (and restricted to being) lists of objects appearing in the game world.
Scripting engines like this allow you to do very high-level things without worrying about details. In our case, you are actually prevented from knowing about the details, because only very high-level operations were supported. It allowed us to iterate very quickly on our various setups, and combined with an in-game console within which we could run script commands, we could do a lot in a very short time.
Contrast this to using C or C++ for gameplay and the benefits are obvious. Versus integrating Python or Lua or anything else, the benefits are less obvious, but I still think that they weigh toward a tightly-focused domain-specific language due simply to the fact that they force the users to think about their problems at an appropriately high level.
So, we are left with two unrelated arguments. First, "is it smart to roll your own general-purpose language?" (I agree with Al on this point) and second, "is it smart to roll your own domain-specific language?" (Where clearly, I feel that this can be a wise choice).
-tom!
# posted by : 6:31 PM
: 6:31 PM
Hi Tom.
You're right, I am trying to answer to different questions at the same time here. First off, I think "general-purpose" is actually a misleading way of describing a language. Every language will be better suited to some problem domains than others. Maybe "wide-application" and "narrow-application" work better?
In a sense, a narrow-application language is a kind of abstraction. Abstractions are best when they restrict the programmer in such a way that they only need to think about things that are essential to the problem they are trying to solve. I can see that a narrow-application language could give you a really tight abstraction. Some thoughts...
I think many of these narrow-application languages would benefit from being graphical languages rather than lexical languages. State machines, flow charts and GUI layouts spring to mind.
XML is often used a narrow-application language but the syntax is often clumsy and verbose.
Some languages, such as lisp and smalltalk, are very good at being platforms on which other "sub-languages" can be built. That might be better approach in some cases.
Software abstractions inevitably need to be refactored. When they are difficult to refactor, programmers tend work-around them in ways that, over time, reduce the quality of the code and slow down development. A proprietary language, even a narrow-application one, will likely be "owned" by a small number of programmers, possibly one. They will be an abstraction-refactoring bottleneck. Furthermore, a refactor is best carried out by the programmer who detects the code smell, not delegated to a language expert.
My feeling on this is, as an industry, we should invest more time in narrow-application _graphical_ languages. I can see that there might be some applications for lexical narrow-purpose languages. However, it is important to consider the abstraction facilities available in existing wide-application languages, particularly those with higher-level features suited to such things. Lastly, setting out to write narrow-purpose languages is too general a goal. What specific domains would they occupy?
Al
Post a Comment
You're right, I am trying to answer to different questions at the same time here. First off, I think "general-purpose" is actually a misleading way of describing a language. Every language will be better suited to some problem domains than others. Maybe "wide-application" and "narrow-application" work better?
In a sense, a narrow-application language is a kind of abstraction. Abstractions are best when they restrict the programmer in such a way that they only need to think about things that are essential to the problem they are trying to solve. I can see that a narrow-application language could give you a really tight abstraction. Some thoughts...
I think many of these narrow-application languages would benefit from being graphical languages rather than lexical languages. State machines, flow charts and GUI layouts spring to mind.
XML is often used a narrow-application language but the syntax is often clumsy and verbose.
Some languages, such as lisp and smalltalk, are very good at being platforms on which other "sub-languages" can be built. That might be better approach in some cases.
Software abstractions inevitably need to be refactored. When they are difficult to refactor, programmers tend work-around them in ways that, over time, reduce the quality of the code and slow down development. A proprietary language, even a narrow-application one, will likely be "owned" by a small number of programmers, possibly one. They will be an abstraction-refactoring bottleneck. Furthermore, a refactor is best carried out by the programmer who detects the code smell, not delegated to a language expert.
My feeling on this is, as an industry, we should invest more time in narrow-application _graphical_ languages. I can see that there might be some applications for lexical narrow-purpose languages. However, it is important to consider the abstraction facilities available in existing wide-application languages, particularly those with higher-level features suited to such things. Lastly, setting out to write narrow-purpose languages is too general a goal. What specific domains would they occupy?
Al
<< Home
