Commentary: Week 8 Mike LeBeau

SSP 205

Spring 2005

An issue which was brought up in class a few times but never very deeply explored is how much information the rules or the model of a simulation can encode before we must consider the results both unsurprising and without value. At one extreme, we can imagine a simulation in which the rules are so detailed and specific that it seems as though the simulator focused on a desired result and found rules which could produce that result – this is the easy case to throw out as insignificant. At the other end of the spectrum, we can imagine a simulation in which the rules are so simple that everyone who sees it is astounded at the results. But how is it that we decide which rules are simple enough? As the rules of a simulation get more and more complex, at what point (if any) do we say that any additional complication to the rules would make the results of the simulation no longer compelling?

Many seem to share the intuition that the rules of a simulation should generally not be complex. Axelrod, for one, claims that “it is important to keep the model as simple as possible. When a surprising result occurs, it is very helpful to be confident that one can understand everything that contributed to the model,” and that “the complexity of agent-based modeling should be in the simulated results, not in the assumptions of the model.” To be fair, it is also possible that a distinction should be made between ‘suspicious complexity’, the type of complexity that seems to contain information that will lead to a specific result, and other types of complexity which may simply be attempts to accurately describe whatever phenomenon is being simulated. But so far, the simulation community appears to rely simply on their intuition to decide whether complexity is acceptable or unacceptable.

I find it surprising that while Axelrod, in his article on advancing and standardizing the art of simulation, claims the need for more shared standard conceptual knowledge of simulations in the academic community, he makes no mention of the need for standards with regard to limiting the complexity of a simulation model. Perhaps the problem of creating these standards lies partially in Axelrod’s point that “simulation results often address an interdisciplinary audience”. Simulations can be produced for many different disciplines and can take many different forms, and I admit it is difficult to imagine any sort of general standard of complexity which could be applied to the types of rules that would be used by many different disciplines. Some might argue that this barrier alone is enough to conclude that such a standard could not be developed. Somehow, though, people manage to decide for themselves whether the results of a simulation are compelling and the rules simple enough, using their own intuitions, which seems to indicate that we use some internal metric to make these choices.

The fact that these intuitions vary across people (i.e., some may find the rules of a simulation acceptably simple while others may not) may also be reason to ignore finding a standard, since no standard would likely be acceptable to everyone. This became very evident to me at one point in class. Before this week’s class, I felt that everyone would probably agree that Thomas C. Schelling’s segregation simulation was certainly simple enough to merit value. But after discussing it in more detail with the others in the class, I became fairly convinced that in fact the simulation might be encoding too much in the model. What seemed at first like a fairly weak claim (that is, that ‘dots’ wanted at least two of their six neighbors to be the same color) was at second glance much harder to accept. Granted, simulations always make simplifications, but this model makes some pretty strong claims about people’s race preferences, and goes as far as to say that people will continue to move over and over until they are satisfied with their surroundings. This is an example of intuition changing even within one person about what is acceptable in a simulation, and goes to show that it is very subjective.

Perhaps it is impossible to produce a generic standard of limits on simulation work, given its interdisciplinary attention and its reliance on intuition and notions like surprise. If so, this is unfortunate, because this lack of standardization is likely to continue to deter those who strongly believe that such standards are necessary, as in traditional science. The course and future success of simulation science may depend on finding new ways to quantify models and determine how much ‘surprise’ really is involved in their results.