Over time, Monte Carlo simulation (also known as random walk modeling) has become a very useful predictive and explanatory tool, helping people make better decisions. Monte Carlo simulation is used for modeling in several domains, including economic pricing, environmental science, sports betting, military planning, game strategy, evolutionary theory, and several other fields. By developing algorithms that accurately quantify and closely represent real life situations, Monte Carlo simulation has been extremely successful in its goal of helping people make better decisions.
Monte Carlo simulation creates a “virtual reality”, artificially modeling an interaction by attempting to quantify and mimic various effects and running repeated experiments in this virtual environment. It looks ahead at future possible scenarios based on this virtual reality, observing and quantifying the unforeseen ramifications of each decision being considered.
The measure of success in Monte Carlo simulation lies in its predictive or explanatory power. If the results of simulation don’t align with what happens in real life, the simulation fails to be useful. This fact alone can be powerful: if a simulation does not produce results that closely models reality, an unquantified variable must exist. In this way, Monte Carlo simulation serves not just as a model, but also as a counterfactual test for any hypothesis.
Monte Carlo simulation is prominent in many fields. In some cases, it provides complete solutions and has revolutionized the way we look at certain situations. It has solved games such as Connect Four and checkers while creating the best chess player in the world, taken over Las Vegas, and is used by many of the largest investment, oil, and environmental companies in the world.
Yet even in its success, Monte Carlo simulation still has flaws. While its mathematics are sound, theorists are cavalier about its usage. Monte Carlo simulation is often unable to predict or assess seemingly obvious dynamics, since theorists misequivocate success within a system to equal success within each subset. Monte Carlo simulation theorists overemphasize global variables and overlook localized variables far more salient, because of a sense of laziness and an inability to identify causality. And many theorists mistreat and misevaluate new or previously overlooked variables, assessing their worth to preexisting models and eliminating the possibility of paradigm shift.
Let me be clear: the problems mentioned here are not problems with the process of Monte Carlo simulation: they are problems with application. When a specific methodology is misapplied so frequently, it outlives its usefulness. This phenomenon has happened consistently throughout the history of science: major paradigms shift based on application rather than theory. From physics to atomic theory to astronomy, paradigms have changed based on their usefulness in answering the questions theorists posed.
Why has the application of Monte Carlo simulation faltered? The main reason that Monte Carlo simulation has been misapplied is because theorists violate the axioms essential to the process. In an attempt to achieve progress, scientists have pushed the limitations of Monte Carlo reasoning beyond where it was meant to go, resulting in flawed data and misleading conclusions.
To further my point, here are some of the axioms and postulates that have frequently been violated by Monte Carlo theorists:
- The underlying processes that occur within the simulation must closely model the processes that occur in reality. Monte Carlo simulation only exists as a model to solve problems, and without a semblance or anchor in reality, Monte Carlo simulation is a useless economic abstraction. Monte Carlo simulation must be subject to the same influences and assessments as its real-world application.
- The utility function within Monte Carlo simulation must closely approximate the utility function of real-life players. Without an accurate utility function, we lose the importability of our simulation as well as the predictive ability to consider the actions of other actors. Players may act seemingly irrationally in addition to being unaware of the ramifications of each action. Even though utility functions are not strictly rational, they must be somewhat accurate to preserve the importability of the model.
- Each trial must be contextually homogenous. It must be assumed that both the initial state and the implications of future states are identical to all players. When this axiom is violated, specific types will react in divergent ways, thus self-perpetuating different types and devolving Monte Carlo simulation into a seemingly chaotic process. When factors such as history, reflection, or utility deviations enter the model, it dramatically alters the result of the simulation.
- The rules and utility function of each actor must be (within reason) public knowledge. Without this, the actions of other actors within the game is inherently unpredictable, and as such, there is no viable strategy with which players can make good decisions. While risk is an acceptable variable within any game, uncertainty should be avoided at all costs, as uncertainty brings a chaotic element into various situations and negates the presumption of rationality.
- Every game is relative to the game space, action space, and number of players. This ordered triple is recognized as a constant within game theory, and cannot be changed without re-creating the entire simulation.
In contrast, there are many faulty assumptions that are made by Monte Carlo theorists:
- Preference is transitive. Many theorists believe that if A is preferable to B and B is preferable to C, that fact implies that C is preferred to A. This is not true unless preference takes the characteristic of dominance. In nature, it is frequently the case that behaviors are merely preferred instead of dominant.
- Introducing new variables into systems can be accommodated for by simply introducing them into already existing models. This is a simple misunderstanding of basic statistics. When adding variables to form a correlation, correlating variable a might be the strongest single variable, but variables b, c, and d might provide a stronger correlation than any combinatrix including a.
These failed axioms result in the following problems: the non-transitive nature of preference causes a misunderstanding of Monte Carlo simulation in situations with multiple equilibria; the ambiguities of utility functions obfuscate the salience of variables, and the desire to fit new variables into models causes theorists to alter their models haphazardly without creating a new model. To anlyze this further, let’s take a look at the following problems:
Problem 1: Preference is not Dominance
One common assumption that many people who use Monte Carlo simulation make is that they assume that when an organism or action performs best in a large system, that fact implies that it will also triumph in head-to-head matchups or other matchup subsets. This is not the case, especially in situations with a lot of uncertainty or with small populations.
Let’s take a look at an example:
|
A |
B |
C |
D |
| A |
0 |
6 |
5 |
-2 |
| B |
-6 |
0 |
-1 |
5 |
| C |
-5 |
1 |
0 |
3 |
| D |
2 |
-5 |
-3 |
0 |
Let’s take a look at the game above. In this case, strategy A is certainly the most effective, while strategy D is certainly the least effective. If there was an equal subset of the population who used all four strategies, you would expect A to be a dominant strategy. In a system where there is a low population and a high death rate affecting all strategies equally, it is conceivable that strategies B, C, and D could die out.
However, this game has no pure strategy Nash equilibrium, as D does beat A despite losing to both B and C. Some theorists erroneously conclude that A is dominant, when this is obviously not the case. As such, if the equilibrium were unstable and there were a high epsilon factor, we could imagine D dying out first, then B and C dying out, leaving A as the only viable strategy, and then D migrating back into the system somehow and dominating the population (and, subsequently, B and C again becoming viable, and so on and so forth).
While this might seem like a rare or fabricated example, it actually happens often when there are often several strategies at work. Oil producers need to refine, manufacture, drill, hire labor, and set prices. Bears need to hunt, mate, seek shelter, and grow fur. Deviant strategies, even if they are successful, can easily get shut out due to anything from a shortage to an unusually cold winter. Within these systems, dominant companies and species emerge. Nevertheless, seemingly defunct strategies can still be viable.
Regardless, this does not mean that the strategy was inferior to the alpha member of their group: it merely implicates that the strategy was inferior within the entire system. The transitive property does not work in game theory: A > B and B > C does not imply that A > C. Condorcet cycles are frequent, especially in new systems that are far from optimized.
This type of dynamic is frequent in oligopolies. In many oligopoly systems, members of the oligopoly are kept in place merely to preserve the system and prevent a dominant member from emerging (and letting the system devolve into a monopoly). Keeping D around to keep A in check is an important to maintain the survivability of B and C and preserving the profitable oligopoly’s existence.
Oligopolies often sell bundled items in terms of their attractiveness: gas stations sell price of gas and location, while airlines sell price, ambience, and flight length. However, the inconsistencies and subjectivity involved with decision theory combined with the ability of side-by-side comparison can often cause changes in consumer behavior, even if indirectly. While A may be preferred to B on a head to head comparison, the mere existence of product C may cause B to now be favored to A.
The most common example of this is in airlines. Consumers in the airline industry have very fickle preferences based on the market, especially based on the type of service they receive. For example, many customers have a simple heuristic such as “Find the cheapest flight”, but that heuristic will change if the cheapest flight is no longer desirable for some reason.
In reality, consumer behavior is not classically rational (in the economic sense). When making decisions, preference is not a linear operator: it does not behave in easily predictable terms. Thus, the variables that model such decisions typically do not work very well, and the transitive and distributive operators of logic cannot successfully be applied. Unfortunately, many theorists still try to apply the transitive and distributive property to many economic models, including Monte Carlo simulation. These incorrect applications can prove quite costly, especially in political and economic domains.
Problem 2: Globality vs. Locality, Causal vs. Statistical Variables
The second major problem with Monte Carlo simulation is its inability to differentiate between weak causation and variance, especially when a causal variable only impacts a subset of the system or impacts different members of the population differently. The salient variables of sub-populations can vary from the salient variables of the entire population. Monte Carlo simulation and supplemental statistical models are incapable of finding and accounting for these differences in variables.
These inabilities are not due to faults of the models or simulation themselves. Statistical models make assumptions about homogeneity that are not existent in reality, and can only apply themselves so much, particularly in situations where there were not enough subjects or information accounted into the model. Statistical models by necessity limit the number of variables that can be tested, and as thus use only the most powerful variables, regardless of whether they are statistical and causal.
In applications where Monte Carlo simulation would apply, doing this is not necessary. Outside of obscure domains (such as quantum physics), statistical causation can always be reduced to real world causation given a big enough sample size and a proper grouping of salient variables. Unfortunately, many people who apply statistics or Monte Carlo simulation take the lazy way out, grouping every member into a population without prioritizing sub-populations or reducing statistical causation into real-world causation so they don’t have to jeopardize their model.
Table 1:
| Table |
Adam |
Betsy |
Charlie |
Dave |
Emily |
Frank |
Gina |
Harriet |
Iris |
Juanita |
| Hot |
85 |
91 |
91 |
88 |
90 |
79 |
86 |
69 |
86 |
78 |
| Warm |
88 |
92 |
86 |
89 |
77 |
85 |
77 |
81 |
86 |
86 |
| Cold |
83 |
95 |
73 |
68 |
85 |
81 |
90 |
79 |
71 |
79 |
A set of students took three different tests of equal difficulty in rooms of 50, 70, and 90 degrees Fahrenheit to test whether room temperature influenced test score. Scores showed that the difference of scores were not statistically significant (Hot = 84.3, Warm = 84.7, Cold = 80.4, SD = 6.98)
These findings are reasonable: after all, the odds are good that these results could be derived from random chance alone. But wait: What if I told you that Charlie, Dave, and Iris were all from Brazil? And Harriet was from Norway? And everyone else was from New England? All of a sudden, it seems likely that temperature is a predictive variable for certain students.
This is because we’ve found a causal reason for a seemingly weak correlation. Using the information we’ve learned, we could devise a new study that would show certain types of students perform differently based on temperature and geography. We’ve found a correlation based not on all people, but on a subclass of people.
In many cases, statistical models are not created as true models, but merely as sums of correlations. While some of these correlations are awfully strong, without causal significance and independence they can often be refined or even eliminated if a better set of causal variables involved. For example, predicting whether or not a team will win a football game can be roughly estimated by using point margin alone, but a better model could easily be created by using other causal variables that eliminate such a variable entirely.
Stats must always be placed in context. You can say that cold rooms cause test scores to drop by around 4 points. And that fact would be true. But it’s not nearly as informative to say that as it would be to say: “Cold rooms has no effect on some students but has a significant effect on students not born in New England.”
When rules or laws are made, they are made globally, keeping EVERYONE in mind. However, tendencies for subgroups of a population will vary from the population as a whole. While aspartame or airport scanner radiation may not be dangerous for an entire population, it could be dangerous for an unknown subgroup, such as Native Americans, coal miners, or people who live in Sao Paolo.
The only way to know if a danger exists is to test each subgroup individually: something that proves “impractical” for most subgroups of people. Thus, only especially vulnerable subgroups (such as children, elderly, sick people, or pregnant mothers) are tested for most health-related issues. These categories are then tested for by using indicator variables.
This illustrates the central theme: there is a difference between global variables and local variables. The way to predict results using global variables (such as when simulating sports outcomes) can never be as precise as local predictors based on individualized indicators.
Therein lies a problem: there are an infinite number of indicator variables. Every conceivable trait can be evaluated as an indicator variable, and within a small population, some traits will have a statistical correlation even if no causal correlation exists. Add that to the numerous combinations of indicator variables that must be accounted for, and we have a prediction nightmare. When we try to also account for locality, this results in an impossible task.
These types of local, causal variables are prominent and common when combined with indicator variables. They tend to be causal and difficult to precisely quantify, and tend to have an apparent causal link that may affect members of a population differently.
Examples include:
- Effort: Some teams are predisposed to trying harder or prepare more vigorously in some games while being more content to sit back in others. This is especially true in games with exceptionally large betting lines.
- Unusual strategies: Teams will thrive or struggle against similar types of offenses or defenses, especially when they are built around a dominant player, such as a dominant center in basketball or a dominant quarterback in football.
- Environmental factors: Factors such as temperature, altitude, color, climate, or disease rate often make very good local variables. These variables are likely to affect different members of a population drastically differently, depending on their experience within each environment and sensitivity to each factor.
- Cultural variables: Variables that vary from culture to culture such as religion, nationalism, public opinions, etc. possess very individualized effects.
Problem 3: Statistical modeling: a reverence for existing models
The third problem with Monte Carlo simulation is its admiration for preexisting models. This reverence is understandable: after all, it is far easier to adjust a current model than to create a new one. Science cannot proceed without ideological stability: therefore, models are not rejected until data makes doing so a necessity.
It is rare to see Monte Carlo simulations recalibrated. Cold War simulations lived far past their welcome, and were part of Pentagon brinksmanship even into the 1990s. Global warming models remained unchanged even when its predictions (both in favor and against global warming) were rebuffed in the early 2000s. Backgammon players used outdated programs (such as Jellyfish) even when they drastically deteriorated in their results against humans over a decade. Von Neumann’s ideas of evolution were prominent long after they were disproved. They remained because no one forced them to change. They remained because there was no replacement.
Theorists despise adding or changing models because it weakens theorists’ predictive power and forces them to reinvestigate their conclusions. In most fields where Monte Carlo theory is applied, data is scarce. Other models have already failed. Data is either hard to come by, unreliable, or unexplained. Causality is often a mystery. There are no other ways to solve games, model brinksmanship, or understand environmental phenomena. Predictive power is precious. Theorists can’t give it away while remaining relevant.
Thus, many theorists go beyond normal scientific practice to account for salient variables. They create an additional function. They use post hoc analysis to explain or accommodate for the new factors. as a statistical analysis or even post hoc. Since the model cannot easily accommodate the new variable and there is a strong bias for the status quo, theorists try to avoid recreating the model, or worse yet, reevaluating the paradigm. This results in a lot of poor models, as a new model is necessary to accurately predict or explain phenomena. As new variables are proven to be more salient and causal, models must not be merely revised, but completely recalculated to quantify the effect of the new variable, often creating models of patchwork instead of uniformity.
This problem also manifests itself in another form: people apply models such as Monte Carlo information in a way that the information is unfalsifiable, and there is no way to test the results. Monte Carlo simulation is used to test mathematical and physics problems and create heuristics in game strategy, public policy, and evolutionary biology.
While it is widely accepted that this approach might produce flawed data, many argue that is better than no data at all. And perhaps there is merit, if given such a duality. But the “bad data is better than no data” approach can be quite harmful: it creates results that others take far too seriously without knowing where the information came from. It starts an ideology without any data support.
In these cases, the Monte Carlo simulation is not a data or statistical tool, but a reflection of belief disguised as sophisticated theorycraft. It becomes consensus not based on existing data, but based on the lack of existing data. In reality, Monte Carlo simulation is not a revolutionary way of attaining information, but rather a tool to sort and perceive information. When extensive, established data is presented side-by-side with experimental, untested data, many give it far too much credence. And, like always, there is much data in between these two extremes.
Monte Carlo simulation unquestionably has great potential and application. However, many theorists abuse Monte Carlo simulation and try to expand its application beyond its limitations, defying its axioms and creating inaccurate or unrefined data. Because of this, it is always important to analyze its use, and retain skepticism until one is assured that Monte Carlo simulation is applied properly.
Kenji Matsumoto 