A Crisis In Science?
Note: This is based off of a discussion I recently lead titled “Science vs. Bayes.” Portions of this were inspired by Overcoming Bias.
Some believe that science is facing a crisis. I don’t mean those who argue that we are approaching “the end of science.” I mean that some physicists are disturbed that those damn kids believe in kooky, untestable theories like String Theory, which isn’t science. Or that some people believe in the Many Worlds interpretation of quantum mechanics rather than the Copenhagen interpretation. To these scientists, the acid test is falsification: is there some way of making a prediction with the theory that can be demonstrated to be wrong? If it cannot be falsified, it is not a scientific theory.
Others argue in response that while falsification is definitely important to science, it is not the end all, be all of it. Other factors must be taken into account when considering what theory to adopt to explain a given phenomena. Sometimes these people are called Bayesians, named for their use of Bayesian Inference, which depends on using Bayes theorem to calculate the probability of a theory being true given a set of other probabilities.
I’d like to argue here that the supposed differences between Bayesians and Scientists are actually non-existent, and that the Bastian’s are simply making explicit certain actions scientists take in the course of doing science.
So what exactly is science? Science is a process and a constellation of interrelated facts justified by means of that process. The process is referred to as the scientific method, and can basically be summarized thus:
- Observe some phenomena in the world
- Form a theory that explains why the phenomena occurs
- Use the theory to make a prediction
- Test the validity of the prediction
- If the prediction turns out to be correct, call the press, hold a huge celebration, and hand out Nobel prizes
- If the prediction is incorrect, publicly admit mistake (or spend the rest of your life tweaking your experiment to try and make it work)
There are several important features to this process that don’t get sufficiently appreciated. The first is that any theory is either disproven, or not-yet-disproven. This process will never tell you that your theory is correct. It only says when it is incorrect. There is a serious imbalance between evidence for and evidence against a theory, where a million correct predictions are out weighed by a single in correct prediction (your theory predicts X, but Y happened. Therefore, your theory is wrong).
A second feature I want to draw your attention to is that it is conservative. A new theory has to overcome the current theory with a new prediction that the existing theory does not make, or calls differently. If you come up with a new theory that has the same explanatory power as the existing theory, who cares? We have no reason to switch.
Which brings us to the third feature, which is that it does not offer a means of choosing between competing theories. If you have two theories that explain a phenomena, and neither has made a false prediction as of yet, you have no reason to prefer one theory over the other. This is not necessarily a bad thing. Eventually, either the theories will disagree in some prediction, in which case we keep the theory that got it right, or it will be shown that the two theories are actually isomorphic to each other, meaning they will never disagree.
But very often, scientists do choose among theories, without waiting for one theory to crush the other with a superior experimental prediction. They appeal to the Law of Parsimony: entia non sunt multiplicanda praeter necessitate, more commonly known as Occam’s Razor: entities must not be multiplied beyond necessity. When faced with two competing theories that explain a phenomena, choose the simpler one.
Oceans of ink have been spilled arguing about what counts as “simpler.” For example, some have argued that the Copernican theory was not, in fact, simpler than the “circles inside of circles” prevailing theory at the time.
Some scientists interpret this as “elegance.” They hold that the laws of nature should be simple and beautiful. Perhaps the most famous example of this is a quip from Einstein. When asked what he would say had his theory of relativity been proved wrong by an experimental prediction, he replied “Then I would feel sorry for the Good Lord. The theory is correct.” Apparently Einstein was so enamored of the beauty and elegance of relativity, he could not conceive of the universe not functioning that way.
Recently, we have begun to develop systems that lets us measure the complexity of theories. My personal favorite, not surprisingly considering my back ground in computer science, is the minimum program length. To measure the complexity of a theory, calculate the length of the shortest program that implements the model. When choosing between two theories, choose the theory which can be implemented with fewer bits of information. I don’t have the space here to unpack it in detail, but anyone interested can look into Kolmogorov complexity theory. There are other measures, such as minimum message length, but they have been proven to be equivalent to each other.
With these theories in hand, we can begun to use Occam’s Razor as an actual tool in the scientist tool box, rather than an intuitive heuristic.
What is particularly interesting here is that Occam’s Razor is firmly a Bayesian tool. Bayesian inference does a great service for you every day; it is what is used in spam filtering. Some hold it up as a competitor to science. Others claim that Bayesian inference is what scientists actually do, and the scientific method as laid out above is incomplete.
To unpack how Bayesian Inference works, I’ll show how Bayes’ theorem is used to spam filter.
The theorem is P(A|B) = (P(B|A) * P(A)) / P(B).
P(A) is the probability that a message is spam. This is found by counting how many spam message have been received, and dividing it by the total number of messages.
P(B) is the probability that a message contains the word “viagra“. This is found by counting how many messages contain the word and dividing it by the total number of messages.
P(B|A) is the probability that a message containing “viagra” is spam. This is found by counting the spam messages that contain “viagra” and dividing it by the total number of message that contain it.
P(A|B), the value we wish to find, is the probability that a message is spam given that the message contains the word “viagra”
So, in English, what this theorem states is that the chance that a message that contains the word “viagra” is spam, is equal to the chance that a message contains the word “viagra” given that its spam, times the chance that a message is spam, that product then being divided by the probability of a message containing the word “viagra.”
If we wish to use this in science, we would read the theorem as “The chance that my theory is right (A), given that it had a correct prediction (B), is equal to the chance that the predicted event would happen if my theory were correct [P(B|A)] times the chance that my theory is correct [P(A)], that product then divided by the chance of that prediction being correct [P(B)].
With a little math, its easy to show that very unlikely events B give very high probabilities that a theory is correct. Conversely, very likely events give very low probabilities that a theory is correct. In the spam example, if the word “viagra” almost never appears outside of a spam message, then P(B|A) will be almost 100%, which essentially guarantees that the result of the formula will be close to 100%. A more common word, for example “the”, would appear in basically every message, and thus would not be strongly correlated to a spam message. Hence P(B|A) would be very low, ensuring that the result of the formula would be a very low probability that the message is spam.
Getting back to the scientific method, we can reformulate it with the Bayesian aspects made explicit:
- Observe some phenomena in the world
- Form a theory that explains why the phenomena occurs
- Calculate the probability that the theory is correct
- Use the theory to make a prediction
- Test the validity of the prediction
- Calculate new probability for theory based on result of prediction
- If the probability is above a certain threshold (i.e. more likely than current theory), call the press, hold a huge celebration and hand out Nobel prizes
- If the probability is below a certain threshold, publicly admit mistake
- Repeat
Notice that in this formulation, we make explicit the criteria that the new theory has to have a higher probability of being true than the current competing theory. Also note that, rather than a simple falsified / not-yet-falsified distinction, theories have a real number value indicating probability of being true. Furthermore, we can measure the effect each individual piece of evidence has on the probabilities. This idea was already present in the previous formulation of the scientific method, where theories that made risky predictions count for more than ones that only make safe predictions, but just like the notion of “simplicity,” riskiness has never been explicitly unpacked.
It also helps to show why we should choose a simpler theory over a more complicated theory: simpler theories by definition have a higher probability of being correct, as the number of ways they could be wrong is lower. This is a result of the conjunction rule in probability: P(X ^ Y) <= P(X). The probability of X and Y must be less than or equal to the probability of X regardless of Y happening or not. So a theory that strings together allot of individual probabilities, the probability of any particular fact in the theory being correct goes down.
To look at it in slightly different way, note that P(X) = P(X ^ Y) + P(X ^ ~Y). An example of this would be the claim that evolution happens as biologists claim, but God is the one who causes mutations. If P(E) is the probability that evolutionary theory is correct, and P(G) is the probability that god exists, then what we have here is the difference between claiming E and claiming E ^ G. Since P(E) = P(E ^ G) + P(E ^ ~G), P(E) is more likely to be true. In essence, the probability that the theory is correct is spread out over all the facts that must be correct in order for the theory to be correct. If your theory includes lots of extraneous facts, they drag down the probability it is correct.
The question now is, does this reformulated version of the scientific method ever give results different from those obtained by using the older formulation? Some have argued yes, and in particular pointed to the Quantum Mechanics. Some reject the Many Worlds interpretation as being too complicated, as it claims there are an infinite number of parallel universes. Copenhagen is simpler because it needs only one universe.
To boil the disagreement down to the essentials, we have this:
Under the old formulation, we should say that the Copenhagen interpretation has passed all experimental tests so far. Many worlds doesn’t make any new testable predictions, you can do physics without supposing there are those other worlds, and that’s what you should do. When Many Worlds makes a new prediction that could falsify it with regards to Copenhagen, we can revisit the issue.
Under the new formulation, we should say that the equations we have that cover all the evidence we know don’t have an exception for large sized masses. Furthermore, to adopt the Copenhagen interpretation is to accept the existence of a natural law that has properties that no other natural law has, to wit: it is non-linear, non-unitary, non-differentiable, non-CPT-symmetric, and is an accusal, faster than light, informally specified phenomena. Therefore it should have a higher threshold of evidence than Many Worlds.
Don’t worry too much about those properties. The take home point for the Bayesian is that the Copenhagen interpretation has many ad-hoc assumptions that drastically increase the complexity of the theory. We should use Many Worlds, because it is simpler, and agrees better with how we understand other natural laws to work. The take home point of the scientist is that science is by definition conservative. We stick with the current theory, warts and all, until we have a damn good reason to switch, and the only thing that counts as a damn good reason is an experimental result that elevates one theory over another.
I suspect that as the frontiers of physics and cosmology expand, we will encounter other instances where the two methods point in different directions. Eventually, we will have to sort out which way we are going to go. But I don’t think of this as a crisis in science. I think of it as the scientific method applied to science itself. Its been 400 years since Bacon first described the scientific method, and almost 50 years since Popper wrote Conjectures and Refutations, so it should not be surprising that we would encounter situations that make us reconsider how we do science. After all, the scientific method is itself a theory, and subject to its own conditions. If it leads to incorrect results, it needs to be revised.