Comparing Hypotheses

It is often desirable to use empirical data to compare two or more hypotheses, usually to identify which hypothesis should be preferred.

This is relatively straightforward. We make a world in which the hypotheses we wish to compare all exist. Then we ask, in that world, which hypothesis is the more likely to have produced the data we have.

An example world

Imagine that we have two hypotheses:
$r_p=0$
$r_p=0.3$
We build a world that has populations corresponding to each hypothesis. Here are the sampling distributions for the two hypotheses and (at the back) for the whole world.

Each of these sampling distributions shows the distribution of sample values for $r_s$ that would be found by sampling repeatedly from that population.

Each of these sampling distributions shows the distribution of sample values for $r_s$ that would be found by sampling repeatedly from that population.

The sample

Now suppose that we have a sample:
$r_s=0.25$
We can easily ask how often such a sample might arise from each possible population. The next figure shows the answer. The red lines correspond to that sample effect size. Not surprisingly, the population with the higher probability density is $r_p=0.3$ .

The red lines show the probability density for a sample size of 0.25 for the two populations.

In the world described here, we really want to compare directly the two red lines – and say that the taller one belongs to the population that is more likely to have produced our sample. There is nothing to stop us doing that, except that we have to call the vertical axis likelihood now. In effect, we are using the red lines to create a likelihood function that runs from front to back. In this world the highest likelihood is that the sample $r_s=0.25$ came from the population $r_p=0.3$ .

The issue

This all seems straightforward – and works as we might expect. We have made a very important assumption, however: that the two hypotheses are themselves equally likely. This assumption results in the two sampling distributions in the figures having the same areas.

To be clear: we have to make some sort of assumption here. We cannot ask sensible questions of the world unless we have fully specified it, including the relative frequency of each population effect size.

Suppose that we had reason to believe that 80% of all population effect sizes were $r_p=0$ . Then the diagram looks rather different:

Now we can see that the sample $r_s=0.25$ is equally likely to have come from either population effect size.

Now we can see that the sample $r_s=0.25$ is equally likely to have come from either population effect size.

It is clear that which possible population effect size with the highest likelihood as the source for our sample depends on how common the population effect sizes are.

An Axiom

We have reached the point where we need to state something that is unavoidable. When we wish to use a sample to compare two or more hypotheses, we need to state what we believe to be the relative likelihood of the hypotheses (before the sample is considered).

Any attempt to compare two or more hypotheses requires an assumption about their prior likelihood.

An easy way to do this is to build a world that contains all the hypotheses we are considering, building that prior assumption into the world.