Portfolio Segmentation

Beware of Simpson's Paradox, and Be Careful

Al R. Vilcius

Risk Management generally claims that segmentation is necessary to sharpen results obtained from the analytic tools used for various purposes. However, this needs to be done very carefully to avoid erroneous results that can easily lead to mistaken and inappropriate actions. This will be shown here, first by a couple of simple examples, then a little theory.

Consider the following (hypothetical) statistical study :

A 1980 study of the difference in smoking habits between men and women showed a higher proportion of women smokers.

The study was repeated in 1990.  It was thought that women were quitting faster, so a larger sample of women was taken to prove this point.

Here are the findings: 

On the basis of these results, the study revealed that men and women were quitting at a similar rate, with the ratio of smokers reduced for both men and women, but the ratio of people smoking overall had increased.

Conclusion: both men and women are smoking less but people are smoking more!!

The same study also asked a question about drinking habits.

These results were as follows: 

The bonus question showed that both the ratio of men and women drinking had increased, with a surprisingly sharp increase for women, but that in total, the ratio of people drinking had actually decreased.

Conclusion: both men and women are drinking proportionately more, but people are generally drinking less!

So there you have it - them's the facts!

This curious reversal is a problem with all demographic studies.  It is called Simpson's paradox.

The  problem also occurs in medical studies where conflicting "results" are often obtained for clinical tests that are not properly controlled.  But that's not all.  There are also many occurances in finance.  Here is another example to consider:

Wagner ^[1] used income tax numbers in his example to illustrate Simpson’s paradox. Here I have restated it (using his numbers) in terms of securities simply as a mater of personal taste and for extra clarity and dramatic effect.
 

This example shows a lower yield in EACH category this year, but the overall yield is higher - check it out for yourself!.

Here is the caveat to risk management portfolio analysis:

Portfolio segmentation into parallel descriptive categories can show improvement or deterioration in each segment while showing exactly the opposite overall.

As Wagner ^[1] points out: "This is not a contrived pedagogical example" - it does ocur frequently in practice.  This example can be used in many contexts: in 1991 I used it in the context of credit quality analysis for corporate loan portfolio scoring.  The relevance to electronic commerce behavioural analysis should be obvious; the danger is that Simpson's Paradox can lead observers to "see" patterns in the data that do not exist.

Now here is a little bit of theory, as promised.

Simpson's Paradox (also known as the "Stein paradox") Is Not Really a Paradox: it occurs whenever an apparently significant conclusion based upon a sample of data, is reversed in each subsample, when the sample is split according to a "lurking variable".

In order to avoid an overly abstract discussion, my explanation here uses the first table (study of smokers) above.

The answer to the apparent paradox is that the proportions of the table are weighted averages, not simple averages.

Pr(smoker)=  Pr(smoker and man)  +  Pr(smoker and woman)
                 =  Pr(smoker | man)Pr(man) + Pr(smoker | woman) Pr(woman)

which is obtained simply by substituting the definition of conditional probability (given below).  Now it is clear that the distribution (ie. ratio) of man:woman is the "lurking variable" mentioned above.

The apparent "cheat" is that the frequency ratio man:woman, i.e. Pr(man) and Pr(woman) changed in the 1990 population to that of the 1980 sample, resulting in essentially different weighted averages being calculated for the population of people.
 

In general,

Then the expression for these fractions is identical to the one for conditional probabilities above.  Notice that if  b=d (which would mean that the number of men is the same as the number of women in the example), then we just have a simple average, which is what most people seem to have in mind when looking at the sample tables in the above examples.

Proportions within descriptive subclasses of non-constant populations is a feature of conditional probability, defined as a ratio:

Pr(A | B)=Pr(A and B) / Pr(B)

When this is applied to 1980 observations on smokers, we see:

Pr(smoker | woman)=Pr(smoker and woman) / Pr(woman)

Assuming that people are either men or women, not both

Pr(man) + Pr(woman)=Pr(man or woman)=1

Pr(smoker) + Pr(non-smoker)=1

Pr(smoker)=Pr(smoker | man)Pr(man) + Pr(smoker | woman) Pr(woman)

Then Bay's theorem gives us the ratio:

Pr(woman | smoker)=Pr(smoker and woman) / Pr(smoker)
 =Pr(smoker | woman) Pr(woman) / [Pr(smoker | man)Pr(man) + Pr(smoker | woman) Pr(woman) ]

from which further contradictions using the little table above can be found.

This situation also comes up in "regression" applications.   The straightforward regression equation often is just plain wrong in some cases, leading to incorrect actions. Caution should be exercised in all finance, economics, medical, and social applications.

The bottom line:

This "paradox" of conditional probabilty deserves caution in practice.

References:

• [1] C. H. Wagner, “Simpson’s Paradox in Real Life”, The American Statistician 36 (1982) pp.46-8

Recommended Further Reading

These titles can be purchased on-line from Chapters.ca

1. the left column below contains the hot links
2. use browser back button to return