ANOVA (and R)

The ANOVA Controversy

ANOVA is a statistical process for analysing the amount of variance that is contributed to a sample by different factors. It was initially derived by R. A. Fisher in 1925, for the case of balanced data (equal numbers of observations for each level of a factor).

When data is unbalanced, there are different ways to calculate the sums of squares for ANOVA. There are at least 3 approaches, commonly called Type I, II and III sums of squares (this notation seems to have been introduced into the statistics world from the SAS package but is now widespread). Which type to use has led to an ongoing controversy in the field of statistics (for an overview, see Heer [2]). However, it essentially comes down to testing different hypotheses about the data.

Type I, II and III Sums of Squares

Consider a model that includes two factors A and B; there are therefore two main effects, and an interaction, AB. The full model is represented by SS(A, B, AB).

Other models are represented similarly: SS(A, B) indicates the model with no interaction, SS(B, AB) indicates the model that does not account for effects from factor A, and so on.

The influence of particular factors (including interactions) can be tested by examining the differences between models. For example, to determine the presence of an interaction effect, an F-test of the models SS(A, B, AB) and the no-interaction model SS(A, B) would be carried out.

It is convenient to define incremental sums of squares to represent these differences. Following the notation of Fox [1], let

SS(AB | A, B) = SS(A, B, AB) - SS(A, B)
SS(A | B, AB) = SS(A, B, AB) - SS(B, AB)
SS(B | A, AB) = SS(A, B, AB) - SS(A, AB)
SS(A | B)     = SS(A, B) - SS(B)
SS(B | A)     = SS(A, B) - SS(A)
The notation shows the incremental differences in sums of squares, for example SS(AB | A, B) represents "the sum of squares for interaction after the main effects", and SS(A | B) is "the sum of squares for the A main effect after the B main effect and ignoring interactions" [1].

The different types of sums of squares then arise depending on the stage of model reduction at which they are carried out. In particular:

When data is balanced, the factors are orthogonal, and types I, II and III all give the same results.


The anova and aov Functions in R

The anova and aov functions in R implement a sequential sum of squares (type I). As indicated above, for unbalanced data, this rarely tests a hypothesis of interest, since essentially the effect of one factor is calculated based on the varying levels of the other factor. In a practical sense, this means that the results are interpretable only in relation to the particular levels of observations that occur in the (unbalanced) data set. Fortunately, based on the above discussion, it should be clear that it is relatively straightforward to obtain type II SS in R.

Type II SS in R

Since type II SS tests each main effect after the other main effects, and assumes no interactions, the correct SS can be obtained using anova() and varying the order of the factors.

For example, consider a data frame (search) for which the response variable is the time that it takes users to find a relevant answer with an information retreival system (time). The user is assigned to one of two experimental search systems on which they run the test (sys). They are also assigned a number of different search queries (topic).

To obtain type I SS:

anova(lm(time ~ sys * topic, data=search))

If the data is unbalanced, you will obtain slightly different results if you instead use:

anova(lm(time ~  topic * sys, data=search))

The type II SS is obtained by using the second line of output from each of the above commands (since in type I SS, the second component will be the second factor, after the first factor). That is, you obtain the type II SS results for topic from the first command, and the results for sys from the second.

Type III SS in R

This is slightly more involved than the type II results.

First, it is necessary to set the contrasts option in R. Because the multi-way ANOVA model is over-parameterised, it is necessary to choose a contrasts setting that sums to zero, otherwise the ANOVA analysis will give incorrect results with respect to the expected hypothesis. (The default contrasts type does not satisfy this requirement.)

options(contrasts = c("contr.sum","contr.poly"))

Next, store the model:

model <- lm(time ~  topic * sys, data=search)

Finally, call the drop1 function on each model component:

drop1(model, .~., test="F")

The results give the type III SS, including the p-values from an F-test.

Type II and III SS Using the car Package

A somewhat easier way to obtain type II and III SS is through the car package. This defines a new function, Anova() (note the capital "A"), which can calculate type II and III SS directly.

Type II, using the same data set defined above:

Anova(lm(time ~  topic * sys, data=search), type=2)

Type III:

Anova(lm(time ~  topic * sys, data=search, contrasts=list(topic=contr.sum, sys=contr.sum)), type=3)

Due to the way in which the SS are calculated when incorporating the interaction effect, for type III you must specify the contrasts option to obtain sensible results (an explanation is given here).


[1] John Fox. "Applied Regression Analysis and Generlized Linear Models", 2nd ed., Sage, 2008.

[2] David G. Herr. "On the History of ANOVA in Unbalanced, Factorial Designs: The First 30 Years", The American Statistician, Vol. 40, No. 4, pp. 265-270, 1986.

[3] Oyvind Langsrud. "ANOVA for unbalanced data: Use Type II instead of Type III sums of squares", Statistics and Computing, Volume 13, Number 2, pp. 163-167, 2003.

[4] Ista Zahn. "Working with unbalanced cell sizes in multiple regression with categorical predictors", 2009.

Home Teaching Research Resources and Other Interests

Page last updated on 05 February 2016
by Falk Scholer