In the context of ANOVA-like tests, it is common to report ANOVA-like effect sizes. Unlike standardized parameters, these effect sizes represent the amount of variance explained by each of the model’s terms, where each term can be represented by 1 *or more* parameters.

For example, in the following case, the parameters for the `treatment`

term represent specific contrasts between the factor’s levels (treatment groups) - the difference between each level and the reference level (`obk.long == 'control'`

).

```
data(obk.long, package = "afex")
# modify the data slightly for the demonstration:
<- obk.long[1:240 %% 3 == 0, ]
obk.long $id <- seq_len(nrow(obk.long))
obk.long
<- lm(value ~ treatment, data = obk.long)
m
::model_parameters(m) parameters
```

```
> Parameter | Coefficient | SE | 95% CI | t(77) | p
> ------------------------------------------------------------------
> (Intercept) | 4.28 | 0.36 | [3.56, 5.00] | 11.85 | < .001
> treatment [A] | 1.97 | 0.54 | [0.89, 3.05] | 3.64 | < .001
> treatment [B] | 2.09 | 0.47 | [1.15, 3.03] | 4.42 | < .001
```

But we can also ask about the overall effect of `treatment`

- how much of the variation in our dependent variable `value`

can be predicted by (or explained by) the variation between the `treatment`

groups. Such a question can be answered with an ANOVA test:

`::model_parameters(anova(m)) parameters`

```
> Parameter | Sum_Squares | df | Mean_Square | F | p
> -----------------------------------------------------------
> treatment | 72.23 | 2 | 36.11 | 11.08 | < .001
> Residuals | 250.96 | 77 | 3.26 | |
>
> Anova Table (Type 1 tests)
```

As we can see, the variance in `value`

(the *sums-of-squares*, or *SS*) has been split into pieces:

- The part associated with
`treatment`

. - The unexplained part (The Residual-
*SS*).

We can now ask what is the percent of the total variance in `value`

that is associated with `treatment`

. This measure is called Eta-squared (written as \(\eta^2\)):

\[ \eta^2 = \frac{SS_{effect}}{SS_{total}} = \frac{72.23}{72.23 + 250.96} = 0.22 \]

and can be accessed via the `eta_squared()`

function:

```
library(effectsize)
eta_squared(m, partial = FALSE)
```

```
> # Effect Size for ANOVA (Type I)
>
> Parameter | Eta2 | 90% CI
> -------------------------------
> treatment | 0.22 | [0.09, 0.34]
```

When we add more terms to our model, we can ask two different questions about the percent of variance explained by a predictor - how much variance is accounted by the predictor in *total*, and how much is accounted when *controlling* for any other predictors. The latter questions is answered by the *partial*-Eta squared (\(\eta^2_p\)), which is the percent of the **partial** variance (after accounting for other predictors in the model) associated with a term:

\[
\eta^2_p = \frac{SS_{effect}}{SS_{effect} + SS_{error}}
\] which can also be accessed via the `eta_squared()`

function:

```
<- lm(value ~ gender + phase + treatment, data = obk.long)
m
eta_squared(m, partial = FALSE)
```

```
> # Effect Size for ANOVA (Type I)
>
> Parameter | Eta2 | 90% CI
> -----------------------------------
> gender | 0.03 | [0.00, 0.12]
> phase | 9.48e-03 | [0.00, 0.05]
> treatment | 0.25 | [0.11, 0.37]
```

`eta_squared(m) # partial = TRUE by default`

```
> # Effect Size for ANOVA (Type I)
>
> Parameter | Eta2 (partial) | 90% CI
> -----------------------------------------
> gender | 0.04 | [0.00, 0.13]
> phase | 0.01 | [0.00, 0.07]
> treatment | 0.26 | [0.12, 0.38]
```

*( phase is a repeated-measures variable, but for simplicity it is not modeled as such.)*

In the calculation above, the *SS*s were computed sequentially - that is the *SS* for `phase`

is computed after controlling for `gender`

, and the *SS* for `treatment`

is computed after controlling for both `gender`

and `phase`

. This method of sequential *SS* is called also *type-I* test. If this is what you want, that’s great - however in many fields (and other statistical programs) it is common to use “simultaneous” sums of squares (*type-II* or *type-III* tests), where each *SS* is computed controlling for all other predictors, regardless of order. This can be done with `car::Anova(type = ...)`

:

`eta_squared(car::Anova(m, type = 2), partial = FALSE)`

```
> # Effect Size for ANOVA (Type II)
>
> Parameter | Eta2 | 90% CI
> -----------------------------------
> gender | 0.05 | [0.00, 0.16]
> phase | 9.22e-03 | [0.00, 0.05]
> treatment | 0.24 | [0.11, 0.37]
```

`eta_squared(car::Anova(m, type = 3)) # partial = TRUE by default`

```
> # Effect Size for ANOVA (Type III)
>
> Parameter | Eta2 (partial) | 90% CI
> -----------------------------------------
> gender | 0.07 | [0.01, 0.18]
> phase | 0.01 | [0.00, 0.07]
> treatment | 0.26 | [0.12, 0.38]
```

\(\eta^2_p\) will always be larger than \(\eta^2\). The idea is to simulate the effect size in a design where only the term of interest was manipulated. This terminology assumes some causal relationship between the predictor and the outcome, which reflects the experimental world from which these analyses and measures hail; However, \(\eta^2_p\) can also simply be seen as a **signal-to-noise- ratio**, as it only uses the term’s *SS* and the error-term’s *SS*.[^in repeated-measure designs the term-specific residual-*SS* is used for the computation of the effect size].

(Note that in a one-way fixed-effect designs \(\eta^2 = \eta^2_p\).)

Type II and type III treat interaction differently. Without going into the weeds here, keep in mind that **when using type III SS, it is important to center all of the predictors**; for numeric variables this can be done by mean-centering the predictors; for factors this can be done by using orthogonal coding (such as `contr.sum`

for *effects-coding*) for the dummy variables (and *NOT* treatment coding, which is the default in R). This unfortunately makes parameter interpretation harder, but *only* when this is does do the *SS*s associated with each lower-order term (or lower-order interaction) represent the ** SS** of the

```
# compare
<- lm(value ~ treatment * gender, data = obk.long)
m_interaction1
# to:
<- lm(
m_interaction2 ~ treatment * gender,
value data = obk.long,
contrasts = list(
treatment = "contr.sum",
gender = "contr.sum"
)
)
eta_squared(car::Anova(m_interaction1, type = 3))
```

```
> Type 3 ANOVAs only give sensible and informative results when covariates are
> mean-centered and factors are coded with orthogonal contrasts (such as those
> produced by 'contr.sum', 'contr.poly', or 'contr.helmert', but *not* by the
> default 'contr.treatment').
```

```
> # Effect Size for ANOVA (Type III)
>
> Parameter | Eta2 (partial) | 90% CI
> ------------------------------------------------
> treatment | 0.12 | [0.02, 0.23]
> gender | 9.11e-03 | [0.00, 0.08]
> treatment:gender | 0.20 | [0.07, 0.33]
```

`eta_squared(car::Anova(m_interaction2, type = 3))`

```
> Type 3 ANOVAs only give sensible and informative results when covariates are
> mean-centered and factors are coded with orthogonal contrasts (such as those
> produced by 'contr.sum', 'contr.poly', or 'contr.helmert', but *not* by the
> default 'contr.treatment').
```

```
> # Effect Size for ANOVA (Type III)
>
> Parameter | Eta2 (partial) | 90% CI
> ------------------------------------------------
> treatment | 0.27 | [0.13, 0.39]
> gender | 0.12 | [0.03, 0.24]
> treatment:gender | 0.20 | [0.07, 0.33]
```

If all of this type-III-effects-coding seems like a hassle, you can use the `afex`

package, which takes care of all of this behind the scenes:

```
library(afex)
<- aov_car(value ~ treatment * gender + Error(id), data = obk.long) m_afex
```

`> Contrasts set to contr.sum for the following variables: treatment, gender`

`eta_squared(m_afex)`

```
> Type 3 ANOVAs only give sensible and informative results when covariates are
> mean-centered and factors are coded with orthogonal contrasts (such as those
> produced by 'contr.sum', 'contr.poly', or 'contr.helmert', but *not* by the
> default 'contr.treatment').
```

```
> # Effect Size for ANOVA (Type III)
>
> Parameter | Eta2 (partial) | 90% CI
> ------------------------------------------------
> treatment | 0.27 | [0.13, 0.39]
> gender | 0.12 | [0.03, 0.24]
> treatment:gender | 0.20 | [0.07, 0.33]
```

These effect sizes are unbiased estimators of the population’s \(\eta^2\):

**Omega Squared**(\(\omega^2\))**Epsilon Squared**(\(\epsilon^2\)), also referred to as*Adjusted Eta Squared*.

`omega_squared(m_afex)`

```
> Type 3 ANOVAs only give sensible and informative results when covariates are
> mean-centered and factors are coded with orthogonal contrasts (such as those
> produced by 'contr.sum', 'contr.poly', or 'contr.helmert', but *not* by the
> default 'contr.treatment').
```

```
> # Effect Size for ANOVA (Type III)
>
> Parameter | Omega2 (partial) | 90% CI
> --------------------------------------------------
> treatment | 0.24 | [0.10, 0.36]
> gender | 0.10 | [0.02, 0.22]
> treatment:gender | 0.17 | [0.05, 0.29]
```

`epsilon_squared(m_afex)`

```
> Type 3 ANOVAs only give sensible and informative results when covariates are
> mean-centered and factors are coded with orthogonal contrasts (such as those
> produced by 'contr.sum', 'contr.poly', or 'contr.helmert', but *not* by the
> default 'contr.treatment').
```

```
> # Effect Size for ANOVA (Type III)
>
> Parameter | Epsilon2 (partial) | 90% CI
> ----------------------------------------------------
> treatment | 0.25 | [0.11, 0.37]
> gender | 0.11 | [0.02, 0.23]
> treatment:gender | 0.18 | [0.06, 0.30]
```

Both \(\omega^2\) and \(\epsilon^2\) (and their partial counterparts, \(\omega^2_p\) & \(\epsilon^2_p\)) are unbiased estimators of the population’s \(\eta^2\) (or \(\eta^2_p\), respectively), which is especially important is small samples. Though \(\omega^2\) is the more popular choice (Albers and Lakens 2018), \(\epsilon^2\) is analogous to adjusted-\(R^2\) (Allen 2017, 382), and has been found to be less biased (Carroll and Nordholm 1975).

*Partial* Eta squared aims at estimating the effect size in a design where only the term of interest was manipulated, assuming all other terms are have also manipulated. However, not all predictors are always manipulated - some can only be observed. For such cases, we can use *generalized* Eta squared (\(\eta^2_G\)), which like \(\eta^2_p\) estimating the effect size in a design where only the term of interest was manipulated, accounting for the fact that some terms cannot be manipulated (and so their variance would be present in such a design).

`eta_squared(m_afex, generalized = "gender")`

```
> Type 3 ANOVAs only give sensible and informative results when covariates are
> mean-centered and factors are coded with orthogonal contrasts (such as those
> produced by 'contr.sum', 'contr.poly', or 'contr.helmert', but *not* by the
> default 'contr.treatment').
```

```
> # Effect Size for ANOVA (Type III)
>
> Parameter | Eta2 (generalized) | 90% CI
> ----------------------------------------------------
> treatment | 0.21 | [0.08, 0.33]
> gender | 0.10 | [0.02, 0.22]
> treatment:gender | 0.18 | [0.06, 0.30]
>
> - Observed variabels: gender
```

\(\eta^2_G\) is useful in repeated-measures designs, as it can estimate what a *within-subject* effect size would have been had that predictor been manipulated *between-subjects* (Olejnik and Algina 2003).

Finally, we have the forgotten child - Cohen’s \(f\). Cohen’s \(f\) is a transformation of \(\eta^2_p\), and is the ratio between the term-*SS* and the error-*SS*.

\[\text{Cohen's} f_p = \sqrt{\frac{\eta^2_p}{1-\eta^2_p}} = \sqrt{\frac{SS_{effect}}{SS_{error}}}\]

It can take on values between zero, when the population means are all equal, and an indefinitely large number as the means are further and further apart. It is analogous to Cohen’s \(d\) when there are only two groups.

`cohens_f(m_afex)`

```
> Type 3 ANOVAs only give sensible and informative results when covariates are
> mean-centered and factors are coded with orthogonal contrasts (such as those
> produced by 'contr.sum', 'contr.poly', or 'contr.helmert', but *not* by the
> default 'contr.treatment').
```

```
> # Effect Size for ANOVA (Type III)
>
> Parameter | Cohen's f (partial) | 90% CI
> -----------------------------------------------------
> treatment | 0.61 | [0.38, 0.80]
> gender | 0.37 | [0.17, 0.56]
> treatment:gender | 0.50 | [0.28, 0.69]
```

Until now we’ve discusses effect sizes in fixed-effect linear model and repeated-measures ANOVA’s - cases where the *SS*s are readily available, and so the various effect sized presented can easily be estimated. How ever this is not always the case.

For example, in linear mixed models (LMM/HLM/MLM), the estimation of all required *SS*s is not straightforward. However, we can still *approximate* these effect sizes (only their partial versions) based on the **test-statistic approximation method** (learn more in the *Effect Size from Test Statistics* vignette).

```
library(lmerTest)
<- lmer(Reaction ~ Days + (Days | Subject), sleepstudy)
fit_lmm
anova(fit_lmm) # note the type-3 errors
```

```
> Type III Analysis of Variance Table with Satterthwaite's method
> Sum Sq Mean Sq NumDF DenDF F value Pr(>F)
> Days 30031 30031 1 17 45.9 3.3e-06 ***
> ---
> Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
```

`F_to_eta2(45.8, df = 1, df_error = 17)`

```
> Eta2 (partial) | 90% CI
> -----------------------------
> 0.73 | [0.51, 0.83]
```

Or directly with `eta_squared() and co.:

`eta_squared(fit_lmm)`

```
> # Effect Size for ANOVA (Type III)
>
> Parameter | Eta2 (partial) | 90% CI
> -----------------------------------------
> Days | 0.73 | [0.51, 0.83]
```

`epsilon_squared(fit_lmm)`

```
> # Effect Size for ANOVA (Type III)
>
> Parameter | Epsilon2 (partial) | 90% CI
> ---------------------------------------------
> Days | 0.71 | [0.48, 0.82]
```

`omega_squared(fit_lmm)`

```
> # Effect Size for ANOVA (Type III)
>
> Parameter | Omega2 (partial) | 90% CI
> -------------------------------------------
> Days | 0.70 | [0.47, 0.82]
```

Another case where *SS*s are not available is when use Bayesian models. `effectsize`

has Bayesian solutions for Bayesian models, about which you can read in the *Effect Sizes for Bayesian Models* vignette.

Albers, Casper, and Daniël Lakens. 2018. “When Power Analyses Based on Pilot Data Are Biased: Inaccurate Effect Size Estimators and Follow-up Bias.” *Journal of Experimental Social Psychology* 74: 187–95.

Allen, Rory. 2017. *Statistics and Experimental Design for Psychologists: A Model Comparison Approach*. World Scientific Publishing Company.

Carroll, Robert M, and Lena A Nordholm. 1975. “Sampling Characteristics of Kelley’s Epsilon and Hays’ Omega.” *Educational and Psychological Measurement* 35 (3): 541–54.

Olejnik, Stephen, and James Algina. 2003. “Generalized Eta and Omega Squared Statistics: Measures of Effect Size for Some Common Research Designs.” *Psychological Methods* 8 (4): 434.