Jonathan Bartlett

Summary statistics after imputation with mice

by

Someone recently asked how they could calculate summary statistics after performing multiple imputation with the mice package. The first thing to say is that if you are only interested in calculating a certain summary statistic on each of the imputed datasets, this is easy to achieve. You can extract each imputed dataset using the complete() function, and then apply whatever function you would normally use to calculate the summary statistic in question.

In the rest of this post, I’ll consider the situation where you are interested in performing inference for the summary statistic (or functional if you will). That is, if you are interested in say the median in your data, you are interested because ultimately you are interested in the median of the variable in the population (from which your sample data came from). Viewed this way, the summary statistic is an estimator of a population parameter, and so we should apply the usual procedure for multiple imputation: estimate the parameter on each imputed dataset and its corresponding complete data variance, and then pool these using Rubin’s rules. For some quantities (e.g. the mean), this is pretty easy. For others, at least as far as I can see, it requires a bit more work.

Read more

Is stratified randomisation in trials (at least large ones) pointless?

by

I recently wrote a blog post giving some comments about the revised FDA guidance on covariate adjustment in randomised trials. One of the things I commented on was recent work by different authors on the impacts of stratified randomisation on inference. It is well known (or should be!) that if stratified randomisation is used, the statistical analysis should adjust for the variables used in the stratified randomisation in order for the efficiency gain to be realised.

A fact that was new to me is that if in the analysis one adjusts (in linear models for continuous outcomes) for dummy variables corresponding to each of the randomisation strata, the true variance of the resulting estimator of treatment effect is the same whether simple (non stratified) or stratified randomisation is used, in the most common situation that randomisation to the two arms is 1:1. This was shown by one of the results of Bugni et al 2018. Wang et al 2019 subsequently showed it also holds if additional baseline covariates (not used in the stratified randomisation) are also adjusted for, and that it also holds for the standardisation type estimator of the marginal risk difference based on a logistic regression working model.

These results mean that, at least for large sample sizes, if one considers strata defined by baseline covariates, provided you adjust for these strata as covariates in the analysis, performing the randomisation stratified on these gains you no additional efficiency compared to using simple randomisation. These theoretical results are in agreement with simulation results mentioned on Twitter today by Jack Wilkinson that prompted this post:

The theoretical results in the above papers are asymptotic results. Thus I can imagine it could well be the case that with small sample sizes stratified randomisation does buy you some additional efficiency.

Moreover, in practice I believe it is more common in the analysis model to adjust for only main effects of the variables used to define the randomisation strata, rather than dummy variables for each of their combinations. My guess is that if an assumption that the outcome only depends on these variables via their main effects is correct, the theoretical results mentioned above that imply no (asymptotic) benefit to using stratified randomisation would also hold true for this type of analysis model.

I don’t have time to actively investigate this myself right now, but if anyone is interested in pursuing it and leading on the work for this, please get in touch.

Postscript – a small simulation illustration

The following small simple simulation study in R follows. The setup is very simple – one binary baseline covariate (X) which influences the outcome and either is ignored in the randomisation (simple randomisation) or randomisation is performed stratified on it to ensure balance. In both cases, the analysis is a linear regression adjusting for treatment (Z) and this baseline covariate (X).

library(blockrand)

n <- 250

nSim <- 10000
est <- array(0, dim=nSim)

#simple randomisation
for (sim in 1:nSim) {

  #simulate binary baseline covariate
  x <- 1*(runif(n)<0.5)
  #simulate treatment assignment, using simple randomisation
  z <- 1*(runif(n)<0.5)
  
  y <- x+z+rnorm(n)
  mod <- lm(y~x+z)
  est[sim] <- coef(mod)[3]
}

#look at empirical SE of estimates
sd(est)

#stratified block randomisation

for (sim in 1:nSim) {
  
  x <- 1*(runif(n)<0.5)
  #stratified block randomisation
  z <- rep(0,n)
  n0 <- sum(x==0)
  z[x==0] <- as.numeric(blockrand(n=n0, num.levels=2)$treatment)[1:n0]-1
  n1 <- sum(x==1)
  z[x==1] <- as.numeric(blockrand(n=n1, num.levels=2)$treatment)[1:n1]-1
  
  y <- x+z+rnorm(n)
  mod <- lm(y~x+z)
  est[sim] <- coef(mod)[3]
}

sd(est)

The empirical SE from simple randomisation (based on 10,000 simulations) was 0.1259364 and for stratified randomisation was 0.1254624. This shows that, at least in this setup, the stratified randomisation, does not materially reduce the (true) variability of the treatment effect estimates.

These results are in accordance with a 1982 paper I just came across: ‘A note on stratifying versus complete random assignment in clinical trials’ by Grizzle.

Hypothetical estimands – a unification of causal inference and missing data methods

by

Camila Olarte Parra, Rhian Daniel and myself have just released a pre-print on arXiv (now published in Statistics in Biopharmaceutical Research) in detailing recent work looking at statistical methods targeting so called hypothetical estimands in clinical trials. The ICH E9 addendum on estimands is having a widespread impact on the way clinical trials are planned and analysed. One of the strategies described by the addendum for handling so called intercurrent events is the hypothetical strategy. This is where one hypothesizes of a way in which the trial could be modified such that the intercurrent event in question would not take place. For example, in trials where patients may receive a rescue medication, we could conceive of a trial where such medication were not made available. The goal of inference is then what treatment effect we would have seen in such a modified trial.

In the paper, building on work by others (e.g. Lipkovich et al 2020), we show how causal inference concepts and methods can be used to define and estimate hypothetical estimands. Currently estimation of estimands which use the hypothetical strategy is predominantly carried out using missing data methods such as mixed models and multiple imputation. To do so, any outcome measurements available after the intercurrent event being dealt with using the hypothetical strategy are deleted/ignored, and an analysis using these methods is performed, assuming the resulting missing data are missing at random (MAR). We set out to see how estimation of hypothetical estimands would proceed using the language and machinery from causal inference.

In this post I’ll highlight a few of the things the paper covers.

Read more