Today I’m pleased to be giving a talk in Ghent as part of an afternoon of talks on the topic of estimands in trials. Treatment effects are often estimated in clinical trials using regression models for the outcome, with randomised treatment and often some other baseline variables as covariates. The coefficient of treatment is taken as the (estimate of) treatment effect. In my talk today I’ll be discussing whether the ICH E9 addendum on estimands is compatible with such effects or estimands, which I refer to as model-based estimands.
The slides can be viewed using the link below, but in a nutshell, my conclusion is that the addendum is not compatible with such estimands, because the addendum specifies that:
- The effect measure should be a population-level summary measure (suggesting, at least to me, things like means, medians, etc, not parameters in models)
- Definition of the estimand should come before specification of the statistical estimation method
Having drawn this tentative conclusion, I reflect on the pros and cons of model-based versus model-free estimands, in the specific context of randomised trials. Although we are very familiar with model-based estimands, I think there are strong reasons in favour of using model-free estimands in trials.
The slides can be viewed / downloaded using the links below.
Thanks Jonathan, very interesting as always. At a quick glance we probably need to interpret “population-level summary measure” as allowing things that will be/are a parameter in a model to deal with the first bullet point, and just relax a little there? Second bullet point I agree, but would this definition of the estimand require any implicit assumptions required to justify its use, for example proportional hazards if it is to be a single HR? I continue to be more interested in the statistical estimation methods than the estimands in my own research because I like the maths-stats involved in developing new estimation methods, but I do agree that estimands raises a lot of interesting questions so thanks for succeeding (where so many others have failed, haha!) in making me think!
Thanks Jonathan. I agree with lots of what you’re saying but don’t understand your framing as being ‘model-free’ vs. ‘model-based’ estimands. I’m thinking-out-loud so keen to hear your thoughts.
Hypothetically, we could define a conditional odds ratio in terms of non-parametric contrasts of Y^a|X=x, run a large trial for each level of X and estimate the odds ratio by restricting to that value (so each trial has a different target population). We don’t do this for obvious reasons, so have trials that permits a distribution of X in inclusion criteria. You point out that then the main-effect logistic regression assumes the odds ratio is identical for each level of X and is biased if misspecified.
Similarly, you define a rate ratio Δ and say that the negative binomial is biased if misspecified but Poisson is not (incidentally I think this idea comes from econ – Wooldridge talks about this a lot).
My objection is that, in both examples, the problem is not that the *estimands* are ‘model-based’ (since you can write down what they are without a model) but that some seemingly reasonable *estimators* are sensitive to misspecification. The FDA guidance certainly seems concerned with the latter.
I’ve heard people say ‘the estimand is Cox model HR’ / ‘the estimand is logistic regression OR’, and in that case I think it makes sense to say it’s model-based, but that’s arguably not really an estimand in that it’s basically picking a convenient estimator rather than thinking about estimands.
Thanks Tim for your thoughts.
I think your last paragraph is what I am / was trying to say – that I think, but may be wrong, that some people when specifying their estimand apparently according to the addendum, state that the summary measure they are using is the HR from a Cox model or the OR from a (covariate adjusted) logistic model, and that is what I was referring to as a model-based estimand.
Thanks Jonathan. Ok, I’m with you – that case seems a reasonable use of the term.
Thanks Jonathan and Tim, this discussion is indeed helpful! It looks to me like what I would call the “population summary component of an estimand” (for example, a HR) might then correspond to multiple types of what we might now be calling “model based estimands” (for example, the HR from a Cox model or the HR from a Weibull model)? — ie a “model based estimand” imposes more structure than just an “estimand” because the statistical model must additionally be specified..? If a “model based estimand” is a usual estimand with more structure imposed on it then that’s well worth clarifying?
If I have understood correctly (and I have doubts that I have) having specified the statistical model for a “model based estimand” we still seem to be safely in “estimand” rather than “estimation” territory, because a statistical model can be fitted using different estimation methods that include, but is not limited to, maximum likelihood. Presumably if we tried to say the “model based estimand” is “A Cox HR estimated using the partial likelihood” then we would finally, most definitely, cross the line from estimand to estimation. Trouble is that the use of the partial likelihood is so strongly associated with fitting a Cox model, saying “Cox HR” seems to be tantamount to further specifying the estimation method, rather than clearly staying within estimand territory(?). Similar comments apply to other statistical models, like linear regression and OLS.
I think one of the reasons why all this is difficult, for me at least, is that estimands and estimation are so joined at the hip that it is really hard to have a conversation about one without the other. Its rather like talking separately about chickens and eggs. Just as I’m reliably informed that the egg came before the chicken, I am also told by statisticians that the estimand comes before the estimation. But if there was a chicken in my kitchen it would be much more the focus of my attention than my egg because, unlike the egg, I can see it running around actually doing something. Similarly for estimands and estimation, I can see the estimation actually doing something so of course I am inclined to think primarily in terms of that.
Thanks if you made it to the end of this stream of consciousness. Probably best if I continue to focus my research on maths-stats and estimation methods, haha, your post has obviously left me over stimulated, thanks again for it, Dan
Thanks Jonathan. I see two separate arguments you are making here, and don’t really agree with either…
*Estimand before estimation*
I don’t think (?) you would insist on full specification of all aspects of an estimand before any consideration of modelling and estimation. If it’s considered reasonable to assume proportional hazards, then it should also be considered reasonable to use a hazard ratio as an estimand. It’s as simple as that. We don’t need any artificial barriers here.
*Robustness to model misspecification*
Having observed an imbalance in a strong prognostic covariate in your particular trial, you kind of need your model to be a decent model. Even with a large sample size. How comforting is it to know that, marginally, over many samples (covariate, treatment, outcome) from a superpopulation, a poor model still produces the correct confidence interval coverage? In most of those samples the model might not be being stress tested in the way it is in your particular sample. I’m saying this not because I don’t think it’s useful to have robustness marginally, but it’s not the be-all-and-end-all. As soon as we measure covariates, we rely on modelling assumptions, even in a perfect RCT.
Thanks Dominic!
Regarding point 1, I think it is indeed a question I am trying to grapple with here as to whether the addendum implies the estimand should be defined without reference to modelling assumptions, and second, regardless of what the addendum says, whether we should try and do this or not. Clearly you are of the view that it is often sensible and reasonable for the estimand to, to some extent, refer to aspects of the model or modelling assumptions that you plan to make in the statistical analysis.
I don’t really agree with your second point. You say that a good model is needed because you might observe an imbalance in a strongly prognostic covariate. Yet there probably exist other variables we didn’t measure which are also imbalanced between groups in that particular trial and you are not worried about the model not capturing the effects of this (unmeasured) covariate. Of course I am not saying that covariate adjustment is not important and useful, but I don’t agree that as soon as we measure covariates we must rely on modelling assumptions.