The Propensity Score Controversy - Simulations

A few months ago, I wrote a blog post about the controversial use of propensity scores (PS). I believe it would be more informative to illustrate the key issues with supporting evidence from simulations.

Let us consider a simple scenario in which the response of interest, 𝑌, is a continuous variable associated with three prognostic covariates, 𝛸₁, 𝛸₂, and 𝑈. We assume that 𝛸₁ and 𝛸₂ are continuous and observed, whereas 𝑈 is binary and unobserved. Suppose the outcome is generated according to a linear model

𝑌 = τ𝑇 + 𝛽₁𝛸₁ + 𝛽₂𝛸₂ + 𝛽𝑢𝑈 + ε,

where 𝑇 ∈ {0, 1} is the treatment indicator (0 = control, 1 = treatment), τ is the treatment effect, 𝛽₁, 𝛽₂ and 𝛽𝑢 are the regression coefficients for 𝛸₁, 𝛸₂, and 𝑈, respectively, and ε is the error term.

Patient heterogeneity between the treatment and control groups is represented by the differences in the distributions of the three covariates. We simulate a fixed external control dataset (real world data, 𝑇 = 0) with a sample size of 50,000, along with 1,000 independently generated experimental-group datasets (𝑇 = 1), each with a sample size of 100, representing 1,000 simulated studies. The assumed distributions of the covariates and the error term, together with the corresponding treatment and covariate effects, are as follows.

The average treatment effect (ATE) is estimated using four PS-based methods: propensity score matching (PSM), inverse probability of treatment weighting (IPTW), augmented inverse probability weighting (AIPW), and targeted maximum likelihood estimation (TMLE). The crude estimate (Crude) of the ATE is also included for reference.

For both the propensity score model and the outcome model, two analysis scenarios are considered: one including only the observed covariates 𝛸₁ and 𝛸₂​, and an oracle analysis including all three covariates, 𝛸₁, 𝛸₂, and 𝑈. The mean estimated ATE and the corresponding Monte Carlo standard deviation (shown in parentheses), based on the 1000 replications, are presented in the following table.

The crude estimate is clearly biased when the external control population differs from the population of the experimental study. Moreover, all PS methods considered here are essentially “doomed” to be biased when an important prognostic covariate such as 𝑈 is ignored. When all three prognostic covariates are included in the analysis - thereby satisfying the key ignorability assumption - all four PS-based methods provide unbiased estimates of the ATE. Among them, PSM is the least efficient, whereas TMLE is the most efficient.