Abstract:

Political events may affect not only people’s political attitudes and behavior but also who responds to surveys. If events change who participates, estimates from event-focused designs may reflect changes in sample compositions rather than genuine opinion change. This is particularly challenging for designs like the Unexpected Event during Survey Design (UESD), which identifies causal effects by comparing respondents interviewed immediately before and after an unexpected event. I develop an analytical framework that extends established UESD best-practices, which successfully address observable differences between these samples, by sensitivity analyses to assess the potential impact of unobserved compositional bias. I illustrate the framework using the 2015 Charlie Hebdo terrorist attacks to estimate rally-around-the-flag effects and demonstrate its broader applicability by replicating 14 published UESD studies on terrorist events and rally-type outcomes. Across cases, political events affect sample compositions, but accounting for observable differences rarely changes estimated effects. Further, unobserved compositional bias would typically need to be implausibly strong to overturn substantive results. The proposed framework provides tools to assess and mitigate compositional bias, which strengths causal inference in UESD and other event-focused designs.