Natural Experiments

Summary

Very much like the economist panel, the PhD respondents showed an overwhelming appreciation of the introduction of natural experiments in contributing to a more precise understanding of cause and effect in economic settings. A similar level of consensus was observed in the second question as well, which asks whether this credibility revolution has led to an imprvoed understanding of public policy issues.

The third question asks whether researchers often seek good answers instead of good questions. A third of the PhD respondents said “Strongly Agree” while only 4% of the Economist panel responded with the same answer. One respondent commented: “Part of grad school for me has been learning to try to avoid this. An interesting source of identification is not by itself a reason to write a paper.”

Question A

“The introduction of natural experiments to economic analysis of the labor market and related areas has led to a more precise understanding of cause and effect.”

Select Explanations

• While the long-run effectiveness of currently developing methods (looking at you DiD) is not yet known, the movement from not knowing anything -> just use OLS -> use IV and RDD has provided insights that were not available prior to the analysis of natural experiments. And some “basic theory” has been called into question, such as minimum wage effects, and this has spurred increased interest in these areas, e.g. labor market power is a growing topic.
• The development of the minimum wage literature is a good example of how empirical work based on natural experiments can move econ consensus
• The results of some of these experiments may have limited general applicability, but their existence has certainly helped us better pinpoint cause and effect
• This is a weird question. Natural experiments allow us to pin down a causal effect by leveraging some random allocation. Therefore, if we observe more natural experiments, we can better understand the causal effect of one variable on the other, in different setups. The potential outcome framework sharpen our understanding of what the estimands mean, when the effects are driven by additional unobservables.
• NatExp gives more precision and confidence. But NatExp is not a research objective in itself.
• Many papers in this field have sound econometrics and yet disagree on everything

Question B

The ‘credibility revolution’ in empirical economics has improved our understanding of a number of public policy issues, including education, immigration and the minimum wage.”

Select Explanations

• The degree to which this is true will depend on the setting. For questions with minimal general equilibrium concerns, I would be more confident, compared to questions where GE concerns are likely playing a large role.
• We still need models! But as stated this seems plainly true. Would anyone really hold the position that we’ve learned nothing from these methods, especially in last 10 years with the rise of huge administrative datasets to use them on?
• Our tools are still highly imperfect but we’ve gone a long way in the past 40 years to make econometric estimates more reliable
• As we have a better understanding of the estimands in setups with heterogenous effects, and we have more observations of different causal effects, under different setups, we can extrapolate better
• Confirmed what we knew, added precision.
• Improved our understanding CONDITIONAL on using a good theoretical foundation. Simply using linear models to derive an ATT/LATE is not sufficient.

Question C

In pursuit of credible research designs, researchers often seek good answers instead of good questions.

Select Explanations

• Yes, but we are also limited by what questions we can answer with given data/experiments. We can’t always identify the ATE of maximal interest, but that doesn’t mean we should ignore a setting where we can learn about a different TE, which may also be of interest, even if less so.
• Part of grad school for me has been learning to try to avoid this. An interesting source of identification is not by itself a reason to write a paper.
• Also there’s no fun in having good answers for boring questions.
• A side effect of the pressure to produce reliable estimates is that many economists will narrow the scope of their question. Sometimes this means by so much that the question is too limited and no longer interesting
• Demand for causality intensifies the lamppost problem.
• Clearly seen with the usage of data mining and terrible instruments that might deliver significant second-stage coefficients.