Causal Claims in Economics

Economic

Summary:

This column analyses over 44,000 working papers over 1980-2023 to assess the evolution of these methods.

Economists play a crucial role in informing policies on pressing issues such as inequality, education, and public health. Over the past few decades, the discipline has undergone a ‘credibility revolution’, emphasising rigorous programme evaluation techniques to establish causal relationships. This shift has enhanced the credibility of economic analyses, but could, on the other hand, have had broader implications for what could broadly be considered marketable research in economics.

The rise of causal inference methods

The credibility revolution is characterised by the adoption of empirical strategies designed to strengthen causal claims. Seminal works by Angrist and Krueger (1991) and Card (1990) introduced natural experiments and instrumental variable techniques to address endogeneity concerns, laying the groundwork for more credible causal inference. Subsequently, methods like difference-in-differences (DiD), regression discontinuity design (RDD), and randomised controlled trials (RCTs) have gained prominence, signalling a paradigm shift toward ‘design-based’ empirical strategies (Angrist and Pischke 2010, Pischke 2021).

To assess the evolution of these methods, in a recent paper (Garg and Fetzer 2024) we analysed over 44,000 working papers from the NBER and CEPR spanning 1980 to 2023. Our analysis reveals a significant increase in the use of causal inference methods over the past four decades. Figure 1 illustrates the proliferation of key empirical methods used in these papers.

To visualise how these methods contribute to constructing economic narratives, we use knowledge graphs to map the relationships between concepts in economic research. Figure 2 presents an example of such a knowledge graph from Banerjee et al. (2015), illustrating the causal impact of introducing microfinance in India.

In this knowledge graph, the authors examine how access to microfinance influences a range of outcomes, from business creation to household expenditure patterns. The high number of causal edges and unique paths indicates a rich and interconnected causal narrative, reflecting the complexity of economic relationships explored in the study. Such detailed mappings can help understand how empirical methods contribute to advancing knowledge in economics.

Publication success versus citation impact

Despite the methodological advancements, there is an ongoing debate about the implications for research dissemination and influence. A particular concern may be that the credibility revolution has given rise to a specific style of economic research that may put more emphasis on the methodological toolbox, rather than the underlying question that policymakers and decisionmakers have to contend with on a day-to-day basis (Jiménez-Gómez et al. 2019). Further, assessing what is economically significant, vis-a-vis what merits consideration on statistical grounds may lead to publication bias, disadvantaging studies that, for example, produce null findings (Chopra et al. 2022) or generate a broad range of theoretically consistent, high dimensional empirical patterns on a broad range of variables of interest that may be jointly significant when viewed as being embedded in a causal chain or a graph.

To explore this, we use knowledge graphs to represent the relationships between economic concepts in each paper. We quantified narrative complexity through measures such as the number of unique causal paths and the depth of causal chains. Our findings suggest a nuanced relationship between methodological rigor, narrative complexity, and research impact.

Figure 3 shows that papers with a higher proportion of causal claims are more likely to be published in ‘top five’ economics journals. Additionally, papers introducing novel causal relationships and engaging with less central, specialised concepts have a higher likelihood of top-tier publication.

Yet, when examining citation counts – a proxy for academic influence – we observe a different pattern. As depicted in Figure 4, while the complexity of a narrative positively correlates with citation counts in top journals, the use of causal inference methods does not necessarily lead to higher citation impact once published. Instead, papers focusing on central, widely recognised concepts tend to receive more citations.

This divergence suggests that while top journals prioritise methodological innovation and complex narratives, broader academic impact is driven more by the relevance of research topics. This raises important questions about the direction and priorities of economic research, highlighting the need for a balance between methodological rigor and engagement with central economic debates (Deaton and Cartwright 2016, Pischke 2021). There is a concern that prominently published research in leading journals could encourage a shift in research focus into areas that may be of marginal broader interest, possibly creating deep ‘rabbit holes’ that may subsequently generate a self-reinforcing publication dynamic, hindering innovation more broadly.

Challenges in replication and data accessibility

The increased emphasis on sophisticated empirical methods brings challenges related to replication and research transparency. For example, Chopra et al. (2022) find a substantial perceived penalty against null results in the publication process, which can distort the scientific record and hinder cumulative knowledge. Such biases can lead to an overrepresentation of significant findings, inflating false-positive rates and undermining the reliability of published research (Brodeur et al. 2016).

Moreover, we observe a rise in the use of proprietary data, with the proportion of papers using private company data doubling from about 4% in 1980 to over 8% in 2023. The use of private data in fields like finance and industrial organisation exhibit the highest proportions. Proprietary data can provide granular insights, but it can also raise concerns about replicability and transparency. Limited access to such data hampers other researchers' ability to verify findings or explore alternative hypotheses (Jiménez-Gómez et al. 2019). Further, the provision of research access to proprietary private data may be skewed towards academics with a broad profile, which could further exacerbate the inequalities in the profession in terms of research access (Fetzer 2022). Alternatively, companies could strategically use (publicly funded) researchers to produce private knowledge goods, outsourcing research and development. Alternatively, they may leverage the credentials of academics or higher education institution to foster brand recognition or to boost corporate social responsibility credentials strategically (Bounie et al. 2021).

Deaton and Cartwright (2016) caution against overreliance on randomised control trials (RCTs) and emphasise the importance of understanding the mechanisms behind observed effects. They argue that without a theoretical framework, findings from RCTs may not be generalisable to other contexts, limiting their policy relevance. The generalisability and scalability of experimental results are crucial for informing policy decisions (Jiménez-Gómez et al. 2019).

Implications for the economics profession

These findings have significant implications for the economics profession. The trade-off between methodological rigor and broader academic impact suggests the need for a more holistic approach to research. As Jiménez-Gómez et al. (2019) argue, experimental economists must tackle the generalisability and applicability of their evidence, ensuring that findings contribute meaningfully to theory and policy discussions. This involves embracing diverse methodologies and focusing on questions with substantial policy relevance (Deaton and Cartwright 2016).

Encouraging transparency and the reporting of null results is essential to maintain the integrity of the scientific process. Miguel et al. (2014) advocate for practices that enhance credibility and accessibility, such as pre-registration and data sharing. Addressing the challenges posed by proprietary data requires balancing the benefits of rich datasets with the need for verifiable and replicable research. Initiatives promoting open science and replication studies can help mitigate these issues (Jiménez-Gómez et al. 2019, Brodeur et al. 2016).

Furthermore, there is a growing recognition of the limitations of focusing solely on statistical significance. As Brodeur et al. (2016) highlight, an overemphasis on significant results can lead to ‘p-hacking’ and inflate false-positive rates. Adopting robust statistical practices and valuing studies based on their methodological soundness and relevance, rather than just significant findings, can mitigate these issues. Emphasising the economic significance and practical implications of research findings is vital for advancing the field (Chopra et al. 2022).

To stay updated on all economic events of today, please check out our Economic calendar

Risk Warnings and Disclaimers

You understand and acknowledge that there is a high degree of risk involved in trading. Following any strategies or investment methods may lead to potential losses. The content on the site is provided by our contributors and analysts for information purposes only. You are solely responsible for determining whether any trading assets, securities, strategy, or any other product is suitable for investing based on your own investment objectives and financial situation.

Risk Disclosure

The risk of loss in trading financial instruments such as stocks, FX, commodities, futures, bonds, ETFs and crypto can be substantial. You may sustain a total loss of the funds that you deposit with your broker. Therefore, you should carefully consider whether such trading is suitable for you in light of your circumstances and financial resources.

No decision to invest should be made without thoroughly conducting due diligence by yourself or consulting with your financial advisors. Our web content might not suit you since we don't know your financial conditions and investment needs. Our financial information might have latency or contain inaccuracy, so you should be fully responsible for any of your trading and investment decisions. The company will not be responsible for your capital loss.

Without getting permission from the website, you are not allowed to copy the website's graphics, texts, or trademarks. Intellectual property rights in the content or data incorporated into this website belong to its providers and exchange merchants.

Recent Searches

Top Searches

Markets

Analysis

User

24/7

Economic Calendar

Education

Data