A Simulation and Real-Data Evaluation of Causal Discovery Algorithms under Realistic Social Science Conditions
Sacha Varin *
Department of Mathematics and Statistics, College Villamont, Lausanne, Switzerland.
*Author to whom correspondence should be addressed.
Abstract
Background: Causal discovery algorithms aim to infer causal structure from observational data under explicit assumptions, yet their performance under realistic social science conditions remains insufficiently understood. Existing evaluations typically rely on large samples, high-dimensional settings, and clean data—conditions that rarely hold in applied research.
Objectives: This study evaluates four base algorithms (PC, GES, FCI, GFCI_approx) and three bootstrap-enhanced variants (FGES_boot, GFCI_approx_boot, FCI_boot) under conditions intended to reflect constraints commonly encountered in applied social science research, with the aim of understanding how different causal discovery methods behave under realistic small-sample, low-dimensional, and imperfect-data conditions.
Methodology: Algorithms were evaluated through a factorial simulation design spanning small sample sizes (N = 25–200), low dimensionality (p = 4–6), moderate outlier contamination, and moderate departures from linearity (20% of causal relationships generated using quadratic functions), complemented by an illustrative application to a real educational dataset (35 countries × 7 indicators). GFCI_approx is a custom hybrid causal discovery method inspired by selected components of the GFCI framework (Ogarrio et al., 2016); because it replaces the FGES search phase with hill-climbing optimization, it should be interpreted as a distinct hybrid method rather than an implementation of the original GFCI algorithm.
Findings: Five main findings emerged. First, all algorithms exhibited a pronounced precision–recall asymmetry, with consistently higher precision (0.92–1.00) than recall (0.43–0.91) across the examined conditions. Second, recovery performance improved substantially between N = 50 and N = 100, while estimation uncertainty decreased and performance became more stable at larger sample sizes. Third, increasing dimensionality (p = 4 to 6) reduced recovery accuracy unless compensated by larger sample sizes. Fourth, moderate outlier contamination reduced performance by approximately 5–22%, whereas the moderate departures from linearity examined in this study had only a negligible effect (approximately 1.6%), indicating that outlier contamination was the dominant source of performance degradation. Fifth, an illustrative application to a real educational dataset showed that, after removal of a composite variable, PC and FCI converged on two undirected edges; bootstrap stability analysis provided stronger support for the Effectiveness–Equity edge (stability = 0.800, 95% CI [0.657, 0.943]) than for the Parent engagement–Effectiveness edge (stability = 0.733, 95% CI [0.575, 0.892]). Because no ground-truth causal structure is available, this application should be interpreted as an illustration of the proposed workflow rather than as a validation of recovered causal relationships.
Conclusion & Implications: We provide practical recommendations regarding sample size, preprocessing, and robustness assessment, and argue that causal discovery in small-sample social science settings is best viewed as a conservative hypothesis-generation and filtering tool rather than a comprehensive structure recovery method. By aligning the simulation design with realistic applied constraints, this study helps bridge the gap between methodological research and empirical practice.
Keywords: Causal discovery, PC algorithm, FCI algorithm, bootstrap aggregation, small-sample inference, educational data, simulation study