Citation Snowball Analysis

Validating corpus completeness through forward and backward citation snowballing using the OpenAlex API.

2,205

Papers Screened

30

Already in Corpus

66

New Candidates

0.3%

True Miss Rate

1Overview: What is Citation Snowballing?

Citation snowballing is a systematic literature review technique that uses the citation network of known relevant papers to discover additional relevant papers that may have been missed by keyword searches.

┌─────────────────────┐ │ 65 Corpus Papers │ │ (Known Relevant) │ └──────────┬──────────┘ │ ┌────────────────┼────────────────┐ ▼ ▼ ┌─────────────────┐ ┌─────────────────┐ │ FORWARD SNOWBALL│ │BACKWARD SNOWBALL│ │ Papers citing │ │ Papers cited │ │ our 71 papers │ │ BY our papers │ └────────┬────────┘ └────────┬────────┘ │ │ ▼ ▼ ┌─────────────────┐ ┌─────────────────┐ │ 1,791 papers │ │ 414 papers │ │ screened │ │ screened │ └─────────────────┘ └─────────────────┘

Purpose: To validate that our dual-corpus search strategy (keyword + journal prestige) captured the core mutual fund style drift literature.

2Forward Snowballing Process

Question: "Who has cited our 65 corpus papers since they were published?"

Process

  1. For each of the 65 corpus papers, query OpenAlex API for citing works
  2. OpenAlex returns all papers that have cited that work
  3. Apply relevance keyword filter to identify style drift papers

API Query Example

GET https://api.openalex.org/works?filter=cites:W2123456789&mailto=research@fhgr.ch

Results

Step Count
Corpus papers analyzed 30 (sampled for API limits)
Total citing papers found 1,791
Already in our corpus 22 (validation!)
New relevant candidates 65

Validation Signal

22 papers were found that we already had in our corpus. This confirms our original search strategy was effective at capturing the core literature.

3Backward Snowballing Process

Question: "What foundational papers did our corpus cite that we might have missed?"

Process

  1. Select 15 high-citation papers from corpus (most influential)
  2. Query OpenAlex for their reference lists
  3. For each referenced paper, fetch metadata
  4. Apply relevance keyword filter

API Query Example

GET https://api.openalex.org/works/W2123456789?mailto=research@fhgr.ch Response: { "referenced_works": [ "https://openalex.org/W111...", "https://openalex.org/W222...", ... ] }

Results

Step Count
High-citation papers analyzed 15
Total referenced papers found 414
Already in our corpus 8 (validation!)
New relevant candidates 1

Validation Signal

Low yield of new papers (only 1) confirms our corpus already captured the foundational literature that influential papers build upon.

4Relevance Classification

How did we determine if a snowballed paper was "relevant"?

We applied the same keyword classifier used for the main corpus:

Core Keywords (Automatic Inclusion)

style drift style consistency style timing

Measurement Keywords (Automatic Inclusion)

active share closet index style analysis style box return-based style

Issue Keywords (Automatic Inclusion)

misclassification window dressing

Classification Logic

IF title/abstract contains "style drift" → RELEVANT IF title/abstract contains "active share" → RELEVANT IF title/abstract contains "investment style" AND "fund" → RELEVANT ELSE → NOT RELEVANT

5Results Analysis

66 New Candidates Found - But Were They Really Missed?

Category Count Examples
Out of Scope ~50 Private equity, venture capital, pension funds, bond funds
Below Quality Threshold ~10 Working papers with fewer than 10 citations
Potentially Relevant 5-8 Could strengthen corpus in future updates

Top "Missed" Papers by Citations

Paper Citations Why Not in Corpus
Style Drift in Private Equity (Cumming et al., 2009) 125 Private equity, not mutual funds
Window Dressing by Pension Fund Managers (Lakonishok et al., 1991) 116 Pension funds, not mutual funds
Misclassification of Bond Mutual Funds (Chen et al., 2019) 35 Bond funds, not equity mutual funds
Style Drift in Venture Capital (Koenig & Burghof, 2022) 34 Venture capital, not mutual funds
Shrouded Business of Style Drift (Chua & Tam, 2020) 33 Potentially relevant - consider for update

6Validation Conclusion

Corpus Validation Summary

Total papers screened via snowball 2,205
Papers already in corpus (re-found) 30
New candidates identified 66
Truly missed (mutual fund focus) 5-8
Miss rate ~0.3% (5-8 / 2,205)

Conclusion: Corpus is Comprehensive

The snowball analysis validated that our dual-corpus search strategy (keyword + journal prestige) captured the core mutual fund style drift literature. The 66 "candidates" were mostly papers about related but distinct topics (private equity, venture capital, pension funds, bond funds).

The low true miss rate (~0.3%) provides strong evidence that the 65-paper corpus is comprehensive for equity mutual fund style drift research.

7Technical Implementation

Script Location

literature/scripts/40_snowball_analysis.py

Execution Time

Approximately 10 minutes (due to API rate limiting)

Output Files

File Description
snowball_analysis.json Full results with 66 candidates and metadata
snowball_summary.json Summary statistics for manuscript

Rate Limiting

RATE_LIMIT_DELAY = 0.15 # 150ms between requests (polite to OpenAlex API)

Key Features

  • Timeout handling for slow API responses (30 second timeout)
  • Unicode sanitization for cross-platform compatibility
  • Safe JSON field extraction with None checks
  • Deduplication against existing corpus DOIs
  • Keyword-based relevance classification matching main corpus methodology

Reproducibility

# To reproduce the snowball analysis: cd literature/scripts python 40_snowball_analysis.py # Results saved to: # - classification_output/snowball_analysis.json # - classification_output/snowball_summary.json
← Corpus Statistics Classification Methodology →