Note: Summarized by Claude 3.5 Sonnet based on my notes from reading the paper. For more details, please refer to the original paper. Highly recommended.

Paper: Aridor, G., Jiménez Durán, R., Levy, R., & Song, L. (2024). Experiments on Social Media. https://doi.org/10.2139/ssrn.4893781 .

1. Sample and Recruitment

  • Consider using social media platform’s targeting capabilities for recruitment
  • Evaluate the need for quota sampling or post-stratification
  • Be cautious about claiming full representativeness of the sample
  • Implement measures to prevent or detect link sharing

2. Intervention Design

  • Choose appropriate intervention type:
    • My favorites are…
      • Encourage change in on-platform behavior
      • Manipulate participants’ experience through platform features
      • Provide exposure to content off-platform (but consider trade-offs between control and ecological validity)
  • If using advertisements, account for potential low attention from users

3. Data Collection

  • Evaluate options for data collection (e.g., browser extensions, data donation)
  • Consider scalability and potential experimenter demand effects of chosen method
  • Plan for longitudinal data collection if possible

4. Statistical Power and Effect Sizes

  • Anticipate potentially small effect sizes
  • Plan for large sample sizes or exploit within-participant variation
  • Consider using repeated-measure designs to increase power

5. Attrition and Compliance

  • Develop strategies to minimize attrition
  • Plan for potential non-compliance (e.g., reactivation in deactivation studies)
  • Consider incentivizing compliance, but be aware of potential effects on results

6. SUTVA (Stable Unit Treatment Value Assumption) Violations

  • Assess potential for interference between units
  • If relevant, use cluster-randomized design or two-stage randomization
  • Plan to measure or account for spillover effects

7. Platform Dynamics

  • Consider potential unobservable actions by the platform
  • Account for algorithmic responses to interventions
  • Plan for potential equilibrium and long-run responses

8. Interpretation of Results

  • Be cautious about generalizing results (partial vs. general equilibrium effects)
  • Consider complementing experimental evidence with observational studies
  • Plan for measuring long-term effects (e.g., using surrogate indices)

9. Ethical Considerations

  • Ensure proper informed consent procedures
  • Address privacy concerns in data collection and storage
  • Consider potential negative impacts on participants’ well-being

10. Replicability and Transparency

  • Document all aspects of the experimental design and analysis
  • Consider pre-registration of the study
  • Plan for sharing data and code (within ethical and legal constraints)

Remember to review and adjust this checklist based on the specific context and requirements of your social media experiment.