Before submitting the initiative for analysis, check the estimated minimum detectable impact or effect size that must be observed to see statistically significant results from this initiative.
The minimum detectable impact required for each initiative is determined using power analysis and a few assumptions on the expected match rate. Power analysis is used to estimate the sample size required to reasonably detect the effect of your initiative.
The match rate refers to the assumed percentage of initiative participants that will be matched with a comparable student through PPSM, using the data you submitted. A 90% match rate is assumed for this power analysis calculation. The number of matched pairs for each initiative determines the sample size. As the sample size increases, the minimum detectable impact decreases. A general rule for impact analysis is to provide as large a student population as possible to see a reasonable difference in persistence rates.
Tip: A reasonable change is typically a ≤ 3.0% difference in persistence rates. Observing greater than a 3.0% difference is unlikely.
Click 'Show me the math' to see details about the assumptions made to perform this calculation:
- Statistical Significance (alpha): The significance level for the analysis, this remains constant at 0.05.
- Power (1-beta): The ability of the analysis to detect the effect of the initiative, this remains constant at 0.8.
- Assumed Match Rate: The estimated percentage of students successfully matched through PPSM (actual matching does not occur until you complete validation). This remains constant at 90%.
- Assumed Number of Matched Pairs: The estimated number of participant-comparison pairs, this is the estimated sample size. This should be equal to the match rate (90%) multiplied by the total number of identified initiative participants.
Click Complete Validation when you have reviewed and validated your submitted data for analysis. Illume Impact will immediately begin matching participant-comparison pairs through PPSM. This process can take up to a day to complete. The initiative will appear in the Pending tab of the home page until analysis is complete.