You'll see a warning about potential suspect results when any of the following are true:
- Fewer than 200 analyzed participants were used for impact analysis
- A small number of analyzed participants could produce inaccurate results, or make it difficult to reach concrete evidence of impact.
- Fewer than 70% of submitted participants were matched
- The match rate is verified to confirm that impact analysis is not potentially skewed due to a significant proportion of participants being ignored. If the match rate is low, this could indicate that the submitted participant and eligible comparison groups are too dissimilar for accurate comparison (e.g. one group has very high persistence prediction scores and the other group has much lower persistence prediction scores). Check that the submitted initiative data accurately reflects the hypothesis you are testing, or review Designing an initiative for analysis using Illume Impact to ensure the best possible setup for analysis.
- Similarity of "After Matching" persistence prediction scores is less than 85% AND/OR similarity of "After Matching" propensity scores is less than 85%
- The similarity of the persistence and propensity scores between the matched participant and comparison groups is verified to confirm that the two groups are similar enough for accurate comparison of outcomes impact. Less than 85% similarity could indicate that the submitted participant and eligible comparison groups are too dissimilar for accurate comparison.
- If either of the "After Matching" graphs do not look like a smooth distribution, contact your Civitas Administrator to submit a support ticket or email firstname.lastname@example.org.
- Overall magnitude of impact (i.e. lift in persistence) is greater than 5% AND there were over 500 analyzed participants
- The magnitude of impact and number of analyzed participants are verified because it is unlikely that a single initiative would show a lift in persistence greater than 5% with more than 500 analyzed participants. In these cases, note whether there were other concurrent initiatives impacting the same students, or verify that the submitted data file accurately reflects the students to be analyzed for only this specific initiative.