BURLINGTON, Mass., 12 November, 2026 — 1EdTech® Global and Learning Data Insights released a new white paper today offering practical guidance for evaluating artificial intelligence in education.
As schools face pressure to decide when AI in educational technology is helpful or harmful, many lack the evidence to make informed choices about safe and effective use. The white paper encourages researchers evaluating effectiveness to ask specific questions, as opposed to the broader, “Does AI improve education?” which is impossible to answer:
- What task is the AI meant to support?
- Is there a trustworthy measure of success?
- Which students does it work for?
- Does it produce consistent results?
For the analysis, researchers reviewed 250 studies to identify how AI was evaluated, what patterns appeared in the findings and where evidence is missing across three assessment tasks: automated scoring, item generation and formative feedback.
Across the three tasks, the review found that stronger evaluations answer three questions:
- What is AI being compared with? Human scores are a useful reference only when raters agree. For generated questions or feedback, studies may use another AI model as a reference, which can favor AI results.
- Can results be compared? Many studies use measures designed for a single task, so their findings should be considered separately rather than compared across studies or products.
- Were fairness and consistency tested? Studies should check results across student groups and repeated runs. If they don’t, fairness and consistency are unknown, and findings may not apply to every student.
With this in mind, the paper provides practical guidelines for developing and testing AI tools in education, as well as best practices when using these tools.
“Education leaders, researchers and technology providers can make better decisions when they share clear standards for evidence. By asking focused questions and using reliable measures, we can better understand which technologies support learning, and for whom,” said Michael Feldstein, 1EdTech Chief Strategy Officer.
The review also found that small, fine-tuned models can be more effective than prompting large frontier models for some tasks.
The white paper is an overview of the research team’s findings and methods. Learning Data Insights plans to publish a full scholarly analysis. The papers reviewed are available through the AI Evidence Hub.