As Matt recently discussed, spending some time reviewing existing research before collecting new data can help us conduct better research. Literature reviews are cost-effective and often help us generate ideas, anticipate data collection challenges, and find existing tools that can advance our research goals. Summarizing existing research has also become one of the most popular uses of Artificial Intelligence (AI) tools in our field, so we recently experimented with using AI to support a client’s  literature review (with their permission, of course). We found that AI can serve as a valuable co-researcher to help us write a stronger literature review—but it also has some critical limitations.

The Project

We recently helped a client—an organization running a summer internship program for rural youth—conduct a literature review to identify the barriers young people in their area face when pursuing higher education or career pathways. Community members had identified the need for this program, and the review explored whether research and public data corroborated their understanding of the challenges young people in their community are facing.

Our Experiments

We tried several approaches to using AI (Claude, specifically) to accomplish this task. In parallel, we also performed each task the old-fashioned way… I sourced journal articles from scholarly databases and synthesized insights with relevant public data. Some approaches to using Claude worked better than others, and so we’ve listed our experiments and graded Claude’s helpfulness at each below:

  • We shared the project context with Claude and asked it to write the literature review for us.
    • Grade: D – The result was grammatically correct but lacked insight and relevance.
  • We asked Claude to identify the best peer-reviewed research papers exploring this topic.
    • Grade: C – Claude did identify several helpful sources, but some other sources were not relevant or not high-quality.
  • We fed Claude particular sources and asked it to summarize the key points, implications for the literature review, and critical limitations of each source.
    • Grade: B – Claude did this well, but some summaries lacked critical implications that were important for our client.
  • We gave Claude specific claims we wanted to make in our literature review and asked for evidence supporting and refuting those claims.
    • Grade: A – Claude generally succeeded at identifying whether high-quality research supported a particular claim (though a human touch was critical in drafting the claims).
  • We asked Claude to help us generate citations for key sources.
    • Grade: B – Claude generated passable citations in APA format but omitted a few author names and published-by dates.
  • We asked Claude to help us edit the literature review.
    • Grade: B – Claude caught many obvious errors, but a human proofreader found additional opportunities to make the prose tighter and more consistent.

Lessons Learned

Use AI to Complement, Not Replace, Human Thinking

In each experiment, we found that while AI would write technically perfect prose, it missed important nuances that make research usable for our client. For example, when we first asked AI to write the literature review, a key insight it identified from the research was that rural youth lack geographic mobility, which limits their economic opportunities. While this is true, it’s too vague to be useful. The final human-generated literature review explored several specific factors that enable and inhibit geographic mobility—including perceptions of economic opportunities, awareness of career paths, family pressure, and the cost of higher education. Our client can design their programming to influence some of these factors and help young people get more clarity on the career path they want to pursue.

On the flip side, we found that AI was particularly helpful as a second reader for key sources. It flagged several important points from a research piece that we missed in our first reading, but confirmed were indeed accurate.

Garbage In, Garbage Out

One consistent challenge we experienced with AI was that it would pull in research sources of varying quality and relevance. For example, when asking AI to identify key sources for our literature review, several sources explored barriers young people face to employment in different countries and cultures—not particularly relevant to our client’s local context. AI also referenced some sources that were not peer-reviewed, despite our explicitly asking it to focus on this type of research.

Critically Examine Your Claims

One of our most useful experiments was giving AI a specific claim and asking it to evaluate the extent to which that claim is supported by existing, high-quality research. This pointed us to research that supported important arguments in the literature review, such as “Educational attainment is a particularly important barrier local youth face when trying to get on a career or higher education pathway.”

Create Clean Documentation

Another area where AI excelled was helping us more quickly assemble summary documents supporting our literature review. For example, AI efficiently assembled a bibliography citing our sources in APA format and a more detailed “data source tracker” that described each key source in two to four sentences.

The Bottom Line

Our experience suggests that AI can be a helpful tool to increase the quality of literature reviews and other research tasks. In these experiments, we found that AI can help us quickly identify important research sources, flag takeaways from sources that we missed, critically assess claims about the research, and help us edit our writing and document our sources. We were also pleasantly surprised that our AI tool did not hallucinate – responses provided were factually accurate and did not include fake sources or conclusions that could not be justified by the underlying data.  However, AI in its current form cannot replace our critical thinking skills and our ability to identify real-world implications of research. Those remain the domain of human intelligence—at least for now.