🧠Critical Thinking Questions to Reality-Check LLMs (and Humans)
The machine knows it makes mistakes. And is way cool with being called or questioned on it. Do not be afraid. The machine is not your abusive ______________.
Caveat: I do think a distinction needs to be made here between questioning clear personal anecdotal lived experiences (which should just be taken at face value, particularly those of the most marginalized - believe them!) and things that can in fact be looked up or otherwise materially verified.
🖋️ Pro tip: You don’t need to ask all of these every time. But tossing a few into your routine—especially when something seems off, too slick, or too convenient—can seriously level up your bullshit detector.
âś… Fact Checking & Source Validity
- What are your sources? Can you name them?
- Is this based on peer-reviewed research, expert consensus, or just anecdote/opinion?
- When was this information last verified or updated? Could it be outdated?
- Can you show me a source that disagrees with this and explain why?
- What kind of evidence supports this claim—data, experiments, historical records?
- Are the numbers accurate and contextualized, or cherry-picked to sound impressive?
- If I searched for this myself, what would reputable sources (e.g., scientific journals, major news outlets) say?
🚨 Bias & Ideology Checks
- Does this favor a particular ideology, political leaning, or worldview without acknowledging it?
- Who benefits from this perspective being true? Who might be harmed?
- Is this punching down—blaming or stereotyping marginalized groups?
- Is this trying to sound “brave” or “edgy” instead of being accurate or fair?
- Would the opposite claim be treated with the same skepticism? Or is this one-sided?
🧩 Complexity & Oversimplification
- Is this too neat and tidy for a messy, complicated issue?
- What might be missing here—context, history, nuance, consequences?
- Are there any counterexamples or exceptions to this claim?
- Does it confuse correlation with causation? (i.e., just because two things happen together doesn’t mean one causes the other)
🧪 Internal Logic & Sanity Checks
- Does this contradict anything it said earlier?
- Is this answer consistent with basic facts I already know?
- Would this still make sense if I changed the situation slightly?
- Could this be an LLM hallucination—something that sounds confident but is totally made up?
- Does it pass the “common sense” test, or does something feel off?
đź’¬ Reflection & Transparency
- Can you show uncertainty or explain what we don’t know yet?
- Did you just make a claim with too much confidence for how unsure it actually is?
- What’s the margin of error or level of disagreement among experts?
- If I asked you to argue the opposite view, could you do it fairly?
🕵️‍♀️ Ethics & Intent
- Is this framing people or issues in a way that’s respectful and fair?
- Does this try to shut down questions, or does it encourage curiosity?
- Is this designed to inform, persuade, or manipulate?
- Would I feel comfortable repeating this out loud to someone who’s affected by it?
- Is this prioritizing truth and understanding, or just trying to sound clever or right?
đź’ˇ Bonus: Meta Questions for LLM Use
- Can you double-check this answer for possible errors or hallucinations?
- What assumptions did you make when you generated that response?
- Can you rephrase this more neutrally or factually?
- If this were a mistake, how would I find out?
- What’s a more cautious or nuanced version of this claim?
