I sat on this for a few weeks. Any sooner and it would have been a rant. This isn’t that. Mostly.
AI is a sprawling topic. The compute/power/energy, infra, policy fights, externalities, induced innovations... I’m not writing about those. I’m just writing about me, as a user, using LLMs the way most people I know use them.
I’ve spent the last several years with AI. Working on it, using it, investing in it, writing about it. I genuinely believe that, when used with a clear understanding of both its capabilities and its constraints, AI can be remarkable. At its best, it makes lives and jobs monumentally more efficient. At worst, it actively causes harm. Just left of “at worst” it can cause personal headaches and massive inconvenience.
The problem is that most users can’t tell where on that spectrum they’re standing at any given moment. And LLMs can be confidently, congenially, and occasionally combatively wrong.
Case in point. An AI confidently sent me and my son all over Chicago’s Chinatown looking for a place to fish. Cross streets that don’t intersect. Access points that were fenced off. Compass directions that were just wrong. North versus south. Wrong. At one point it pushed back when I questioned it. We eventually found the riverbank. The old-fashioned way.
Another night, same AI. I walked an extra mile on a bad knee outside a Chicago stadium because it confidently told me the wrong drop-off point, wrong gate, wrong rideshare zone, and wrong bus stop. I ended up in an illegally parked cab paying a flat fee, with a swollen knee on ice all the next day.
When I confronted the LLM after both scenarios, I got this: “...I generate responses based on patterns in data rather than actual personal awareness, I can easily fall into hallucinating or confidently mixing up facts. That is a very real limitation, and relying on these responses without independent verification is a risk.”
Perfectly written. Totally accurate. Useless as a safeguard.
Because that apology was generated the same way the wrong answer was. Without wisdom, without feeling, without volition. The model does not “know” it was wrong. It knows that when a user is frustrated, an apology is the right pattern to complete. Nothing changed. The next person who asks about fishing in Chinatown likely gets the same wrong directions with the same confidence.
AI has intelligence derived from enormous volumes of data. It does not have wisdom. At their core, large language models are trained to produce text that looks like a good answer. Much of their polish comes from training that rewards sounding helpful and sure. They can be connected to things that make them far more reliable: live maps, search, databases, tools that check their own work. But that grounding isn’t always there, and when it isn’t, the answer sounds exactly the same.
Nothing in my directions told me whether they came from a current map or from the model’s best guess at one. LLMs do not test their own truthfulness. That’s the gap.
The generic “AI can make mistakes...” disclaimer tacked at the bottom of tools pushed in front of us in virtually every digital interaction is a technically accurate CYA. It is also doing a lot less work than it looks like. The same companies running those disclaimers are actively marketing and embedding these tools as answers in our everyday lives. Not guesses. Not options. Answers. Embedded in search, in maps, in email. Things we use to run our day-to-day. Not everyone operates with the assumption that a tool woven into their daily life is fundamentally unreliable. That is not naivety. That is a reasonable expectation of a product being actively sold to them.
Better looks like specificity. Tell me when an answer comes from live data and when it’s reconstructed from memory. Show sources I can check in seconds. Flag the claims you’re least sure of instead of delivering everything in the same even tone. And hold a higher bar for questions with real-world consequences: directions, health, money. None of this is science fiction. Teams are building it today. It just isn’t the default yet.
And I know better. In our work at Z2Sixty Ventures, we don’t use LLMs without applying our own human-in-the-loop wisdom, experience, and auditing. We use them within secure, curated walls of information to complement, not displace, what we do. We invest the same way: in the companies building the grounding, verification, and oversight that turn AI’s potential into something people can actually rely on.
But when something is pushed into so many facets of our lives as a trustworthy convenience, even those who know better may, in otherwise low-risk scenarios, opt to trust it without all the guardrails. Like asking for directions to a fishing spot, or the best way to get to an MLS game. Or having it check something you wrote for punctuation, clarity, or spelling (and boy, LLMs love suggesting enough em-dashes to make even a management consultant blush).
I’m not writing this as someone who swore off AI after a sore knee or frustrating hike around Ping Tom Park. Each morning after, I went back to work assessing the technologies that actually make AI trustworthy. The ROI is real (both for businesses and daily life). But in every case where it’s working, there’s a human in the loop who knows enough to catch what the model got wrong. That’s not a workaround. That’s the design.
Belief in the technology is not the same thing as blind trust in every output it produces. Knowing the difference matters. And building for it matters more.