Your Leadership Team Is Judging AI From the Wrong Seat

This morning I sent Wade, the AI teammate we set up at hello EIKO this week, a link to a post by Andrej Karpathy and asked it to write and publish an article about it on our website. In under half an hour it had read the post, gone through the articles we've already published to match their format, drafted the piece, made the header image, and lined it up to go live on our site (after waiting out an outage at our web host). My part was reading the draft and providing feedback and edits. Nothing in that process looks like the AI that shows up in a typical leadership team's week, where it's mostly drafting emails and summarizing meetings.

That gap is what Karpathy's post is about. He's a founding member of OpenAI and the former director of AI at Tesla, and on October 11 he sketched out who actually understands what AI can do right now. By his napkin math, about 6 billion people have barely touched AI, one to two billion use free chatbots now and then as a better search engine or a writing aid, and only around 20 million (mostly software developers) watch AI agents finish projects that used to take them weeks or months. He calls it a funnel, and he says the gap between those groups is widening.

Drawn to scale, the group that has seen what AI can really do barely shows up. If you're reading an article like this one, there's a good chance you're in the blue section, and so are the people you report to.

The line that stopped me was about who sits in the second group. It isn't just casual consumers. He puts "many professionals outside of math&code" there, and executives by name, because they "spend a lot of their time talking to other people, so while they understand what is happening intellectually, it is still second-hand and a bit abstract."

Leaders only hand AI small tasks, so it looks small

I don't read that as a criticism of executives, and I don't think Karpathy means it as one. It's a description of what the job hands to AI. A day of meetings, one-on-ones, and email produces a steady stream of small tasks (summarize this thread, tighten this paragraph, give me three options for this subject line), and on small tasks the tools are handy without being astonishing. Karpathy's explanation is simple: "A question with a paragraph answer barely stresses the system." You only see what's changed when you hand it something large, hard, and multi-step that you actually care about, and leaders rarely have a reason to.

So the people setting AI budgets, deciding who to hire, and writing the usage policies are forming their views on the version of AI that does the least. Picture a VP who signs up for a free account, asks a handful of questions with paragraph-sized answers, and concludes (reasonably) that it's fine. That conclusion is accurate for what was tested. It was the wrong test.

It cuts both ways, too. Some leaders land on "it's a nice productivity perk" and fund it accordingly, while others swing hard on a vendor demo or a headline and approve something ambitious without any feel for where it breaks. Both are making a strategic call on second-hand information, which is exactly the condition Karpathy describes.

AI is strongest on work you can check

The second idea in his post is the one I'd put in front of any leadership team. He describes AI's capability as jagged, strongest in work that is digital, checkable, and valuable, because that's what the labs train it on. Code is the clearest case, since you can run it and see whether it works, and that's why software developers are the ones feeling the ground move first.

Here's the thing: your company has a lot more work that looks like code than you'd think. A monthly reporting package has right answers you can check against the source data. So does a pricing model, a reconciliation, a contract reviewed against a clear checklist, or the internal tool your operations team has wanted for three years and never got budget for. Wade's article was the same kind of work, since I could read every word before it went live, and that check is what made it reasonable to hand over the whole job. Where you can check the output quickly, you can hand over much bigger pieces of the job, and where you can't, you keep the tasks small and keep a person firmly in charge.

That one distinction (can we check it?) will tell you more about where to start than any vendor's roadmap.

Someone on your payroll has already seen what AI can do

Those 20 dots aren't all at tech companies. Ask around your own company and you may find someone who has quietly gone deep, like an analyst who rebuilt a forecasting model with an AI agent over a weekend, or a controller who automated part of month-end close and hasn't mentioned it because they weren't sure it was allowed. Their experience almost never makes it to the leadership meeting intact. By the time it gets there it's been compressed into "AI is saving us some time," which is the second-hand version all over again.

Those people are the fastest way down the funnel for the leaders above them. Ten minutes of watching a colleague hand an agent a real task from your own business does more than any briefing deck, because it's your data, your process, and someone you already trust.

Five steps for your leadership team this quarter

• Give every leader one real project they already own (a pricing review, a market analysis, a board memo) and have them hand the AI the whole job, with the materials and the goal, instead of asking it one question at a time. Day one is simply seeing how far it gets.
• Pay for the good tier (Claude Pro, for example, is $20 a month) and use its agent features, the modes where the AI works through a multi-step task on its own, reading files, searching, and building things. A few paid seats cost far less than one strategic decision made on the free version.
• Find the people already doing serious work with AI and put them in front of the leadership team, live, with real work.
• Start with the work you can check, and expand from there as your people learn where the tools hold up.
• Decide who reviews AI output before the volume shows up. Reviewing Wade's draft was the one part of that article I still did myself, and as these tools take on more of the work, the checking is what slows you down.

Which seat did you form your opinion from?

Karpathy ends his post on how strange it is that all of this is happening at once: most people have barely used these systems, a large group sees a modest lift, and a small sliver are feeling "the vertigo of the curve going vertical." Inside a single company, all three of those groups can sit on the same floor. The encouraging part is that the third tier isn't a closed club the way the labs are, since it's open to anyone with a paid account and a problem worth handing over. So when you form your next opinion about what AI can do for your business, it's worth asking yourself which seat you formed it from.