The messy middle: Lessons from AI leaders

Earlier this year, Forever Human AI held an event for business and tech leaders from across the Pittsburgh community to talk about the AI projects they’re working on, what they’ve seen work (or not), and the challenges they’ve faced along the way. 

With eight presentations from across industries, one takeaway from the day was that there were very few polished success stories. Each speaker built something useful, but many people are still struggling to point to a clear number on the P&L. And we all shared challenges with context, memory, and, of course, people. Here are more of the themes that stood out:

1. Context is the product

 Working across different industries and tools, every speaker arrived at the same conclusion: output quality relies on the context you supply. Depending on how you use AI, that context could be about you, your company, or your users. In every case, context needs to be an asset you build over time rather than a prompt you write once.

2. Trust is earned in small, verifiable increments

Every speaker had a memory or hallucination story - and I’m sure you do too. I once spent way too much of the night arguing with my system about whether an API token was already configured. Another of our speakers asked Copilot for a list of his direct reports’ emails and got 12 of 23 - padded with people who had left the company a decade earlier. Another speaker described a system that was brilliant one day and had no idea what was going on the next.

Improvements in this area won't just happen on their own. It takes effort. My advice is to train AI the way you would a new intern. Start with low-stakes, easy-to-verify tasks. Require approval before taking action. Give feedback to help it understand your needs. Maybe encourage it to write daily summaries to help its memory. Most of all, always check its work.

3. Human-in-the loop has to be a defined strategy

Who presses the button, and what exactly are they confirming? The answer must be spelled out. Leaders and individual contributors both need to create processes that hold us accountable for what we produce. For example, if we're creating a system to automatically resolve ServiceNow tickets, we add a confirm-and-close step with agreed-upon business rules that say what is allowed to close without a person. For other AI uses, it’s as simple as checking and editing content before hitting send. But, no matter what, we need oversight to ensure that AI is helping (not hurting) user and customer experiences.

4. Individual productivity is not enterprise AI

For real ROI, AI capability needs to go beyond productivity tools for single team members or departments. Give people ways to share their AI wins - and if someone suddenly seems extra productive, see if it’s something you could scale. The end goal is enterprise AI where the capability reaches far past the person who created it.

One leader doing this well held a monthly showcase forum for AI plus had a channel for sharing. He also monitored tokens. When he found one developer consuming ten times more than anyone else, he asked what was going on rather than assuming the worst and found a bot that handled 20 recurring ticket types. That bot went to production.

5. Adoption is about leadership, not technology

People don't resist technology, they resist what might happen to them. Getting fired. Losing productivity from working in new ways. Making mistakes. You need change management from day one to address these concerns and questions - and create a culture of learning about and embracing AI in public.

6. Ship before it's perfect, but don’t skip governance

We heard several cautionary tales at the event. Someone vibe-coded an executive dashboard full of call center data and published it with usernames and passwords sitting in the source. A developer granted an AI write access to a production database. Both were caught quickly through consumption monitoring and conversation, but the risk was real.

With so much pressure everyone is under to move fast, you need governance that can minimize risks. One valuable practice is to build a project checklist (executive sponsorship, alignment, team readiness, data readiness, testing, and post-go-live monitoring) and don't launch without completing or having a clear plan for each step.

7. Stop shipping your org chart

When nobody owns the whole experience, the customer inherits every seam between teams and systems. Bolting a chatbot onto each silo only makes it worse. Again, context may be the answer. If your AI understands the customer journey map and where each customer is on that map, it can intelligently connect them with support. The goal is to create AI agents that move past simply answering questions to making intelligent predictions and taking proactive action to get ahead of problems.

8. Orchestra and jazz band

Deterministic software is an orchestra: there is a right answer, a score, and a conductor. Probabilistic systems are a jazz band, improvising within structure. Neither is inherently better - but if something HAS to be right every time, don't make it probabilistic. Use a deterministic solution, or use AI to help you write or call on one.

So what’s next?

If you follow me on LinkedIn, you’ll hear me go deeper on these ideas. And send me a message if you’d like an invite to our next AI sharing event (or are interested in co-hosting with our team!). We’re hoping to make this a regular thing - and perhaps grow the event beyond the Pittsburgh region.