Aishwarya Naresh Reganti + Kiriti Badam

AI products differ from traditional software due to non-deterministic inputs/outputs and the necessary trade-off between system autonomy and human control.

"Most people tend to ignore the non-determinism. You don't know how the user might behave with your product, and you also don't know how the LLM might respond to that. The second difference is the agen..."

Successful AI deployment requires a 'problem-first' approach, starting with low-impact, high-control versions to learn before scaling complexity.

"So we recommend building step-by-step. When you start small, it forces you to think about what is the problem that I'm going to solve. In all this advancements of the AI, one easy, slippery slope is t..."

Competitive advantage in AI comes from building iterative feedback loops (flywheels) rather than just being first to market with a static agent.

"It's not about being the first company to have an agent among your competitors. It's about have you built the right flywheels in place so that you can improve over time."

AI leadership requires rebuilding professional intuition through hands-on learning and staying current with rapid technological shifts.

"I used to work with the CEO of now Rackspace, Gagan. So he would have this block every day in the morning, which would say catching up with AI 4:00 to 6:00 AM... I think leaders have to get back to be..."

The black-box nature of LLMs makes predicting the output surface difficult, requiring builders to anticipate a wide range of non-deterministic behaviors.

"LLMs are pretty sensitive to prompt phrasings and they're pretty much black boxes. So you don't even know how the output surface will look like. So you don't know how the user might behave with your p..."

Peer-to-peer multi-agent systems are often less effective and harder to control than a single supervisor agent orchestrating sub-tasks.

"I feel like kind of misunderstood is the concept of multi-agents. People have this notion of, 'I have this incredibly complex problem. Now I'm going to break it down into, hey, you are this agent. Tak..."

Be skeptical of 'out-of-the-box' AI solutions for enterprises; real ROI requires a pipeline that accounts for messy data and infrastructure.

"When someone comes up to me and says, 'We have this one click agent, it's going to be deployed in your system.' ... I would almost be skeptical because it's just not possible. And that's not because t..."

In AI products, implicit signals like 'regenerations' are often more accurate indicators of user dissatisfaction than explicit feedback like 'thumbs down.'

"In ChatGPT, if you are liking the answer, you can actually give a thumbs up. Or if you don't like the answer, sometimes customers don't give you thumbs down, but actually regenerate the answer. So tha..."

Successful AI adoption requires a culture that views AI as an augmentation tool for experts rather than a threat to their job security.

"You want to build a culture of empowerment, of augmenting AI into your own workflows so that you can 10X at what you're doing instead of saying that probably you'll be replaced if you don't adopt AI....."

AI products require a tighter, unified feedback loop between PMs, engineers, and data scientists, replacing traditional siloed handoffs.

"A lot of old contracts and handoffs between traditional roles, like say PMs and engineers and data folks has now been broken and people are really getting adapted to this new way of working together a..."

AI development often fails when teams focus on technical complexity instead of clearly defining and breaking down the core customer problem.

"In all this advancements of the AI... one easy, slippery slope is to keep thinking about complexities of the solution and forget the problem that you're trying to solve. When you're trying to start at..."

The majority of AI product work is deep workflow analysis and data investigation rather than model building.

"80% of so called AI engineers, AIPMs spend their time actually understanding their workflows very well. They're actually in the weeds understanding their customer's behavior and data. And whenever a s..."

Production monitoring should be used to filter and identify specific traces for manual error analysis and failure pattern identification.

"You cannot practically sit and evaluate all the traces. You need some indication to understand what are the things that I should look at. And this is where production monitoring helps. And once you ge..."