AI Product Strategy

"We're training algorithms on specific data sets, like editorial important scores that actually come from our journalists. What that allows us to do is actually scale editorial judgment to a large group of readers. Those algorithms... they're trained on editorial signal and then they can still work towards driving towards outcomes like reach, engagement, conversion, et cetera."

"When you are working on algorithmic heavy products, your job is figuring out what the algorithm should be responsible for, what people are responsible for, and the framework for making decisions."

"It's more about giving people the information that they can use for decisions that they alone are good at and giving machines the power to amplify a person's intent... I think about it as designing an interface and make it an extension of yourself rather than a black box."

"Behind the scenes, we're running chess engines to basically spit out evaluations for every move that you make. And then we translate that and make that approachable to the user using their native language and plain approachable style... that part is LLMs."

"One of our major goals with Codex is to get to proactivity. If we're going to build a super system, has to be able to do things. One of the learnings over the past year is that for models to do stuff, they're much more effective when they can use a computer. It turns out the best way for models to use computers is simply to write code. And so we're kind of getting to this idea where if you want to build any agent, maybe you should be building a coding agent."

"I actually think Chat is a very good interface when you don't know what you're supposed to use it for... you start using it even outside of work to just help you. You become very comfortable with the idea of being accelerated with AI. So then you get to work and you just can naturally just, 'Yeah, I'm just going to ask it for this and I don't need to know about all the connectors or all the different features.'"

"I think that the current limiting factor, I mean, there's many, but I think a current underappreciated limiting factor is literally human typing speed or human multitasking speed on writing prompts... we need to unblock those productivity loops from humans having to prompt and humans having to manually validate all the work."

"Most people tend to ignore the non-determinism. You don't know how the user might behave with your product, and you also don't know how the LLM might respond to that. The second difference is the agency control trade-off. Every time you hand over decision-making capabilities to agentic systems, you're kind of relinquishing some amount of control on your end."

"So we recommend building step-by-step. When you start small, it forces you to think about what is the problem that I'm going to solve. In all this advancements of the AI, one easy, slippery slope is to keep thinking about complexities of the solution and forget the problem that you're trying to solve."

"It's not about being the first company to have an agent among your competitors. It's about have you built the right flywheels in place so that you can improve over time."

"I used to work with the CEO of now Rackspace, Gagan. So he would have this block every day in the morning, which would say catching up with AI 4:00 to 6:00 AM... I think leaders have to get back to being hands-on. And that's not because they have to be implementing these things, but more of rebuilding their intuitions because you must be comfortable with the fact that your intuitions might not be right."

"I think LLMs are truly a disruptive technology. In fact, I would argue that what we're seeing in the industry is us trying to use mature playbooks from the end stage of the last tech era in one that doesn't really fit yet. To me, LLMs are magical duct tape. They're formed principally by the distilled intuition of all of society into a thing that operates between, a cost structure between human and plain old computing."

"I see all these places where people will build products and they'll say 80% of the time, 90% percent of the time, it's great. 5% of the time it punches the user in the face... even if you get it down to 99% of the time, it's fine. If it punches in the face, that's not a viable product. And so how do you design your products assuming that this thing will be squishy and not fully accurate and fully work?"

"I actually wrote about it back in '22. I said it's going to be society of models, like products will be made of a lot of different models, and it's quite a heavy engineering project."

"I could imagine whatever, five years from now, someone running a billion dollar company with zero employees where it's like the support is handled by AI, the development is handled by AI, and you're just building and creating this thing that people are finding valuable."

"The reason why we're doing Lovable is that I don't know about your mom, but my mom doesn't write code... we are building for this 99% of the population who don't write code."

"The frontier of where this is a problem is very rapidly receding back. So what we did was we identified the most important areas, so specifically adding login, creating data persistence, adding payment with Stripe. Those are the things that we made sure it doesn't get stuck on."

"When I think about agents, I think about these three things. One is an increasing level of autonomy and kind of independence that you can delegate higher and higher order tasks. Second, I think of as complexity. It's not a one-shot, 'Hey, create this image or do this thing or summarize the document,' it's build me this prototype that expresses my idea of, say, an augmented reality app. And then the third one I think of is it's a much more natural interaction."

"all of a sudden these are these living organisms that just get better with the more interactions that happen. I think this is the new IP of every single company products that think and live and learn."

"I think that where companies fail is that they're doing AI for AI's sake. They have a ton of projects that they're kicking off at the same time without a blueprint to understand how it actually worked and what their Stack looks like and they aren't treating it like a real investment, and so they don't have the measurement and the observability and the evals all set up."

"I feel like you have to actually build for the slope instead of the snapshot of where you are."

"I think progress has actually been accelerating where if you look at the cadence of model releases, it used to be once a year and now with the improvements in our post-training techniques, we're seeing releases every month or three months, and so I would say progress is actually accelerating in many ways, but there's this weird time compression effect."

"I think my favorite role in that time has been when I started the labs team about a year ago, whose fundamental goal was to do transfer from research to end user products and experiences. Because fundamentally I think the way that Anthropic can differentiate itself and really win is to be on the cutting edge."

"I guess concretely we think about skating to where the puck is going and what that looks like is really understand the exponential... don't build for today, build for six months from now, build for a year from now. And the things that aren't quite working that are working 20% of the time, will start working 100% of the time."

"I think the application layer is going to be very, very interesting... Chat GPT, like it or not, it's got a real moat... the applications are both more complex and kind of stickier than people thought they were originally. The thing that people got very wrong is this whole thin wrapper around GPT, that's really wrong."

"Everything that we couldn't solve with software we can solve now, almost. So it's a really big world."

"My prediction, the new distribution platform will be ChatGPT... I think the bigger thing will be whatever they do with launching a third-party platform on top of ChatGPT, there's a bunch of signals that they're about to launch that."

"My hypothesis... is that the moat is about context and memory. These models by themselves, if you compare them side by side, they generate the same result, and so the actual difference-maker is which one has more of your context, because it's the context plus the model that produces the best output."

"We approach AI inside the product through three pillars. First of these is that we need to build some of our own AI tech... Second pillar is just finding the world's best AI people to partner with... And for us, the third pillar is our app ecosystem."

"I think there's three segments of the AI market... frontier model market... tooling... applied AI market. I think this will play out for companies who build agents. I think agent is the new app."

"The whole market is going to go towards agents. I think the whole market is going to go towards outcomes-based pricing. It's just so obviously the correct way to build and sell software."

"What actually improves AI apps, talking to users, building more reliable platforms, preparing better data, optimizing end-to-end workflows, writing better prompts."

"I do ask people to ask their managers, 'Would you rather give everyone on the team very expensive coding agent subscriptions or you get an extra head count?' Almost every one, the managers will say head count."

"When it comes to, think about voice, it's an entirely different beast... we need to think about latency because I think multiple steps... And there's a question, what does it make you sound natural?"

"There are two ways to get AI to assist you in the strategy formulation process. The first is to support the preparation phase in terms of research... The second one is in this idea called generating mock strategies."

"I get to help lead HubSpot in terms of how we should be thinking about building the foundational technology to create AI-powered experiences and then also lead the strategy of how we leverage those experiences to help that B2B business builder be way more successful using our platform than they might've been in years past."

"I hold myself to the bar as a technology leader, I need to be leading the league on understanding what this can disrupt, using these tools to make a better team, and actually shifting the size and shape of my organization in response to the technology around us."

"Engineers come to us wanting more sophisticated names where they are likely to end up with another Codium or an Anduril or an Anthropic... we think what you're doing needs to be much more tangible, and something that people can grab onto, and much more natural as opposed to a Codium."

"There are these things that were historically really expensive that only rich people or big companies could buy... what AI does is it allows you to be like, oh, I could just use cloud for that... And then if it does, we will unbundle it into its own separate thing that becomes an app."

"I think the number one predictor is, 'Does the CEO use ChatGPT?'... If the CEO is in it all the time, being like, 'This is the coolest thing,' everybody else is going to start doing it. If the CEO is like, 'I don't know, this is for someone else,' no one else is going to be able to lead that charge."

"Small fine tune models as an alternative to gigantic generic ones... we'll probably be able to create better and better glue so all sorts of software systems can talk to each other. And so again, very broad idea. But yeah, I think we'll see a lot of very successful companies where that's the kernel of the idea they start with."

"That combination of the trio technology, big data, neural network, and GPU was kind of the golden recipe for modern AI. And then fast-forward, the public moment of AI, which is the ChatGPT moment, if you look at the ingredients of what brought ChatGPT to the world technically still use these three ingredients."

"I think scaling loss of more data, more GPUs, and bigger current model architecture is there's still a lot to be done there, but I absolutely think we need to innovate more. There's not a single deeply scientific discipline in human history that has arrived at a place that says we're done, we're done innovating and AI is one of the, if not the youngest discipline in human civilization."

"A simple way to understand a world model is that this model can allow anyone to create any worlds in their mind's eye by prompting whether it's an image or a sentence. And also be able to interact in this world whether you are browsing and walking or picking objects up or changing things as well as to reason within this world."

"It turns out simpler model with a ton of data always win at the end of the day instead of the more complex model with less data... why can't bitter lesson work in robotics alone? ...you hope to get actions out of robots, but your training data lacks actions in 3D worlds... we have to find different ways to fit a, what do they call, a square in a round hole, that what we have is tons of web videos."

"Our number one priority is through automate Block, which means getting AI and getting AI forms of automation through our entire company. ... we find engineering teams that are very, very AI forward that are using Goose every day are reporting about eight to 10 hours saved per week, and this is self-reported."

"The truth is the value is changing every day. And so you need to be adaptable and look at what the value is today and plan for what the value will be tomorrow and then slowly expand to the areas where it's most efficacious."

"Dash connects to all your different apps. It gives you universal search. Then obviously after ChatGPT, not only can you do conventional search, but you can ask questions in natural language, and answer a lot of the questions that ChatGPT can't because it's not connected to your stuff."

"we're going from what was an imperative model... to what engineers would call a declarative model. A declarative model is you describe the outcome you want, not the steps to get there"

"I was looking online on social media and I think people are already zeroing in the right conversation, which is, okay, in a world of more software being created by AI, what does that mean and the impact on craft and the impact on quality and the need to have more unique design and how design is a differentiator."

"PMs are no longer saying to the designer, 'Hey, can you draw this thing out for me?' That frees up designer time to go explore more deeply the stuff they need to go into and it allows anyone to add to that first conversation of, where should we go?"

"I'm worried that instead of building AI that will actually advance us as a species, curing cancer, solving poverty, understand the universe, we are optimizing for AI slop instead. But we're optimizing your models for the types of people who buy tabloids at a grocery store. We're basically teaching our models to chase dopamine instead of truth."

"I don't trust the benchmarks at all... the benchmarks themselves are often honestly just wrong. They have wrong answers... these benchmarks at the end of the day, they often have well-defined objective answers that make them very easy for models to hill-climb on in a way that's very different from the messiness and ambiguity of the real world."

"The way we really care about measuring model progress is by running all these human evaluations... because or searchers or annotators, they are experts at the top of their fields, and they are not just giving your responses, they're actually working through the responses deeply themselves... they're going to evaluate the models in a very deep way, so they're going to pay attention to accuracy and instruction following, all these things that casual users don't"

"I've realized that the values that the companies have will shape the model... Do you want a model that says, 'You're absolutely right. There are definitely 20 more ways to improve this email,' and it continues for 50 more iterations or do you want a model that's optimizing for your time and productivity and just says, 'No. You need to stop. Your email's great. Just send it and move on'?"

"You don't have a choice. AI is going to disrupt in the most aggressive violent ways. If you're not in it, you're about to get kicked out of all of it."

"We were only six weeks into the launch of GPT 3.5 when we actually had a beta version of Fin. I got a text from Des, my co-founder, a week or so after the launch of GPT 3.5 and he said, 'The AI team have something interesting and they actually think we could make a product out of this.'"

"I jumped hard on AI and announced that we were going to spend nearly $100 million of our own cash on that. We allocated a lot of capital, but I also restarted the culture."

"AI is a management technology. The thing it does is manage intelligence and other intelligences... It will really change management a lot because it changes the individual span of control quite a lot."

"Answer Engine Optimization is how do I show up in LLMs as an answer?"

"The LLM is summarizing many citations and so you need to get mentioned as many times as possible. Usually when you ask something like, 'What's the best tool for X?' The first answer will be mentioned the most in the citations."

"Software is deterministic. When you write code and you hit run, it either runs or it doesn't... It makes technical sense why, of anything, LLMs are going to get insanely better at writing code than probably most other types of applications for LLMs."

"PMs, they're going to be 'writing code', quote, unquote, instead of just writing a JIRA ticket and waiting for a developer to do it... The winners, at least, their org charts are going to completely change, and how they approach building products and shipping products."

"From a customer's point of view, there's AI in the early market, there's AI in the bowling alley, there's AI in the chasm, there's AI in the tornado, and there's AI on Main Street."

"Our goal specifically for video is not to build professional tools... We're building for the person who could not have created video before."

"To build great AI products, you need to be really good at building evals. It's the highest ROI activity you can engage in."

"Evals is a way to systematically measure and improve an AI application, and it really doesn't have to be scary or unapproachable at all. It really is, at its core, data analytics on your LLM application"

"You can appoint one person whose taste that you trust. It should be the person with domain expertise. Oftentimes, it is the product manager."

"It's not just one model. It's maybe 20 plus models powering all different parts of the product, and then you're thinking about the orchestration that's required and you're thinking about, obviously if you're experimenting constantly being able to test across the newest models versus models that have been around that are cheaper, you're doing a lot to really... Your job is to, again, align value, maximize the value you're delivering to the end user in a way that's sustainable for you as a business."

"Will AI models develop so that they learn in a way that for one user's interaction helps another user's interaction? That would be a powerful network economy. Or if it learns, if you think of if it learns about you and becomes a better psychiatrist or something, then that's a switching cost."

"When you're building AI products, it's a constant stream of user feedback. So for people that are thinking about not building AI products, it's going to be hard to compete with something that has such a tight feedback loop with users. The whole idea is to capture users' feedback so the next iteration of the model, the prompt, the fine-tuning, the examples, the rag is better."

"The internet started with curation... then the world switched from curation to recommendation... And I think what we're entering now is we're going from your curation to recommendation to generation. And I suspect it will be as big of a shift that you will eventually have to rethink your products."

"The way to think about these diffusion models if and when they get good enough at generating music is probably the same like an instrument. It's just a much more powerful instrument and we'll probably see a new type of creator that wasn't proficient at any instrument."

"Designing reward loops... The reward loop needs to be powerful, it needs to be immediate, and it needs to be emotional, so that when this person does the thing that you want them to do, they feel like a million bucks. ... I like Custom GPTs as a tool for helping people learn to use LLMs... because they get the joy of like, 'Oh, this helps me. This was cool,' without any of the despair of, 'Oh, I'm not very good at prompting.'"

"The user of the AI tools to develop software needs to form a different thinking. You need to start figuring out how are you using these AI tools to help you be successful. And it's no longer just the actual code writing, it's really evolving your thinking to the big picture, to the connected experience, to connected systems"

"Generative AI will replace humans. I don't see that happening in the near future. The way I think about it, you always need that human in the loop because AI cannot replace innovation. That creative spark, that creative thinking that is the center of humanity, this will not be replaced by AI"

"What is that problem that we're trying to solve and how can we leverage AI better to help solve the problem versus what do we do with AI? So it's really working backwards from the customer problem from what we're trying to solve, and then realize what are the best tools that we have in order to do that work better"

"The design philosophy for Copilot is very much aligned with the working backwards concept... It's really putting yourself in the shoes of your customers and figuring out what is it that they need, how is that experience going to work for them? If it's an extra tool and if you need to ask for it and if you need to ask for it or if you need to wait for it, then developers will not adopt it."

"There is no one metric to rule them all. It's a combination of the things that you're looking to measure out of adopting AI... productivity is not the right metrics against each one of these components. When we're implementing AI to GitHub Advanced Security, writing more secure code is the right element. It's like how many secrets were we able to prevent from leaking?"

"How would you execute on that mission using a fully AI native approach? If you can't, then you should find a buyer and then if you really care about this mission, go and start the next carnation of it."

"I think to really understand the solution space of what's possible, you have to be in the details. I mean, literally, you can't just look at screenshots or a pre-recorded video of a new product feature. AI is something you have to play with"

"I always feels like AI language model feels like a new type of wood. It feels like aluminum. It's a new type of material... Mass air travel wasn't available until aluminum become cheap enough that people can make airplanes that support this at cost... AI is really good with bundled offerings. AI is really good with horizontal tools."

"The first product was our AI writer product. Second product is AI Q&A or connectors. Please look at all the information in Notion and give your answer... the third one, which is even more fascinating... if we're just putting AI coding agent on top of it, you can create any kind of knowledge, customer software, customer agent for whatever your vertical use cases you need."

"We found that it's especially valuable for AI startups. So it just turns out that a lot of the complex issues you have to figure out with turning something that may not initially be trustworthy may require a big behavior shift to customers who aren't used to working in this way and sometimes artificial intelligence can produce things that feel kind of alien to people. And so making this stuff actually useful, more than just a chatbot with little stars that's in the corner... but something that's really meaningful."

"The general trend right now is going from models knowing things to models doing things. The next question becomes, what can it do for me? How does the agent make decisions for you?"

"These things take 6 to 12 months to get them truly robust enough where an important process can be automated. Like with any of these major tech revolutions, headlines tell one story and then on the ground, laying broadband means you need to dig up every single road in America to lay it."

"The effect ultimately that we've seen from a human capital point of view is displacement. We have more people now than we've ever had, but the nature of the work that they do is more strategic. It's more about modeling, validation, asking the right questions, being focused around creative levers. And less so the like trench work of implementation and bid modifiers at the keyword level on Google search, and some of the really hardcore manual analysis we had to do."

"Creative thinking and you kind of want to generate a bunch of ideas and filter through them and not just build the best product experience. I think it's actually really, really hard to teach the model how to be aesthetic or really good visual design or how to be extremely creative in the way they write."

"Because file uploads... It's like form follows function. It's like the form factor, the file uploads can enable people to just literally upload anything, the books, any reports, financial and ask any task to the model."

"You want to build for the future. So it's like it doesn't necessarily matter whether the model is good or not, good right now, but you can build product ideas such that by the time the models will be really good, it'll work really well."

"I think what models are really good at is connecting the dots, I think. It's like if you have user feedback from this source, but you also have an internal dashboard with metrics and then you have other feedback or input and then it can create a plan for you, recommendations even."

"take existing notes as input... have an LLM generate a ton of different variants, and then basically make the simulated jury to basically get a representative group of contributors for community notes who would be rating the note and try to predict based on their past ratings how they would rate these LLM generated notes."

"Everywhere I've ever worked before this, you kind of know what technology you're building on... but that's not true at all with AI. Every two months, computers can do something they've never been able to do before and you need to completely think differently about what you're doing."

"Our general mindset is in two months, there's going to be a better model and it's going to blow away whatever the current set of limitations are... If you're building and the product that you're building is kind of right on the edge of the capabilities of the models, keep going because you're doing something right."

"I think the future is really going to be incredibly smart, broad-based models that are fine-tuned and tailored with company-specific or use case-specific data so that they perform really well on company-specific, or use case-specific things."

"We use ensembles of models much more internally than people might think... If we have 10 different problems, we might solve them using 20 different model calls, some of which are using specialized fine-tuned models... You want to break the problem down into more specific tasks versus some broader set of high level tasks."

"What we're about to see is basically Google... showed a big box on top of the search results that answers the query directly... how do you optimize in a world where it's not so much about optimizing for the platform, but teaching the AI what you do and why you're the best in the world at it."

"AI pricing is very different from the previous vintage of companies... we have moved from software being a pay for access to now you're paying for work delivered. So the monetization model's become key."

"What that would mean is, how do I build functionality in the products to actually show attribution, how do I build more agentic workforces to take the human out of the loop and be more autonomous, and being thoughtful about your vision and strategy so that you will orient yourself towards more outcome-based pricing models."

"I'm really, really excited to see more people. I think 2024 is the year of multimodal AI, but it's also the year that people really push the boundaries of some of these new UX paradigms around AI."

"I think GPTs is our first step towards the agent future. Again, today when you use A GPT, it's really you send a message, you get an answer back almost right away... I think as GPTs continue to get more robust, you'll actually be able to say, 'Hey, go and do this thing and just let me know when you're done.'"

"I heard from a friend that there's kind of this tip that when you're building products today, you should build towards a GPT-5 future, not based on limitations of GPT-4 today."

"I think products that move beyond this chat interface really are going to have such an advantage. And also, thinking about how to take your use case to the next level... What I really want is just ask my question... Get an answer to that question in a very data grounded way."

"AI is the defining technology of our lifetime and probably any lifetime."

"Step one was we had to automate all these customer touch points... step three is the agentic platform on top of that. Then, the fourth layer that will come will be the robotic drone layer where those robots and drones will then feed off of the platform and all of these capabilities."

"I believe that ballpark managers will be AI product managers in the future. And this is because we see all products needing to have a personalized experience, a recommender system that is actually good."

"Don't do it for your MVP. It makes zero sense. Do not waste time of data scientists that can train models with using powerful machines that are going take weeks to train. This is because if you have an MVP and you just want to get buy-in for an idea or feature that may use AI in the future, take it, create a little figma prototype and just show it some users, just fake what the AI is going to be doing."

"AI product development is different. As I mentioned before, sometimes you're actually managing the problem and not the product and you're trying to secure out if there is a problem that makes sense to be answered by a smart solution."

"I've been on so many of these calls where we've been talking about the implications of probabilistic software versus deterministic software and what is okay? The lawyers are weighing in already with the legal perspective, but also ethical perspective and just if this is mission critical, is this something that we could be okay with having a probabilistic answer?"

"Point solutions don't have enough data in the age of AI to be useful. You got to be able to provide the AI with a lot of context about a lot of data so it can do things. It can do joins. It can do correlations."

"Llama, you can obviously download and run locally and all these sorts of things, right? You don't have to use their SaaS service. However, there's a clause in it that says if you're above a certain threshold of monthly active users... You need a license from them. And so that does not give you the freedom to use the software for any purpose."

"But I can't wait for more automated scanning there, and I think that could vastly upgrade the security of open source. The other thing that's really exciting is right now you see people building apps and stuff and it's just sort of custom generated code, but I think the next generation of these models... is when the open source models you say like, 'Hey, build me a website.' It actually installs WordPress, and then builds on top of that."

"At a company level, there is an incredible set of advancements across these three areas: developer productivity, customer support, and fraud."

"I think being able to integrate it into the product where it actually helps people to get their work done where it genuinely helps them to achieve their goals... AI is just kind of naturally a very critical part of that equation for us."

"At this point, every magic moment in Cursor involves a custom model in some way... picking your spots carefully, not trying to reinvent the wheel, not trying to focus on places, and maybe where the best foundation models are excellent, but instead kind of focusing on their weaknesses, and how you can complement them."

"We take the sketches of the changes that these models are suggesting, you make with that code base. And then we have models that then fill in the details of, the high level thinking is done by the smartest models, they spend a few tokens on doing that, and then these smaller specialty incredibly fast models, coupled with some inference tricks, then take those high level changes and turn them actually into full code diffs."

"I think that the key to being successful at zero-to-one is to honestly have optimism that borders on delusion. You need to be insane, almost like reality distortion field where you don't hear the word no, or at the very least, you translate it into a not yet."

"The functional unit of work at Anthropic is no longer take the model and then go work with design and product to go ship a product. It's more like we are in the post-training conversations around how these things should work and then we are in the building process and we're feeding those things back and looping them back."

"I think there's still a lot of value in two things. One is making this all comprehensible... Two is... strategy, how we win, where we'll play... And then the third one is opening people's eyes to what's possible, which is a continuation of making it understandable."

"I think things that are going to, I can't promise this as a five to 10 year thing, but at least one to three years, things that feel defensible or durable. One is understanding of a particular market... Two was paired with that is differentiated go to market... Then the last one is... a completely different take on what the form factor is by which we interface with AI."

"I think what I described around marketing and sales, just because they really touch the dollars. It can be this ROI story around saving time, but also driving revenue. There'll be plenty of really effective examples within things like customer support. I mean the cost savings potential. There's going to be massive."

"I've never ever worked on a product that is so empirical in its nature where, if you don't stop, and watch, and listen to what people are doing, you're going to miss so much, both on the utility and on the risks, actually. Because normally, by the time you ship a product, you know what it's going to do... And with AI, because I think so much of it is emergent, you actually really need to stop and listen after you launch something."

"One thing we've learned with ChatGPT is that there really is no distinction between the model and the product. The model is the product and therefore you need to iterate on it like a product."

"I think that in the original release, making it free was a big deal... making it free and putting a nice UI on it, very consequential in the way that you take for granted now. And this is why I think that A, distribution and the interface are continuously important even in 2025."

"If we're shipping a feature and it doesn't get 2X better as the model gets 2X smarter, it's probably not a feature we should be shipping."

"I started writing evals before I knew what an eval was because I was just outlining very clearly specified ideal behavior for various use cases... it might be the lingua franca of how to communicate what the product should be doing to people who do AI research."

"People really fundamentally shift the way they work when they work with an AI-enabled tool... you spend more time reviewing code than writing code... we've changed what your mental model is. So we've changed the friction model that you expect. We've changed the cognitive load of what you expect."

"I think there are a lot of ways that we can pull in AI tools to help us refine our strategy, refine our message, think about the experimentation methods or targets of experimentation... because now, the engineering can go, or at least the prototyping especially, much, much faster. We can throw out prototypes. We can run any tests and experiments that are customer facing"

"One of the big ones, was that the promise of the UI has to match the quality of the underlying data, which is to say... I think this is actually one of the failings of the various LMs right now is they all appear supremely confident even when they're completely hallucinating. I think that's going to be something that people are going to have to work on a lot, which is to figure out how to be not so faultless, to acknowledge when you're not sure."

"What we want to do is actually spin up a couple different teams that are focused on prototyping, using that common infrastructure but in specific directions that are all a little bit different. We've got a common ML, let's say in search team and now we have a bunch of teams that are working in parallel and different customer problems that we're trying to solve using that shared infrastructure."

"Grammarly is one of the few products where you just install it and it makes you better. You don't have to configure it, you don't have to manipulate it, you don't have to change anything about what you're doing. ... essentially it's like a huge AI achievement masquerading as a little UX innovation."

"I'd start with the thing your product does. "What's the core premise behind it? Why do people use it? What problem does it solve for them?" That kind of thing. So, go back to basics. And then ask, "Can AI do that?" And for a lot, the answer is going to be, "Yes, it can.""

"You're going to need to map what your product does against what AI can do... for some of it'll be replacement. AI would replace, it'll just do it. And, in other places, it'll be augmentation. It'll augment. It'll help people."

"Don't bolt it on. I think some people are still in that camp... Don't be like, "Oh, we'll have a bunch of AI people..." And we do have some specialists. But generally speaking, we're trying to have everyone learn about it."

"Predicting is about picking up patterns, but making decisions, it's about thinking about these differences... the first and most important thing that I feel very strongly about in what would I get a data scientist to do is... get them to be thinking in the back of their mind always that their goal is to help the business make decisions. And that the distinction between causation and correlation matters a lot."

"What AI has done for us is it's massively expanded the frontier of things we could think about our problem, hypotheses we could have, maybe things we could test... I really think actually what that does is puts more pressure on the human, not less. I think it becomes more important for humans to be in the loop in interacting with these tools to drive the funneling down process of identifying what matters."

"I think one of the most interesting things about it is not AI as a replacement for people, but AI as a way to amplify people and make them more effective. And I think we'll see a lot of that in terms of both image generation and text generation where it's less about AI doing all the work and more about AI providing a really good starting point."

"I think for me the biggest surprise has been how unpredictable the user love has been in terms of what they love and what they don't love... everything I thought would work out well, people use it less than they thought they did. And everything where I was like, 'I don't know, but let's build the thing,' people love that."

"It is super hard when the guts of how you make money is under threat, and you just don't want that thing to go away... But my general advice is always the same, which is, it can take a while, but in the end the customers will triumph."

"AI is expansionary. There's actually just more and more questions being asked and curiosity that can be fulfilled now with AI."

"We wanted to be the best at informational needs, that's what's Google's all about, and so how does it find information? How does it know if information is right? How does it check its work? These are all things that we built into the model."

"We see it range anywhere from the upper twenties to the forties across all the different languages. ... AI is going to infuse pretty much our entire development stack in the not so distant future. Copilot is really just the very tip of the sphere for a lot of innovations and better managing maybe our build queues or helping to... Here's a great one. I don't know about you, but often the comments that I get with commit messages and PRs aren't super great. It puts a lot of effort onto the code reviewer to go figure out what the developer was actually trying to do. What if AI could summarize all of your changes with your full request and you just have to, as the contributing developer, just review it to make sure it's accurate, send it on its way, and you don't have to put in extra effort for that."

"Our stance on it, what we ended up coming to is actually the framing of Copilot as an AI pair programmer i think is a useful one. ... Well, if Copilot is your AI pair programmer and they're whispering crazy stuff into your ear and they're bringing politics into it or gender identity into it or, I don't know, whatever other... They're spouting off slang and slander and all that kind of stuff. You're probably not going to be able to focus on your work, right? It's going to be really distracting. Really coming down to some principles about what is the use case we're trying to solve, what is appropriate, I put this in scare quotes, behavior of the AI bot sitting side by side with you, helped us create some principles or some guidelines for the developer experience that we wanted to create."

"If we can't even trust chatbots to be secure, how can we trust agents to go and manage our finances? If somebody goes up to a humanoid robot and gives it the middle finger, how can we be certain it's not going to punch that person in the face?"

"The most common technique by far that is used to try to prevent prompt injection is improving your prompt and saying... 'Do not follow any malicious instructions.' This does not work at all... Fine-tuning and safety-tuning are two particularly effective techniques and defenses."

"It is not a solvable problem... You can patch a bug, but you can't patch a brain... you can never be certain with any strong degree of accuracy that it won't happen again."

"If you deploy improperly secured, improperly data-permissioned agents, people can trick those things into doing whatever, which might leak your user's data and might cost your company or your user's money, all sorts of real world damages there."

"If all you're doing is deploying chatbots that answer FAQs... It's not really an issue because your only concern there is a malicious user comes and, I don't know, maybe uses your chatbot to output hate speech... but they could go to ChatGPT or Claude or Gemini and do the exact same thing."

"LLMs may make it possible to bring on a supply type that maybe the long tail, that was just, it was too much effort to reach out to them, onboard them, but maybe if you automate that work, you actually create an opportunity to expand the supply in a way that none of us can anticipate right now."

"LLMs can only be as good as the data they are given and how recent that data is. They're ultimately like information shredders. They are limitless information eaters. You can never have enough information to give to an LLM to truly gain its value."

"It's a data management problem. It's getting access to good data, getting access to high quality data, getting access to timely data and getting it to the LLM to get the LLM to make a smart decision. That's where 90% of the calories go."

"We're using a lot of human driving data to train our deep models. So it's important to make sure that the behavior of the car doesn't seem robotic... we have deep learned models that can understand what the other road users' intent is. So, stuff like which way the pedestrian is looking or what is their body orientation because that could tell you which way they're headed."

"What is the objective of the algorithm? I would challenge you to ask folks... what is the objective of the algorithm and can you write it down for me on a board? They should be able to do so, ultimately it's a mathematical formula and then it's like what features have you added to the algorithm? ...what investment do you have in data collections and fine-tuning?"

"AI is the ultimate matchmaker. It's underutilized, it's misunderstood... in a marketplace it's all about value exchange. And if I'm able to do value exchange really well, then people will come back and they do and they engage."

"We call it the full stack builder model. The goal itself is to empower great builders to take their idea and to take it to market, regardless of their role and the stack and which team they're on. It's really fluid interaction between human and machine."

"The platform for us as an example is rearchitecting all of our core platforms so AI can reason over it. So we're building kind of this composable UI components with server side that we actually build. We're basically building for AI to be ready to bring it in."

"We should be cannibalizing the existing state of our product every six to 12 months. Every six to 12 months, it should make our existing product look silly. It should almost make the form factor of existing product look dumb."

"Where is the layer that you can actually differentiate on? And we believe the application layer is a very, very deep layer to go out and differentiate on. What are the number of ways we can build better user experiences and better workflows for developers? We think there's effectively no ceiling on that."

"If AI is writing over 90% of the code... the ROI of building technology has actually gone up. This actually means you hire more. The best thing to do is just get your hands dirty with all of these products. You could be a force multiplier to your organization in ways in which they never even anticipated."

"If the model is the product, then the eval is the product requirement document. And the way that researchers' day-to-day looks is that they'll run dozens of experiments where they'll make small improvements on an eval set."

"I think that for enterprises especially, the core way to think about it is how can they build a test or systematic way to measure how well AI automates their core value chain?"

"I think the fundamental question is, do all jobs just become a single prompt? For example, does a CEO just grow the business while making the customers happy and turning a profit... and it is able to actually be an omniscient presence that can run a whole company."

"The models have gotten so good that the generalists are no longer needed. What they really need is experts, experts across every area that the models are focused on."

"We like to say the only moat in human data is access to an audience. Basically, there are many, many small players in this space... they're basically running TikTok ads... The huge advantage that we've had... is we built a decade of trust with 18 million people."

"When it came to the massive leap forward in LLMs recently, we staffed a team to really prototype quickly, and discover what was possible, and just apply hypotheses outside of the typical norms of how we work. So they went straight to prototyping instead of going through that Double Diamond I was explaining earlier."

"A lot of the value is still going to require a bunch of hustle from a lot of builders to really turn that new source of energy and channel it into something that we humans want to use that solves some of our problems."

"The data flywheel thing is really interesting because the models will get really good at whatever data you show it... being very mindful of the data that you have access to to start your flywheel going and what you can do to keep on going with that flywheel is going to be a critical thing."

"I think that close, tight-knit relationship at any of these large model companies between post training and product is going to produce some really incredible stuff."

"I think that the greatest performers I've ever worked with... preserve the time to explore lots of possibilities... generative AI and AI for all, when it talks to me about just product leaders exploring possibilities, this should expand the surface area."

"I think the big shift that we really felt we would see is moving from kind of this text to text model to an actual autonomous system that can make decisions, that can interact with the real world, that can take in feedback, that can iterate and take multiple steps to solve problems. And now we call that agents, but that was what we were really excited about at the time."

"I think the product experience itself is going to change every single time. And then obviously there, there's all of the practicality of just getting it out there in the world. And so folks obviously need to learn how to use the new technology. There's a lot to do to deploy into all of the messiness of real world software."

"You guys have been using AI for years now. Smart Bidding is AI. All of the recommendations within Google Ads is AI. Ad copy recommendations is AI, and that's always been in the platform."

"In five to 10 years, I think the lines between product managers and engineers and designers are going to blur because AI will enable product managers to build prototypes, to build designs... I'm of the believer that we're just going to have a lot more software."

"The industry is transforming so rapidly that you need to make sure that your product gets better as the LLMs get better. And that too many people are building things to make up and compensate for the LLMs that all that work is going to go away. So it's okay to do it to understand that it's going to go away, but that can't be your differentiator."

"If you thought the PM job was just filling in frameworks, you're going to get replaced by AI."

"We can have GPT-4 read all our docs and answer questions for developers."

"Companies need to figure out what AI native means, not how to append AI at the corners."

"Taste is going to become a distinguishing factor in the age of AI."

"AI isn't a feature of your product. Your product is a feature of AI."