A practical guide to integrating AI in your product
AKA what every software company is working on right now
At Keeper, we leaned heavily into Gen AI integration long before it became an obvious bet. Surrounded by tax goliath incumbents, and squeezed at the margins - it felt like we had no choice.
As early as 2020, we were using LLMs across customer support and some our ML algos. They even featured us in their PR (back when other startups needed to be convinced to integrate AI). In exchange, we got access to their latest models first.
In this post, I will provide a high-level overview of lessons learned about integrating Gen AI into product based on lessons learned over the past 5 years as CEO & head of product of Keeper.
I promise to stay practical and stand squarely only on emperical experience — rather than theorizing. I will not try to cover:
Futuristic speculation beyond simply “Gen AI will keep getting cheaper & faster”
Internal employee productivity improvements (lots has already been written here)
Pros / cons of specific AI infra providers (pick whoever you want)
Overview
Building product with access to Gen AI is a paradigm shift. It requires re-underwriting assumptions about what is and isn’t hard, around how to measure success, and it requires a new set of infrastructure to get right. Here’s how I currently think about it.
Net-new capabilities
Things that weren’t possible are now possible, and some of those things are tremendously valuable to your customers. The biggest categories of net-new capabilities are:
Live assistance. LLMs enable Q&A style assistance that’s not only instant, but with rigorous optimization can outperform the quality of the average account manager / customer support manager. This is bigger than just “automate customer support” — it’s a new category of feature altogether, if designed well. The digital equivalent of having the company’s best customer success manager sitting next to you all the time whenever you’re using the product.
Extract data that used to be very hard to parse. Freeform text, photos, videos, voice, and unstructured documents are now practically just as easy to parse as answers to multiple choice questions. Are there ways this multi-modality could save your customers time? For Keeper this has meant live chat-based assistance, and freeform document dropoff and enabled us to completely re-architect our tax filing flow to match human services.
Instantly generate individually personalized content. Text, images, video, voice — every user now has a personally assigned copy video / video editor / illustrator ready to produce whatever is most helpful at a moment’s notice. At Keeper, we use this to explain user tax bills, send them personalized reminder emails.
Incremental improvements
Besides the flashy stuff, there are also lots of more incremental benefits of AI integration:
Dynamically generate copy. While “GenUI” is overblown hype in my opinion, I do think there’s a place for a more constrained value. Think: personalized cart abandonment emails, churned user re-activation, and easy-to-understand explanations if the user clicks “learn more”.
Replace hard-rules and static triggers. Traditionally, product developers have had to implement very specific rules for when to trigger an endpoint or what suggestions to show when. A lot of that can be transitioned to lightweight gen AI black-box so long as the constraints are clear and there is decent monitoring in place. An example here is determining when to send a cart abandonment email — hard global rules will less effective than a constrained AI that has been given all the necessary context.
Improve ML algorithms. Finally, certain components of your company’s ML algorithms should get re-written. The most common example here (starting with GPT-3) is natural language inference — LLMs are just so much better at parsing and cleaning text. While I don’t expect core algorithms (e.g. Meta feed rank) to get completely replaced by a gen AI blackbox anytime soon, I do think certain steps in the algorithm will get incrementally upgraded. At Keeper, this has impacted our write-off detection algorithm.
But you need good infrastructure
Don’t all those rewards sound nice? Well … the catch is that they require a lot infrastructure to get right. Simply launching a way to use AI chat assistant is not going to “wow” your customers. The bar is already high — they can just go use ChatGPT instead. So, here’s what it takes:
Comprehensive & centralized user profiles. The #1 thing needed to unlock AI intelligence is context. Every company should have a centralized “user profile” object that gets passed into the prompt of any customer interaction. At Keeper, we basically stream all user profile information to a centralized endpoint that then gets summarized periodically for future use. This is the equivalent of having comprehensive “client notes” at a services-oriented org. This profile needs to include:
Interactions the user has had with the product or marketing (product analytics)
Inputs the user has provided to the product (demographics, documents)
Past conversations the user has had with your company representatives (both AI and human)
What the user is currently seeing or doing at the time of interaction
Up-to-date embeddings. Every time a feature gets released, or the company’s stance on a particular question gets updated, there needs to be rigorous systems in place to ensure that embeddings are updated accordingly. A good customer support organization should already have this, but with AI they won’t be able to rely on quite as much “common sense” or undocumented / implied company strategy.
AI monitoring systems. Lots has been written about AI observability and it’s really not rocket science. You basically just need a metrics dashboard, some form of user feedback, a core AI-powered test set, and a periodic human-powered testing process.
AI security & compliance infrastructure. Obviously you’ll need to make sure your use of AI doesn’t violate the privacy policy and that your agent’s capabilities don’t enable backdoor access to other user accounts. Also not rocket science.
Let’s go!
We get to live through the biggest wave of industry change since the advent of internet. Leverage it right and get reward it. Ignore it and become an HBS case study.
I’ll try and update this article as we all (collectively) learn more.