Welcome And Guest Setup

SPEAKER_02 0:00

As Agentic AI moves from impressive demonstrations into the real world, how can companies capture its promise without creating risks they can't manage or defend? We'll find out on this episode of Shift Shapers.

Announcer 0:14

Change either energizes or paralyzes. The choice is yours. This is the Chip Shapers Podcast, bringing the employee benefits industry interviews with individuals and companies who are shaping the industry's shift. And now, here's your host, David Saltzman.

SPEAKER_02 0:34

And our guest today is Karash Hernandez, Chief Operating Officer of TrueAgence. Now, Carash's background is in statistics, predictive modeling, and building systems that support decision making for large corporations. Long before generative AI became the business world's favorite new shiny object, he realized its potential to deliver genuine one-to-one personalization at scale, and that's not easy to do. At TrueAgence, he helped build governance and compliance into the platform from the very beginning, which gives him a particularly useful perspective on what it takes to move from AI into intriguing proof from which gives him a particularly useful perspective on what it takes to move AI from intriguing proof of concept to a trusted customer-facing business tool. Welcome, Karash. Thanks for being here.

SPEAKER_03 1:21

Well, thanks for having me, David.

The Moment AI Got Real

SPEAKER_02 1:23

Before we get into a gentic AI, take us back to the moment when you realized that generative AI was more than just another overhype technology. What did you see that convinced you that something fundamental had changed?

SPEAKER_03 1:37

Right. AI for a long time has been a buzzword that people have used for machine learning. And I would say eight, nine years ago, before the most recent version of the Transformers model came out around 2019, 2020, it was all hype. It wasn't really AI. It was a neural network model or a support vector machine or something. And it was usually predicting a simple outcome. Yes, no, maybe a continuous outcome like revenue generated or whatever. And so I always looked at that quite skeptical whenever somebody used the word AI around me. Because I kind of know how the software is there. But I would say it was probably around 2021. I think they were still on ChatGPT 2 or 2.5 at the time. And I had recently read the paper Attention is All You Need, which was fascinating. And I started paying attention. And so I signed up for an earlier version of ChatGPT. And I decided to ask it detailed interview questions, the sort I would ask people who were coming in to work in the modeling teams that I would run it. And I asked pretty detailed questions about the model building process, compliance, governance. And I get pretty technical. And the answers I was getting back from it were as good as some of the best hires I'd had. And I would say surprisingly good given the technical detail that I was going into. And it was at that point that I realized that this is more than just height. It's more than just another fancy modeling technique. And it was going to have real impacts. It was around that time that I decided to found true agents with my partner. I figured at that point that uh the train was coming and you either needed to get on or get run over.

What Makes AI Agentic

SPEAKER_02 3:26

So for listeners who know the term but maybe don't understand the distinction, what makes AI agentic? And how is an AI agent different from a chatbot or a tool that simply generates content?

SPEAKER_03 3:39

Sure. It's agency. It's in the name, right? So it's the ability to take actions based on what you've done. Chatbots are very good at returning you an answer to your question. They can tell you, for example, hey, my your account balances uh $3,000, but it can't transfer money from one account to another. It can't necessarily send you notification when you've hit a certain balance threshold of your spend has exceeded $1,000 in restaurants. That requires a certain amount of agency. It requires being able to take the information and turn it into an action. And that's the primary difference between the old school chatbots and agents that you see today.

From Hype To ROI Pressure

SPEAKER_02 4:22

Is there a shift from like show me what it can do to show me how you do it? And if there was, what caused that?

SPEAKER_03 4:31

There has been. And I think it's it's part of a normal tech hype cycle, frankly. You know, if you look at the early days of internet tech startups, and it was a shiny object. It was something cool. It was something we hadn't seen before. And everybody was excited about it. But there's a natural maturity side that happens. It goes from that's cool to well, what can it do for me? And I'd say that that shift really started in like 2022, 2023 and has continued on. So there's fewer, hey, we need a pilot just because it says AI and we need to be showing that our company is moving forward and is tech forward. Um, to, well, okay, well, what's the ROI? And what's the return that I'm getting? And how do I make sure that I'm protected? It does something that it's not supposed to do. And I'd say that it's just a natural progression for any tech. And I'd say that because it's moving so fast, one of the interesting things that's happening is we've jumped ahead from, okay, well, what can it do for me? And we've gotten to that place so quickly in this cycle that some of the compliance people and the third-party risk management people are still playing catch-up and they're still trying to figure out how to get their arms around this new technology in a way that protects their companies. And I think that's the next sort of maturity step that we're stepping into now is they're starting to get some of the tools in place and they're just starting to get their minds around how to think about this technology in a way that's operationalizably.

SPEAKER_02 6:02

You know, it's kind of like Moore's Law on Speed.

unknown 6:05

Yeah.

SPEAKER_02 6:06

Yeah, and it's it doesn't seem to show any sign of slowing down. So let's talk about some of the challenges.

Responsible AI In Operations

SPEAKER_02 6:13

When companies say they want responsible AI, what should that mean in practical operational terms? Not as a principle on a poster in the lunchroom, but in a way that a system actually behaves.

SPEAKER_03 6:25

I think that there's two parts to it. There's two things that you need to cover. And it covers two sort of operational disciplines that already exist. And this is one of the things that I think is funny. And I talk with uh compliance people on a regular basis. If I have to, I could get our product implemented. But they're still just now starting to think about this in the right way. Because, yes, there is a model that drives all of this stuff. And so the traditional model risk management frameworks still apply. You still have all of the normal machine learning metrics that get applied to LLMs, complexity and entropy and all those things, those are important and you still need to be tracking this. But at the end of the day, what the compliance and PPRM groups are interested in is not whether it's predicting one token ahead better than the last model. What it's it's focused on is is the output it's producing useful for the user and safe from a compliance and legal risk exposure perspective, right? And those are two fundamentally different questions. And so I tell people that when you think about how these models work and how you're going to put guardrails around it, yes, model risk management is one method and it's necessary, but not sufficient. What you also need to do is you need to think about these things as agents, not just models, agents that communicate with your end customers. And there is a place where that happens in almost every company today, and it even happens in call centers. Call centers are staffed by humans that have interactions with clients on a daily basis. And guess what? Compliance is not 100% in call centers. I spent a fair amount of my time implementing risk management policies at call centers, and we would go and sit with agents and listen to calls and track metrics for each of the agents. In the very best, in the very best customer service organization, compliance is in the low 90%. 91, 92, 93 is really pushing the top end. But agents make mistakes. They say the wrong thing, they look up the wrong policy. That stuff happens all the time. Sometimes they just have a bad day. And so when you think about how you're going to measure these models, yes, have model risk management, yes, there's entropy and perplexity, those types of traditional metrics. But I would say that you also need to think about these things as agents that take actions and communicate with people. And when you do that, you have to look at operational QA. Operational QA is the sort of thing that gets implemented in call centers, where they track compliance rates, they track one call resolution rate, they track handle line, they track all of those things. And there's a direct analog from what human CSRs are doing to what the agents are doing themselves. So if you're going to talk about operationalizing responsible AI, you have to take model risk management and operational QA, OQA, and kind of put them together to get a complete picture and make sure that you built the same sorts of guardrails around your agents that you have around your CFRs. You have tracking and have monitoring and have compliance rates and have even a risk assessment in cases where they've done something that they weren't supposed to. And the same sort of metrics and framework needed to put in place for agents as well.

Human Oversight That Actually Works

SPEAKER_02 9:46

Last year, Sam Altman said that he saw the future of AI, at least in the near term, being the systems and humans working together. So talk a little bit about how maybe business policies, legal requirements, monitoring, all that stuff fit together along with human oversight.

SPEAKER_03 10:02

Yeah. So I think that there is this natural fear that comes with the idea of AI taking over certain parts and aspects of work that's done inside company. And I tell people that AI is not going to replace humans in the near term. What's going to happen is it's going to augment humans, and it is going to act as a productivity tool. So when you're thinking about deploying agentic AI, it's not going to be deployed on its own with no ground rails and the oversight. I'm not sure that anybody is claiming that they are. The question is, is how do humans interface with the systems and the agents in a way that is not duplicative of what and also adding benefits? And I think there's a couple of places. For us, as an example, there is configuration management and version control for the agents, right? As part of responsible AI, you should be tracking every configuration that is in there and version control in them. So every time you release a new one. Somebody's going to have to go in and do that. Somebody's going to have to do some manual review of sampled output, just like we did in call centers. In call centers today, there are thousands of calls that come through for an agent in a given moment. They don't necessarily track every single one. They sample, they review quality, they provide feedback. And the same sort of thing would have to happen here. We will sample from the agent messages and the agent actions. We will then have a human do a manual view, probably also an AI review as well. And then make sure that we have feedback going back into the system. It's never going to be the case, as far as my company is concerned, that humans are completely not bad. I think it's a mistake to try to do so. I don't think the technology is mature enough for that quite yet. And I think that there is also just a little bit of a measure of CYA from TPRM groups and from Google compliance groups to make sure that there is somebody in there watching the system and making sure that it's operating as it's supposed to. And I think that's appropriate. And I think it's necessary. And I think any type of responsible AI framework that doesn't have that in it is whistling past the graveyard.

Why Frameworks Stay Too Vague

SPEAKER_02 12:24

I know there's some organizations, some national organizations have created what they call AI governance frameworks. I guess the question is maybe two-part. Have they stopped short by showing businesses how to operationalize them? Do they go far enough? And if they don't, what's missing between those high-level standards and a system that a company can actually manage?

SPEAKER_03 12:45

Sure. So there have been a few standards that have been put out, notably NIST has put out a framework. ISO organization has put out their own framework. And I think those frameworks are good and good starting points, but they've been left purposely vague. They're very good at creating high-level steps, like there's a monitoring step, then there's a feedback step, then there's a feminine loop somewhere in the framework. But they've stopped short of giving you like operational metrics to track. They stop short of these specific types of reports that you should be creating. And so I think it's largely been left up to individual companies to figure out how to do it. I think that's partially because standard groups are notoriously thorough. And so I don't think that they have had enough time and enough real-world examples to build all that operational metrics out and divide that level of guidance. Also, it's it's also very use case specific. And so a use case for one company is very different than a use case for another. And so for that reason, they have not put that at them. What I think is missing is essentially what I've said. If you look at a lot of the operational tracking that is being done now, it very heavily relies on traditional model risk management. It very much relies on, hey, show me that the perplexity and the entropy is good and stay there. Uh show me that you're not seeing degradations. And it really is focused on more traditional uh machine learning, machine learning uh modeling metrics. So I think that where they're missing is the operational side. They're missing the analog for the way that we manage CSR. And I think that's what we've tried to build in for our system. And I think it is something that once people get it explained to them, it becomes obvious that the discipline that you have in your call centers needs to be deployed for your agents as well.

Metrics That Catch Bad Outputs

SPEAKER_02 14:35

So what should a company actually measure? And how does it know when it's out of bounds?

SPEAKER_03 14:40

So there's a couple of things that I think are important to measure. I think the first thing is what I would call the difference between the perfect output and what you're getting. So this traditionally has been done with golden sets where they'll have, okay, well, here's an input and here's a perfect output. And then they will prepare that to the output if they're getting to the same input when they'll model. And they usually do what's called a cosine similarity. It's like a semantic similarity between the two states. The idea is that the higher that percentage is, the closer those things are semantically. And essentially it's telling you that your agent is producing appropriate output. It's producing what it's supposed to be producing. I think that's an important step, but I think that's not the most important step. The most important step to me is once you have that output, you need to be looking at a set of an approval matrix and saying, listen, there are different things that need to be approved about if the message report ever goes out the door. There are legal and compliance guidelines. These are not nice to hats, these are required and illegal if you don't if you don't have them. There are things like brand guidelines, tone of voice. There's things like just factual accuracy. Does this match the reference material that was provided to the agent? And I think if you look at that accrual matrix, all those different categories of things that have to be approved, you need to look at each message that is produced against that matrix. And then you come to an approval or a decline decision, right? And I think that any system that is going to be deployed in a corporation in the United States is going to have a guardrail system that does that approval or decline decision. Once you have a system that does that, the next question is, well, how often is it wrong? And so what you're going to end up with is you're going to end up with a little like confusion nature where you say, should you know, system approve, system decline? And then based on a sampled review, should it have approved and should it have decline? That's going to give you a type one and type two error rate, right? False negatives and false positives. And in this case, I think a false positive is much worse. Because what you're saying in that case is this thing, this thing was approved when it shouldn't have been approved. From there, you need to do a risk classification. Because not all errors are the same. If it's an error for tone of voice, that's not that big a deal. It's a low classification. That's an error because it said something that was against a regulatory guideline. That's a much bigger issue. So that confusion matrix, the type one, type two error rate, and then in particular the false positive rate. And then everything that triggered a false positive rate needs to be risk-classified, low medium I. Anything that's high should be reviewed very carefully. You need to go look at your configurations. You need to use that as feedback to go in and fix the system. Because there will be errors. The question is number one, how significant was that error? And is it happening more often than under the sort of purely human interface it's replacing? Because the answer in most cases is that there's actually fewer errors than the human system and they're less severe. And that's been that's been borne out in our system over and open open.

Trust Measured Against Human Error

SPEAKER_02 18:05

Should that really be the question that you ask when it comes to trust? I mean, you touched on trust earlier. Is the real question whether AI makes fewer and less serious mistakes than people do? Is that where we need to measure today?

SPEAKER_03 18:16

I believe it is. The truth is that every action that an agent is going to take is already taken today by businesses. It's just usually through manual processes run by Google. It's through a call center. You call in, hey, I want you to transfer funds. I want you to open a new account for me. I want to close my account. These are all specific actions that every business still has to take today, right? You still have to listen to your customers and take actions that they want. The question is, when humans are doing it, what's that compliance rate? What's the noncompliance rate? And when they're non-compliant, how bad is it? And then you compare that to AI. And I think that what people are going to find is particularly at this point in time with models as mature as they are, if you put in appropriate fire rails and put in appropriate monitoring and a feedback system, you're going to very quickly end up in a place where you're making fewer mistakes than humans do. And the mistakes that it is making are tend to be less severe.

SPEAKER_02 19:16

So let's talk about where the rubber meets the road.

Segment Of One Personalization

SPEAKER_02 19:19

You saw AI as a way to finally deliver on that long-promised segment of one. Talk about what segment of one means and how can companies use agentec AI to create a more personal and responsive customer experience without crossing that line we're talking about into inconsistency, manipulation, or total loss of control.

SPEAKER_03 19:37

Yeah. So segment of one was a buzzword. I heard, man, it had to be like early 2000s, 2001, 2000 period. If I remember correctly, it was uh it was Experian, I believe, that was coming in and giving a pitch on that. And the concept was listen, instead of talking to people as large homogenous blocks, you know, a segment of high spenders or a segment of transactors or whatever it is that will talk to each person individually, where they're at, based on the rich data that companies have internally about that. And it was a great idea. Never actually worked out in practice. It was a buzzword, and it invariably ended up being like six to twelve segments of your customer base, and you had six to twelve different templates, right? One for each segment, with some wordsmithing around something that would appeal to that group based on how redefined the segment. Better than nothing, we still saw pretty significant lifts when you go from like just one blank communication per everybody to, you know, six to twelve segments. Still a lift, but certainly not a segment of one that they meant, where they meant each person would be spoken to individually. And you just couldn't do it. It was too complicated. There was too many different splits, people are there's too many different types of people for that to have worked back then. You know? Doesn't mean it wasn't worthwhile. There's an old saying in statistics that I love, which is all models are wrong. Some models are useful. The idea being that the world is too complex to truly model accurately, but is it better than not having to model?

SPEAKER_02 21:16

Well, even when you say segment of one out loud, it sounds kind of aspirational.

SPEAKER_03 21:20

Yeah. Yeah. Although I think the pitch back then was a bit maybe overblown, but that's sale.

Where To Start In Benefits

SPEAKER_02 21:26

For a benefits organization that's experimented with generative AI and now wants to take the next step, what's the most sensible place to begin? And what questions should it ask before allowing an AI agent to interact with customers?

SPEAKER_03 21:39

I would say what is their guardrail system, what are their metrics and tracking around how often it's making a mistake and how it reclassifies those mistakes? Can you get a view of that on a regulated basis? If the answer to that is no, I wouldn't go well with it. I think it's incredibly important from a reputational standpoint to make sure that anytime you deployed an agent, it's staying within the boundaries. And so I think that's non-negotiable. And that's frankly something that I've seen more and more of these pitches that I have. They expect that to be there. And so I would suggest that any company that wants to get into this, that wants to reap the benefits, because we've seen dramatic benefits to doing this, have your framework right. Know how you're going to measure success, know how you're going to measure risk. And then make sure that the companies that you're working with can give you the type of information you need, the visibility, and make sure that the feedback is there so that they can show you how risky it is, they can quantify it, and they can show you how to fix it with a sustenance progress. Any company that wants to do this, minimal. That's what you need.

The Future Regret Leaders Will Share

SPEAKER_02 22:46

As Agentec AI becomes more capable and more common, looking back, what do you believe business leaders will wish they had put in place today?

SPEAKER_03 22:54

I think it it's gonna be it's gonna be the guardrails and tracking, honestly. The benefits are gonna be there. Anybody who's been working in um decision management for any length of time knows that when you go to more detailed segments and you go to more tuned models, you generally get results. For our own clients, we've seen, I mean, on average, it's 100% lift in conversion rates. Then relatively consistent. So the results are there. I think what's going to matter going forward is not whether or not you can put a system in place and whether or not it can produce results. I think that's gonna happen regardless. The question is, is how well are you managing it? How well are you managing it? How well are you optimizing it? And I think the same that's gonna be true of any analytically driven system that you put in place. Will it be better than nothing? Yeah. Yeah, it most likely will. But if you're not managing it appropriately, you're gonna end up with problems. And it's not gonna give you the level of results that you were expected. So I think that will be if people have a big regret, it'll be that they didn't take the time to put in that monitoring, tracking, and feedback system.

SPEAKER_02 24:10

Gee, it almost sounds like in a way we're going back to the future. It sounds a lot like Peter Drucker and J. Edwards Demings and those folks who were talking about operationalizing, but not with intelligence, with machinery.

SPEAKER_03 24:20

Yeah. Classic, like uh, you know, Shiwei control chart. Is this thing staying within the bounds that it was meant to? And uh, there's a reason that Demings is kind of a god of operation. There's a reason his ideas are still around today. They were true then, they're true now. Any system that you put in place is not going to be perfect. And you'd be monitoring it to make sure that it's staying within operational boundaries.

Closing Thoughts And Thanks

SPEAKER_02 24:45

And that's a great place to end our conversation today. Karash Hernandez, co-founder and chief operating officer at TrueAgence. Thanks so much for a fascinating conversation.

SPEAKER_03 24:55

Well, thank you for having me, David.

Announcer 24:58

The Chip Shapers Podcast is a production of Chip Shaper strategies and may not be reproduced or quoted in hold or in part without our express written permission. Copyright 2020, all rights reserved.