PegaWorld iNspire 2023: Accepting AI at T-Mobile: Driving Adoption to Achieve Business Outcomes

Using next best action, T-Mobile has driven the adoption of AI at scale in assisted channels. Putting the right action in front of its customers and team of experts, T-Mobile has put the customer at the center, enabling the right journey. Learn how next best action has driven adoption of AI – allowing T-Mobile to realize the best experience for its customers and team of experts while driving better first call resolution. T-Mobile showed its team that trusting the action not only leads to better resolutions but also better business outcomes.

Transcript:

- All right, everybody, we are ready to get started. My name's Axel Wells. I'm a industry principal here at Pega in the communications vertical, and I have the pleasure today of introducing two people I've been working with for over four years. I feel like I've been on this journey with T-Mobile as they have rolled out CDH, and I'm really excited about the topic that they have selected today, which is accepting AI. And I think it's gonna be an interesting topic. I look forward to hearing your questions after they get done presenting in the Q&A, and I'll come back up here and step through that with them. With that, we have two great speakers, Brian?

- Good afternoon, everyone. My name is Brian Sprigings. Thank you, Axel. I am the director for AI Strategy for T-Mobile, and I'm joined here of my partner in AI and decisioning,

- Lisa Kravitz. And I report to Brian here on the AI strategy team at T-Mobile.

- And as also mentioned at the start, you know, we've been working together for four years now with Pega as we've gone on our journey, how to deploy AI across our ecosystem. And there's a tremendous amount of things that we could talk about, you know, the learnings, the things we thought, the things we learned as we've gone through, the things we're learning today still. But one area in particular we're really passionate about and a lot of great learnings and some things we didn't even anticipate, you know, four years ago, is around frontline adoption. So how do you get people to use AI?

- Thanks, Brian. So our team's evolution has actually been very closely intertwined with our NBA project and with Pega, Brian and I originally started engaging with Pega on the NBA project when we were still part of the customer care retention strategy team. So our original use case for Next Best Action was all around trying to help our customer care representatives identify customers who were likely to cancel their service or deactivate, and trying to find the appropriate solutions that would help reduce that risk or prevent them from deactivating. And our story today, well, as we evolved with that program and as we expanded the use cases of our Next Best Action project from far beyond retention to include use cases like sales, upsell, features, even including in our action catalog, just conversations, just simple conversations that we thought would be beneficial for the customer, our team evolved alongside it. And now today we are the T-Mobile AI Strategy team.

- Perfect.

- So our story today is really going to be about how at different phases of our journey of developing Next Best Action, we ran into various different adoption challenges, but by building our strategy that really held close to our principles as the Un-carrier, T-Mobile's known as the Un-carrier, by holding to those principles and standards, we were able to build alignment with our upper leadership, with our stakeholders, and with frontline users in order to drive some pretty amazing business outcomes. The overall outline of the story we're gonna tell takes place over the course of a number of years starting back in 2018 when we first engaged as a team with Axel and with Pega. And it was a time of complete transformation for T-Mobile's customer care team. And it was a time when we were addressing adoption challenges regarding how we would match Next Best Action's capabilities to the unique challenges that our customer care agents were having at that time. 2019 was a year of design for Next Best Action and how we were going to implement Customer Decision Hub or CDH and anticipating any adoption challenges, so that we could address them early on. In 2020, we launched a small POC or pilot in our care organization, and that was a time of really listening to a lot of feedback from the frontline in order to address their particular adoption concerns. 2021 was post pilot scaling up to the rest of consumer voice users and trying to avoid compliance measures as we addressed adoption challenges. 2022 was a time of evolution, so you may know that T-Mobile recently merged with Sprint. So this is a time, what we call integration of systems and the adoption challenges that come along with that. And finally, here we are in 2023 building for the future, looking to continue to see these amazing results in the business outcomes, but do it the right way, building in responsible ethics into what we're doing. So again, we begin our story back in 2018, and T-Mobile at that time wasn't known as the network powerhouse that it is today, but it was known for its customer service. T-Mobile employees wearing their magenta gear in the wild regularly get stopped by strangers to tell us how much they love our customer care. It's happened to me at an airport. I know every one of my coworkers has a story of being approached by a complete stranger out in public, just for someone to say, "I've never had to wait more than 30 seconds," or "Your care reps just solved everything immediately, I love T-Mobile because of the customer care." But one of the principles that we hold as the Un-carrier is that we won't stop. So rather than rest on those laurels, our leadership was about to completely transform the underlying organization of customer care into what's called now team of experts. So you'll hear Brian and me talking about our customer service representatives as experts. And what that meant was moving from specialized queues. So something you might know as a retention queue or a collections queue to roles where these experts had to resolve whatever was thrown at them. They had to resolve the entire customer's issues, opportunities, anything that the customer brought up, they had to solve it without transitioning or transferring to another care rep. As you can imagine, that's really customer friendly, really relevant for the customer. It reduces transfers, it builds accountability for the care reps themselves. But it's really difficult to do. If you're trained as a point guard, well, now you have to be a forward, and you have to be a center as well. So this is difficult work for our experts, and we were looking for a solution in NBA that could help make their jobs easier. One of the aspects of their jobs in this new organization was the visual audit, doing a full review of the account management system to look for additional opportunities to service our customers. And this was something that NBA could really help us with, and to address adoption challenges in that particular moment, as we were building out, kind of trying to gain buy-in from operational leaders and frontline users, we told them that we could really help them optimize the investment of time that they were making in this visual audit, by making sure that it was time well spent. Next Best Action would help them find those elevated opportunities. So rather than having to scan the entire court for an opportunity with a customer, NBA would give them the assist. I'm gonna throw the ball over to Brian to talk about what we saw in our handle time because of that strategy.

- And what we're doing here is we're kind of weaving through some of the key metrics as we go through, even though we weren't launched at this time. We're bringing forward a little bit of what we saw from a a handle time perspective as part of the story. And the big thing, again, as Lisa was mentioning, and people are probably very familiar with, like throw the spaghetti on the wall terminology. You know, there's a huge expectation that our experts or anyone in customer service could be addressed, everything the customer needs. We want them to bridge and ask, "Is there anything else they can help you with?" And there's a huge investment of time in there. And we do that for a couple reasons. We try to prevent the customer from calling back. We also do it to ensure that we're maximizing revenue opportunity or the save opportunity with the customer, but it's not necessarily very effective or efficient to throw spaghetti against the wall all the time. So that's where NBA really came, and this is what we shared with our operational leaders is this here is gonna help you take the spaghetti outta the kitchen. You don't need to throw on the wall anymore. We're gonna use data and analytics to really supercharge that experience and let the expert know exactly what they should be bridging to, not just throw something out there or I got an incentive to chase. We were trying to put the customer in front of that. And when we did that, as you can see from the numbers, and we've got to split into two parts, our treatment versus control, which I'm sure everyone here is very familiar with. And then what we do is we do a subset of that treatment, which is an accept. So the customers actually said yes to that. So what we saw here is that our treatment versus control only saw a 7% increase. So even though we were asking our expert to do a lot more, we were able to take some of that spaghetti throwing away and get into more efficient areas where customers were likely to say yes. So we were investing that time well, and as you can see, it was all driven from those conversations with our accepts. We saw a lot of that activity. We're able to approach the customer more right size to what's right for them and invest that time in the right spot. And we saw, again, a slight increase, but it unlocked a tremendous amount of incrementality for us.

- In the next stage of our journey, we were focused on the design and the development, the implementation of NBA and CDH within the Un-carrier. And it was a time of kind of shifting strategies more to anticipating what challenges we might possibly encounter from our users and from the frontline leaders. One of the Un-carrier principles that we built into our strategy at that time was that we believed that technology should be in the service of the people. And this for us meant a really balanced approach to the way we communicated the potential of NBA. So our business case was primarily built upon saves and sales. So reducing deactivations and selling new lines. But would we be happy, we asked ourselves, if we reached our sales goal, but demolished net promoter score? No. Would we be happy if we reached our saves goal, but it cost us our entire CNA budget? No, that wouldn't be very balanced for the customer and our frontline users. So what we did was we looked to start anticipating tracking more than just that the original business case metric, not just our revenue metrics, but also potential costs and potential customer experience metrics as well, like CSAT, like NPS, and on the cost side, credits and adjustments and handle time, like Brian was already talking about. And what that did was it helped us build trust with our stakeholders across the organization. It showed them that we were thinking about not only what was important to us for our particular project, but what was important to them. Now, the challenge with that is what's important to them changed a lot. Our business case stayed the same, but what was important to the frontline organization, our regional field operations offices might change from region to region or from site to site or from month to month. But having a very broad perspective of all the impacts that we were having by implementing NBA allowed us to be really honest about where the investments were being made, like with handle time and where the benefits would be realized. And again, that helped us build a lot of trust and helped us to respond to whatever the flavor of the week might be.

- So as Lisa was mentioning, you know, as you know, typically you're using one metric to drive your adoption or your adaptive models and we need to take a balanced approach. Again, these KPIs are relevant to us. So we also knew that if you're doing what's right and putting the customer at the center of the experience, you know, all the metrics should also be very positive. And we saw that with our net promoter score as well. So again, treatment and control, taking a look at, you know, it didn't matter if we had that conversation in our treatment group, we saw a slight improvement. But again, the more dramatic improvement is when they were engaged in a conversation. So it wasn't just an accept, it was opening up what Lisa would like to call a can of care, just saying to the customer, "Hey, I've noticed something, let's have a conversation about that." And a lot of times customer was very acceptant to that and they're like excited to hear what it was. And other times they're saying, you know, "Thank you, but now is not the right time." That's great input for the models as well. But again, we saw a tremendous, you know, response from our customers that we're welcoming this type. We even had one of our customers mention to our agents, "You're like my doctor, you know what I need before I need it." And so we were building that confidence with our customers, with our frontline that we were bringing that to the forefront, that right experience and our net promoter score really showed the, not just maintaining it, but how we're able to build and grow it.

- Absolutely, now gaining buy-in, like we just talked about with our operational leadership really works from a top down approach. But what about the frontline users who are going to be actually using NBA throughout their calls every day? We were introducing a completely new strategy for them to implement. They were used to listening to what the customer was telling them and responding directly to that request. But now we were asking them to consider something that the customer isn't telling them, potentially something the customer doesn't even know themselves or something that the AI could identify that the humans could never have figured out due to the wealth of data it would require or the calculations necessary, and respond to that as well. And we already talked about how that would require some significant investments. So, how did we address this adoption challenge when we knew we would be asking them to do more and invest as well? One of the Un-carrier principles we like to hold to is that we take smart risks. So what we explained to the frontline users at this time in training materials and in conversations was that we were optimizing this AI on customer relevance, not on profit margin, not on anything else, but rather we were putting first the customer, that's what we do as the Un-carrier. And by doing so, it would mean that they as frontline would have better conversations, better experience in their own job. This would just be more happier customers, deeper relationships. They would have a better experience themselves. And that's how you turn a risk into a reward. Now, they did tell us very early on, they had a concern about opening a can of worms saying, well, what if NBA tells me about some dissatisfier that the customer didn't even mention? Yes, yeah, that definitely happens. That happens a lot of times. But what we explained was this is the right time to talk to them about it when we already have them on the phone today, rather than having them need to call back later. If we can proactively address that issue while we have them on the line, they'll see that we're looking out for their best interest. And again, that'll deepen the relationship with them. Another concern that they have was, well, maybe I won't know what to do. Maybe I won't know how to address the scenario that NBA suggests to me. And so when implementing, or when designing our particular implementation of CDH and NBA, we actually built in some suggested wording for them to see. Now the Un-carrier doesn't believe in scripting our customer care, but the suggested wording really helped them transition from the call reason into this additional proactive conversation. And that's how he explained, you turn a can of worms into, as Brian said, my favorite phrase, the can of care. Now, this was working, as we promised, we would see a difference in our first call resolution. Ryan, what'd we see?

- And we saw a difference in our first call resolution. So our first call resolution is a seven day metric. So did that customer, that band call back in the next seven days? It's one of the indications whether we were able to get in front of the reason for the call the first time. And since we were able to start bridging to some of that proactive space that Lisa was mentioning, we're able to fast forward or bring forward those additional calls and we were able to have a reduction in that type of effort our customers were experiencing, those things they should have asked or forgot to ask, or were gonna call back in another time. So in our response, again, those are our exception declines, we saw 3.3 base points, percentage points improvement. So NBA was helping us bring those needs forward and reducing that effort from our customers. They didn't have to call in as much, even though some of our transactions are sales generated and other activities, we were still able overall to remove some of that effort from our frontline.

- Yep, now that results among many others from the pilot, really impressed the VPs of operations. And we hit this important milestone, which was the green light from those leaders to expand to all of consumer voice. But again, this just meant an evolution of our adoption challenges. And here was a new one where our pilot participants had been volunteers. They told us they were really excited about this technology, and they had helped us throughout the design and development phase, giving us feedback, helping to build the tool essentially. And now we were going to scale across thousands and thousands more users who weren't necessarily that excited about the novel new tool. And we needed to address that in some way. So one of the things we did was we came back to that promise of relevance and we say, you know, Un-carrier does not demand compliance, rather it provides value. And one of the things we did to live up to that was we expanded our catalog. So I mentioned how originally our business case was built on saves and sales, but at this point we started to think a lot more about what else could we include in our action taxonomy to make sure we always had something really relevant for the customer and have a good experience for the frontline call representative and expert as well. So we built beyond sales, beyond retention to include upsell types of actions and feature type of actions, and even just conversations, conversations like talking about how your device experience has been lately or just talking about, just probing into how has your network experience been lately. And as Brian mentioned, we saw a lot of great results from that, just because it meant that we were continuing to have investments in them in order to build trust and have the most relevant conversations between them and their customers. One of the other requests that the VPs made of us was that we continue to invest in engagement with the frontline. Meaning that we would continue to have conversations with their coaches and their managers to ensure that we were gathering feedback from them and make sure that we were continuing to share the exciting results. So we didn't have the resources on our little scrappy team yet to do so. So we started engaging with some of the others from our larger customer care strategy group in order to build a team that would have a regular cadence of conversations with frontline leaders at each site to ensure that we were sharing that two-way feedback. And again, that built a lot of accountability and trust within the teams. We even created a new metric just for them that would help them understand their adoption and their utilization rate.

- So the discussion rate, so this is really ensuring that they weren't just looking at, you know, the action that was available, but they're actually getting into that conversation with the customers. That's the intent of it. And as Lisa was mentioning, kind of like the bell curve with you've got your early adopters and trying to bring everybody else into the mix, a lot of people are afraid of AI. It's a really scary, you know, topic not just for our frontline, but for our leaders. And I would think there's, you know, maybe this is the wrong audience to feel this way, but I'm sure everybody here has faced those people, the skeptics or it's gonna take my job. It's replacing me. As we mentioned, we empowered our frontline. This is team of experts. There's no other cue for you to send a customer, you own that relationship. So we were really focused and now we're saying, hey, now follow this bouncing ball. We're gonna tell you some other things. So it was really tricky trying to navigate through that empowerment and show how AI can help, you know, accelerate that experience that they're trying to drive. So bringing in this discussion rate was really more clear on what the desired outcome we have from the frontline. Again, we weren't trying to get just accepts, we wanted 'em to get into a conversation, 'cause we knew that conversation would drive that overall value. And we saw some, what we saw a bit of a step back initially, I think this is where as we were integrating with, you know, this long expansion to these channels and these groups that haven't used it before, trying to get them all in to try to believe in this stuff, it took a lot of time to get them there. And through that time, you know, sometimes you have to take a step back as you're bringing all these new users on, you're trying to understand, again, keeping a very strong relationship with the frontline, with their leaders, understanding what some of those objections or roadblocks were, allow us to continue to evolve, our talking points, the areas we focused on, and showing the evidence that what happens when you do engage here and trust the machine, great things happen. And again, we took that step back, did that reinvest and we were able to not only get back to where we were, but able to grow that discussion rate with our new users coming on board.

- Absolutely. Last year, 2022 was a big moment for the Sprint and T-Mobile integration. It meant some pretty big changes in our technology systems. It meant changes in our leadership, it meant changes in the priorities, for us, what we were trying to drive. But one of the Un-carrier principles, we say we're one team together, we try not to work in silos. We say we're a team of bridges working across left to right. So never before had it been so clear that we needed to have some underpinning structures of change management. And so the first thing we did with Pega's help to a large extent, was establish our governance operating framework. And maybe some of you also have this framework to help your project, but it's a central strategy forum with offshoots, so a VP governance board, channel execution forums, working sessions. But it really allowed us to find out about big changes that were about to occur in new ways that we hadn't been able to before and prepare for them. From a frontline perspective, we knew we also needed a way to scale feedback from them as well. Within the reorganization, we had lost the resources that the frontline engagement team that I just told you about. That was gone, so what we did was we established, we built in a Jira based ticketing system into the NBA widget within our account management system. So that allowed experts to give us feedback right in the moment and capture all of that interaction information, so that we could go back, look at the call, look at the screen capture and see exactly what had happened and fix it. That allowed the frontline to give us feedback, to give us suggestions for new actions they thought should be in our catalog. There were even times where the frontline realizes there's an engagement rule that was broken that they find out before we do, and they figure it out. There's something just not quite right here. And they gave us that feedback and establishing those listening systems helped an enormous amount in order to manage all of this change that was occurring.

- And as we, and what happened here, as we were bringing Sprint in this integration, all the other change management we were dealing with, we recognized that our forums were not necessarily connected with all the right groups. So as we were starting to build and scale out kind of our, you know, our Next Best Action, the business is still running the business. There's BAU happening, marketing's still doing marketing doing, finance still doing what they're doing, everyone's still doing it. So as teams are making changes to their process and their programs, they inversely impacted us. So as you could see, we ended up going down from a call volume perspective, our NBA presentations. And we had to make those adjustments incorporated into our governance forum to allow us to have more reach to those outside groups, so that way they understand how we were using it and how their system interacted with us. And when we were able to do that, we were able to again, identify the opportunity, fix it, and move forward. And you know, the governance framework we have is really the lifeblood of what we do. It brings in that feedback from frontline. It brings our data scientists, it brings our strategy team, marketing, brings everybody together. And what we're trying to do here is we're trying to help everybody evolve their job as we're deploying AI, you know, AI is gonna help us. It's not necessarily gonna replace everyone's job, but we have to do things differently and leverage AI in order as a marketer to reach out to customers. So by reaching out and bringing out these other groups within our ecosystem, we now have those, you know, those alerts and those relationships to ensure that we're making changes involving our business, that NBA is part of that journey.

- Absolutely, and now here we are in 2023, looking to the future. And now the point we're at in our NBA journey is that we're looking to drive adoption at the enterprise level, not just with the frontline or our internal stakeholder holders, but really everyone across the enterprise at T-Mobile, at the Un-carrier. One of the big shifts Brian and I have seen is that maybe even just as recently as six months ago, Brian and I were constantly on a bit of a road show trying to explain the amazing power of Next Best Action, the exciting things that could unlock across our entire business. Whereas kind of post the ChatGPT news, it's part of the zeitgeist, and now people are really actively coming to us and asking how they can become part of our AI ecosystem. So that's exciting. One of the things that we did recently and this year was we built some of our NBA KPIs into what's called the QGP, the quarterly game plan. And by doing so, it just continues to drive accountability with those frontline, with the operational leadership. We're not in as close contact with them as our program has really scaled up, but it means that pressure remains on them as they report up to their leaders about the adoption metrics that are in that report, like discussion rate. At the same time, as proud as we are of our results, one of the other Un-carrier principles is that we do it the right way, always. So as part of our governance, we've built out an AI center of excellence and as part of that center of excellence, we've built a new responsible AI forum, part of this larger ecosystem that Brian was talking about through governance. And that allows us to bring even more stakeholders in from legal, from cybersecurity, procurement even, folks from all across the organization that we hadn't worked with before to build new partnerships and talk about how we keep a very humanistic, central approach to the way we apply AI. And I know our title was all about the driving the business outcomes, and I said our business case was all based on saves and sales. So we'll finally get down to the bottom line and get to those metrics now.

- Yeah, before I talk about the, you know, metrics, just add a little bit to what that Lisa's mentioning, that evolution, the journey we've been on is getting those stakeholders in and trusting the technology, helping them see what it does, making sure that they're part of it. This isn't a black box, this isn't a silo. We welcome all the teams to work with us directly. We share our information, everything's transparency. This is a journey that isn't Lisa, Brian and you know, our team with what we're on. This is a journey that we are on as a company. This is everybody at T-Mobile, trying to put the customer at the center of what we do. Trying to leverage, you know, data and analytics to, you know, supercharge these experiences. Know what customer is having, you know, a network outage before they even know it. We may be in bed, we know the tower went down, and knowing is that customer gonna get picked up by another tower or is that an issue? Do we need to get in front and let you know, or is there something more important for you to do, right? So this technology is really helping us, this approach to really unlock that, again, put the customer at the center of that and you know, this is still a journey, so we're gonna get there, but as you can see from what we're showing here, the results were fantastic. We saw our treatment control groups, 41 points higher, we saw our survival, we were just, you know, entrenching that relationship with customer, bringing those issues forward, they're calling about a bill, and we fixed a network issue with them. And you saw that again, they were staying longer, they were buying more, our frontlines believing, you know, and all our folks within the company believing that this solution, this approach can really drive that customer experience to the next level. And for us, this is really kind of Un-carrier 2.0. Originally when we went Un-carrier, it was all about putting the customer at the center and what happened? Amazing things happened, profits went up, growth went up, everything went much better when you focus on that customer. And AI has allowed us to continue to put that customer in focus, but allow teams to have control and understanding and influence around what we're trying to do. And that partnership and relationship has been nothing but stellar performance and results that we've seen across the board.

- And I would note you might notice that the scale of these results is somewhat different than the previous slides. We were previously talking about percentage points and here we're talking about basis points. What's notable though is a few different things. With sales rate, it's the enormous scale at which we've deployed Next Best Action. So even though the incrementality is, you know, in the basis point measurement, the scale, the number of interactions that are running through Next Best Action every day, week, month is what makes this really remarkable and quite impactful. And then in terms of survival rate and the saves that we're seeing, what you may or may not know about T-Mobile is that we have the lowest churn in the business, and it's dropping. So to be able to realize improvements on that, of any amount is it's really difficult to do, and it really catches people's eyes. So this again, is really impactful to our business overall.

- You don't have to go into those, you know, segments. I know if there's any marketers here, you love your segments, who doesn't love a good segment? So, you know, NBA kind of helps us get away from those, you know, those big segments. It really looks at right now, what is right for that customer? And we bring that forward to that customer. So yesterday might have been in the ABC segment, maybe tomorrow they're in the XYZ segment. But right now what's important to 'em, what's relevant, what are we trying to accelerate in that relationship? And by focusing on that side, it's just, it's just been, you know, an excellent story from a results perspective. And now the trick that we have is how do we continue to expand this across, you know, the enterprise? How do we get our other channels lit up? We've got some real estate and digital, we're in the process deploying our automated bots. We've got retail, we've got all these different touch points we have, some, you know, with human beings, other with technology that are assisting the customer. Some cases, getting in front of it with preemptive measures or proactive measures or reacting at that time and really having a solution, an action that's right for every customer at that time. So we can really get that specific and that granularity to that and get away from everybody who wears a magenta shirt gets this offer, everybody's wearing a yellow shirt, get that offer. We just say, you know, "What's right for you? What is the data telling us? What's that eligibility?" So that way, we can do what's right for the customer.

- Absolutely, and T-Mobile intends to continue to be a leader in this space. We're really excited about the potential for continued potential for the predictive AI and helping identify the best opportunities for our customers. The adaptive models powering the AI to match up those right solutions at that very moment for our customers. The new generative AI applications, I'm sure we're all looking forward to. And you know, not everyone is the Un-carrier. We're unique in that sense. So you won't all have the same principles that I've talked about today or the same brand manifesto, but every one of your companies has your own vision, your own mission that you can build your strategy around to build that allyship, to build stakeholdership with folks on your frontline or within your organization at HQ in order to put you in a really good position to also score that slam dunk.

- All right.

- Well, thank you, Lisa and Brian, that was a fantastic presentation. Now we'll take questions from the audience. If you have a question, come up to one of the mics, and we'll just go through them. So we'll wait a second here, see if there are any questions. Oh yeah, if you can go to the mic, please.

- [Audience Member] My name is from Tech Mahindra. So my question is the, so the data and the transformation that you shared, Brian and Lisa, that's more focused on I would say B2C side, right?

- Correct.

- [Audience Member] And so on the B2B side, what would you do different? Would it be the same roadmap on the retail side? Your thoughts on that?

- That's great, I'll start if you wanna-

- Yeah.

- I'll talk about the B2B and B2C. So I think the business segment's very unique in particular for T-Mobile. We've got our micro customers, the small businesses, they typically act and behave more like a consumer. And then we have our medium large enterprise, we've got our government relationships and others. So I think the micro space, there's a lot of applicability there and there's a lot of talk on how we continue to expand the journey that we have right now for there. But as you go through the various other tiers, some of those relationships are managed a bit differently. So it's about how do you get, you know, those account executives to understand, what's the data saying and help them leverage that so they can get proactive, identify some of those issues that are out there that they may not be able to see. And so that's an area. So we do see a slightly different application for our enterprise, but our micro group should look and feel very similar to what we've done on the consumer side. Anything you wanted to add?

- Yeah, I would say when expanding to a new use case or a new line of business, it's been pretty well like manageable within our existing governance structure when it comes to going to an entirely new channel or some lines of business like as complicated as B2B, it does require even greater investment of that governance system at the very top levels. I think that's what we've learned.

- Yeah.

- That you really have to build strong partnerships at the top, so that the information and the priorities and the funding gets cascaded down properly. Because if you try just working like directly with those who are like the immediate, like the worker bees, I might say, not everything gets cascaded up in the same way. You know, upstream is a lot harder than downstream. I think that's what I would, that's the recommendation I would make. Start as high as you can to get that buy-in and stakeholdership.

- And just to add too, like you are gonna repeat, every time we've expanded, we almost go through our four year journey again, maybe not takes us four years, right? We've got a lot of those learnings. But again, I mentioned earlier, a lot of people are afraid of AI. They're think they're gonna lose their job or they don't understand. I don't know how to hit my quarterly sales number using AI. I know if I send out a million DMs, direct mails, or you know, emails, I'm gonna get X. So I know that formula, but how do I now do this in a, ah, you know, like I just gotta drive traffic to the retail, that's my job. So how do you get them to start transitioning over? So having those conversations over and over again, having some of our talk track very tight and then because we have so much data, we're able to kind of show what that impact is. And again, people love seeing some of those revenue numbers, but it's still those experience numbers are just as impressive, especially for a company like T-Mobile. We put a lot of of effort into our customers into what we do. So, but you expect to kind of go through that process and learn and whatever you think you know, you're probably gonna be a little bit different when you get through it all. So be kind of open and agile to those concerns and needs for your other stakeholders, 'cause you've gotta bring them along in the journey. You can't leave anybody behind or else, you know, it's just gonna take away from the power of what you're trying to do.

- And it helps to build from your strongest parts. So like finding those advocates or champions or evangelists in your current kind of structure or whoever's already involved in your NBA program can really help try to build allyship.

- And sometimes you gotta slow 'em down a bit. 'cause everybody, you know, the whole generative AI thing, everybody's expecting, you know, like flip a switch and we've got generative AI running around, but when you care about your business, your customer, you gotta do things right, and it's a process. So sometimes you gotta help people slow down a little bit to get excited about those numbers and help them understand how they can change and build their strategy around an AI and NBA system.

- Great, any other questions? Yes?

- [Audience Member] Excuse me, hello. Okay, whenever we start talking about AI, I mean from a data science perspective, we get really excited, right? Because of the power and the profits and the margins and the increased performance. But the first thing I start to think about is like the compliance standpoint of like, how do we make sure that even though we think it's the right thing for the customer, that they're getting treated fairly, that they're getting the same opportunities as other people. So how do you get out in front of that, especially with something so new that, you know, I think a lot of regulators are still trying to wrap their brains around as well. I know you're in more telecommunication, but on the lending side, like we are highly regulated there. So how do you get out in front of that and make sure that you're being very proactive in that compliance, like monitoring basically?

- I think it's a great question. And I would say we're not as regulated, but we definitely have a lot of regulations as well in telecommunication space. You know, even recording a call or what number you display, there's a lot of things we have to go through. So one of the forums Lisa mentioned quickly during our presentation was what we call responsible AI. So we bring our lawyers, legal team, our data scientists together, our strategy experts, and we talk about what can we use data for? So can I use a credit class for certain actions, or is there certain use cases we can't, because you just end up using the data you're not supposed to do? So we bring those stakeholders together and we go through it and sometimes we're, if it's a little bit gray, we're not sure, we put that customer thing right in front. So if we're not sure, we're probably not gonna do it. We're gonna err on the side of caution right now. But it's about bringing those stakeholders in and it's also being able to make sure you have the mechanism that you can explain why things are happening, so that you can logically say, well, this customer got this, because of these attributes or these scenarios and this is what it's learning and what it's trying to do. So we're always taking a look at some of those outcomes to understand them to see what we potentially need to do, working with those stakeholders to make sure we're using those attributes in the right way. Making sure we're compliant, legal. There's a lot of concerns around generative, because I think for a lot of people, because it almost can be a bit of a wild, wild west, when you start letting your models kinda kind of go a little bit more aggressive. For us, we are taking a very smart approach and making sure what we put out there, we understand what it does, we can speak to it, and we continue to evolve and add onto that. So we've seen a lot of success by bringing in those stakeholders to help guide us to we're making the right decisions.

- Yeah, we've never engaged with our lawyers more than-

- Oh my goodness.

- Creating this, there's two, right? There's the first part which is the legal concerns and there are tools even built into the platform like ethical bias check that you can leverage to address some of those legally, you know, protected classes and how we use some of those data. But then there's also just beyond the legal questions, just some of the other ethics and how do we bring in our humanity and apply it to our AI tools. Like you think of an axiom that doctors adhere to, like do no harm. AI would never be able to translate that, as humans, we know what that means. But to the AI built off data, like what is due? Is that part of the process or is that the outcome? Is that no, does that have a threshold? Harm, how bad is harm really? So it really requires through these forums, a lot of stakeholders discovery, talking about like discovery, and translation of what we think is important to us as humans into a coded language that the AI and the technology can understand.

- All right, any other questions? Yes?

- [Audience Member] You mentioned that the KPI, the AI was optimizing towards was the seven day resolve rate. And my question is, did you guys already align that with the care team in terms of incentivizing them or did you have to work that out with them?

- It is a great question. We actually did, we set the metric at the time wasn't around FCR, our first call resolution. It was a byproduct. By doing it right, so one of the things we did is we brought every KPI we could think of into the view, we really wanna understand everything, the good and the bad we're doing. Lisa mentioned earlier CNA, which stands for credits and adjustments. So are we, because we're doing these things now, giving customers more credits? Are customers, you know, happier, you know, are they more upset? Are they, you know, deactivating, are they adding more? So we looked across all those metrics to understand not just, you know, what we're getting in an aggregate, but what each action group with different customer groups were actually getting us from an impact perspective. So for us, the big thing was looking at all the KPIs. As Lisa mentioned, we've got a lot of stakeholders in our company and everyone's got their own metric. So you can't run this and destroy somebody else's business. We've gotta be very thoughtful in what we do. And again, a lot of that came by having a big reach to bring as many stakeholders that would talk to us into this conversation, into our forums to make sure we were balancing the decision we were making. But FCR was a great byproduct, but we were not necessarily surprised. It was an expectation. We had a few things, bets that we had, and I think this was one of the bets that we weren't surprised that it didn't get worse, but I think we were surprised at how how much it got, yeah, how well we did on it. And that really, again, got a lot of people at the table, again, fast tracking that and that metric, getting that conversation, that issue up right front really helped us. Great question, thank you.

- Yep, lots of great questions.

Tags