PegaWorld iNspire 2024: From Zero to Hero: Transforming NAB’s Customer Experience with the Customer Brain

Right. Thank. Thanks, everybody for coming. I thought we'd have a pretty healthy turnout for this, uh, this session. Um, it's my great pleasure to introduce Jess, who I think probably needs no introduction. I suspect many of you saw her today on Main stage. And what a fantastic job she did. But we've also got, uh, Lisa joining us for this session, uh, who's also part of the team at NAB. Um, they're going to go into a little bit more detail around, uh, what was talked about this morning.

So looking forward to the session and I hope you all enjoy it. Um, over to you. Hi. Thank you very much for joining. Um, I think we'll start off with just a little bit about that. Um, as I mentioned this morning. Do you want to use the. I'll just go old fashioned. Hold on.

Yeah. How about now? Yay! All right, I'll go old fashioned, like a mic might bust out into song with this. Um, so, as I said this morning, I'll tell you a little bit about National Australia Bank, for those of you who don't know us. So we have been around for 160 years, um, servicing personal and business customers. We're Australia's largest business lender. Um, we've got over 10 million different customers and we've got 700 locations across Australia and New Zealand. So, so quite a wide footprint that we, that we serve.

Um, what we wanted to do today was delve a little bit deeper into our journey. So Lisa and I, Lisa was really the brains behind the brain. Um, but our journey of implementing the customer brain at, at NAB And so I started about two years ago, and that was right after we, um, you know, National Australia Bank announced a new position at group exec of digital data and analytics. And I think that's pretty cool. It's pretty differentiating to kind of put those three capabilities together. And it was a real recognition that digital design data really sits at the heart of building great customer experiences. Um, and so that's why this morning, you heard me talk a little, a lot about how data is at the heart of what we're what we're doing at, at NAB. Um, so I wanted to start off a little bit on what we said was the four core building blocks of how we got started. So Lisa's going to talk us through why the music started now.

I was like, oh, um, for you, Lisa. Thanks. So we when we were sort of thinking about what to sort of talk through today, we thought we'd sort of highlight what we think some of the critical things were to to our success. We're pretty proud of what we've been able to do and what we've been able to deliver. And so we thought it might be helpful to sort of share some of those, those insights with you all today. And so when we're sitting back and reflecting on what is it that we think we're really important to us to get right and things that we think we did pretty well, um, are really around four core components. So really about driving sponsorship throughout the business. Um, actually, how do you construct a team from scratch to actually go about working through and building this capability out? Really important.

The technology, of course, and ensuring that you can embed it within your organization in a way that delivers value quickly. And then also we want to talk about the system, but the system that we built specifically to actually bring our use cases to life and actually deliver actions. So Jess, I'm going to throw it back to you to talk a little bit about sponsorship censorship. This was yeah, this was the first key point, wasn't it? So we kind of built sponsorship in two ways. Um, firstly, I was lucky enough to have Lisa join the team really early on and she's got nearly 20 years of experience at NAB, so she had some insider information. And to people who really believed in data, the people who really like got the brain. They didn't understand the ins and outs of how they thought it could work. But they knew that in order to drive better customer experience, you know that this could could help them.

So, um, we went one of the guys who was on the screen, a guy called Andy, he really bought into what we were doing. So for him, we started building up this kind of bottom up momentum. We sat with his business. We said, what are the business problems that you have? Where do your customers struggle and how can we better support them? And so we had 2 or 3 of these very critical people in the organization that we got onside and we started small running like tests and learns with them around how this capability could be integrated into their business. But bottom up is only one half of the story. We also went top down. So, um, I think I joined and our cdao threw me right into the frying pan, and I was on board within the first three months of being in the organization.

And that was really important because we wanted to center the strategy around our overarching bank strategy. Right. It's not something that's off on the side, some new tech innovation that we're just going to to play with. It's something that was critical to what we were doing and who we wanted to be at NAB. Um, so we started off at board and at first we just went in and talked to them around customer experience, personalization, how, you know, um, sentiment is shifting. And people who bank with us expect personalization. And if they don't find it, actually they'll start switching. Um, but what was really interesting is we then went back every quarter, really, we were there giving them updates on how we were progressing and building their their sponsorship slowly. Because it's one thing to talk about a strategy on a piece of paper.

Um, but we then went back when we delivered our first couple of use cases, talking about how customers were responding, what the engagement metrics looked like. Um, up until more recently when, as I said, our stakeholders were there telling, telling their stories and there was a moment with board where I came back and said, Lisa, I think they finally got it. And it was when we had enough kind of experiences live, and we brought through the different journeys that customers were going through. So I think the point there on sponsorship is you've got to build it early on, but then you have to maintain it and keep it and keep them updated as you go, because there are so many possibilities with something like the Customer brain that people get it in theory, but it's hard to really imagine until you've got some, some real tangible examples. Nice. So that was my job for quite a while. But really importantly, also like cell division, then you've got to build the team to actually deliver on that vision. Otherwise we're going to be in even bigger trouble. And where we sit within our organization, within digital data and analytics, we've actually got an enterprise data and analytics function.

So we've got a CDO. And what I think is really unique about this is I sit on a leadership team with the people who run our data platforms, the people who are putting the data onto the platforms, the people who look after data governance. And then all my fellow colleagues who run analytics for the entire entire bank, every single part of the organization is service with with an analytical function. And so what makes our team, I think, quite different is I've got data engineers, I've got data analysts, I've got data scientists. They work hand in gloves with the decisioning team to help frame up the opportunities to help identify those data items that are important. So the business will come to us and say with a problem, and the first thing we'll do is collectively with the decisioning team and our data scientists, we'll go back with what? Here's what, here's what you think the problem is. Here's what we see in the data. And here are some ways that we can respond.

I think secondly, I'll start and maybe Lisa can add some color. But building a decisioning team, particularly in Australia, there's not a lot of people who know Pega. So we were building a team from from scratch. And so we really looked for people who, um, thought in customer context but had a real appreciation for data. And, you know, there's a lot of great data and analytics and marketing people who understand it conceptually and can learn the tooling. And I think that that was really powerful. We weren't going out trying to find the best Pega experts in market. We were going out finding the best people that could help contextualize a business issue. Um, use data and then put together some contact strategies to bring that that to life.

Yeah, it was a really fun part of the journey because even me personally, I was learning Pega as I went on the journey around the implementation here. So I've got a background in sort of marketing, analytics, measurement and martech, but was super excited with the capability. And it's a privilege to work for Jess in terms of actually building this out. I'm going to talk a little bit about the the technology. Um, I was at PegaWorld last year and we were super early on in our journey then. And the first question anyone asked me was like, are you early on, how many channels have you got connected? I'm like, well, we've got all of our channels connected. And I was like, what do you mean? And so I think this is something that's a little bit unique about our journey and what we've chosen to do.

So we have had for quite a while at NAB, um, a reasonably sophisticated marketing automation, um, capability that delivered a whole host of outbound experiences. And these were all connected up through the same technology platform. So I was actually, you know, prior to Pega, able to send outbound messages on a batch basis through to all of these channels, including banker and our mobile app and our internet banking channels. So when you log into our mobile app, there's usually a couple of banners or containers and a mini app that you can actually place notifications in. And they were already all hooked up. And so when we wanted to introduce Pega into that, we had to have a number of conversations from an architecture perspective around what are the boundaries between our marketing automation system and and where does decisioning sit in relation to this. But actually one of the things that we chose to do was actually leverage the outbound capability that we had, and plug Pega straight into that to give us critical mass of channels literally from day dot. And it really helped us service a lot of the use cases that were being asked to deliver from our stakeholders. So they were coming in and they had a business problem.

They always wanted to go multichannel. So this was a really good way for us to go multi-channel really, really quickly. However, we've also noted that inbound is absolutely critical to a decisioning practice and we needed to actually uplift channels, inbound channels where we felt that was really important. And the first place that we focused on doing that was in the mobile and the internet banking space. And so we converted that to to real time back in December last year. We're doing some work now to add inbound push, and we're also doing some work now to that banker channel actually encapsulates our retail bankers. Our call center actually connects through to our business banking team. Um, so we're really thinking about for our different banker suites, where does real time become important there. And that's that's part of our roadmap going forward that we're thinking about to.

Yeah. In real time. So important for you know, you bought a real time, you know, with Pega it's a real time interaction management system. And sometimes, you know, it took the marketers a little bit of time to get their head around that. But why can't an outbound work? We're making a decision last night and I'm like, well, I don't want a decision from last night. I want a decision right now, in the moment for a customer. And that's why it was so important to us to uplift, you know, we uplift all of our inbound channels so that they are real time, whether that's, as Lisa was saying, mobile and internet banking or in the banker channels, because we don't want the customers to be doing something online and then go into the branch and you know, that staff member is unaware of that context. Now, that being said, all the outbound channels like it's great to have really beautiful channel experiences and that's what's managed through some of our outbound channels.

But that real time interaction management did take us a little while to get people's people's heads around. But I think it's an important distinction because it's a it's just that little bit margin above on your customer experience. And I can't talk specific results. But since we've actually converted a lot of our outbound to inbound, we've certainly seen a big uptick in engagement there as well. 100%. The other thing that I think is really and Jess touched on it when she talked about the team and the different skill sets that we have in the team. One of the other things that we were able to do and get momentum on quickly was actually access to data. Um, we had data that actually already fed our outbound marketing capability. We actually used a lot of that data to actually drive some of the base data that we used in our initial x-cars, but also having the data engineers and data scientists and data analysts sitting end to end.

We actually also really quickly built and constructed a pattern to get new data for new use cases into CDH really quickly as well. We usually run monthly releases of new data flowing in, depending on what the use cases are that we're looking at delivering, and we've just got a really clear, established pattern and way of working around that. And I think that's just really enabled us to to move smoothly. And also then when customers come to us with new use cases, we can articulate the things that need to be done. um, to actually get something. Get something up and running. Um, so we also wanted to talk through I like blocks of four. I don't know why, but I do. So bear with me.

But there are four things around actually delivering the system for for use cases that we thought would be useful to to share with you today. Um, and we'll touch on, on each of these in, in detail. Um, when we were starting to think about actually building and bringing use cases to life. The first thing we wanted to do was make sure we nail our frameworks. I had many conversations with Jessica. Make sure you get your business issues and groups right. And so we had a lot of conversations around how do we want to set those up. And actually then we were really transparent with the business around this is what we're considering. We went to a number of exec sort of groups across marketing and digital to get sort of feedback and endorsement of the approach we're actually taking, and those frameworks are actually being used across other pieces of our marketing technology toolset that we use today as well.

So they've actually been taken and and moved forward into other spaces. But the other thing with frameworks that we did do a lot of thinking about upfront were what are the standard pieces of data that we actually want to use across many of our actions? Um, and so, you know, simple things around making sure we're talking to active customers, we've got the right consents, we're thinking about the right credit risk exclusions for a lot of our lending actions. And we stripped back and identified sort of standard sets of data that we would actually want to use when we're building actions out. And we're, again, really transparent with the business around what they were. Um, I, I've been in marketing for a long time, and I was, um, reflecting on, um, going into a conversation with a group of executives on what standard exclusions were and what they would not find this interesting. Surely not. But it was one of the most animated discussions because they were just genuinely interested in what it is. How are we using data and just really appreciated?

I think the transparency that we we demonstrated to them with what we were doing and that sort of links to this theme of governance. We set up sort of two core forums to help govern what we were doing around the build out of of the Customer brain. The first is a customer decisioning forum, which is a forum that actually governs how we're using that standard set of data. And that's something that that runs a couple of times a year. Or if something changes, we go back out to that group. But we also set up a customer council, which consists of a number of execs across both digital and marketing and across data and analytics to actually, um, manage the customer experience. Um, so customer council has visibility around, um, what goes into customer brain decisions we're making when we start talking through sort of what we're doing with, um, weightings, we engage that council to make sure that we've got the right things around the customer experience and that everyone's comfortable with what we're signed up for. Yeah, like I think everybody's going to have an opinion, right? So that's what you're going to need to manage with this.

And I think customer council does a good job of bringing it back to our overarching strategy as a bank. You know, who who are our customers? How best do we serve them. So I think it's a really, really important step. The next one and we've touched on this a little bit, but we we when we actually started to work through how do we pick the use cases to start with once we've implemented our production sort of instance of, of Pega, where do you start? Where do you choose? And this is where what Jess talked through with sponsorship and what you saw Andy speak about on the video at the at the keynote. Um, we to look for areas to start with. We were looking for two things.

We were looking for business stakeholders that were really willing to jump on board when you're testing and building and developing actions with a new capability, and we were also looking for areas where we could demonstrate impact. So we chose to really focus on two key areas to start with. One was around home lending, and we chose home lending because we've got a great stakeholder in Andy and his team. Um, it also from a home lending perspective, it connects through to our banker network. So we thought that was really important to actually help build excitement and sponsorship at that level. Um, but also we are focusing in on that. That home lending space also enabled us to really start to tailor actions for that set of customers where you actually had overlapping actions, which of course, is incredibly important when you're talking about a decisioning practice. Um, the second area we chose to focus was around our mobile app. So I sort of touched on this before.

When you log into knobs mobile app, you basically on your account, you get your account summary page, and there's a banner ad at the top and the bottom and a whole notifications mini app. We get 10 billion customers logging into our mobile app every single year. And so that's a huge opportunity for us to be able to connect with customers with something that is relevant across a sales service or engagement type messages. So the home lending use case has helped us build depth within a particular domain discipline, but actually the choosing to focus on that digital and mobile app space actually enabled us to have breadth and also start to use cases across all of our digital active base, which is a very, very large proportion of our customer base. And if I think around home lending, that use case where we have that advocate, they had ideas around what they thought we should be putting in and we put those things in. We workshopped them together. But what's interesting is now that that's one of our most mature portfolios, we can use Pega Pega, Journey optimizer, and we can actually see where customers are falling out, where there's gaps in our perceived experience, what we think they're going to do. We can actually see what they are actually doing and plug those gaps with new contact strategies, which I think was really groundbreaking for our stakeholders, because you're going back to them proactively and showing them where gaps, where they have in the customer experience. The last piece here that we think was really critical in terms of our delivery system for use cases was ways of working.

I've kind of put that photo up there a little bit deliberately, because our poor marketers have just been through a couple of other, um, migrations of other legacy technology systems. And we came along and I think they're a bit like, oh my gosh, what's going on here? How do we actually deal with and navigate this space? And I honestly think I probably spend about 50% of my time in the first 6 to 8 months, actually, with the marketing team really working through, how do we re-engineer our ways of working so we can actually ensure that we make the most of CDH and 1 to 1 customer engagement? A lot of that was just education in the first instance around what is the difference between what we've had. Like what is Decisioning Lead? Why is this important? How does it change the customer experience? And spending a lot of time actually just working through that and demonstrating use cases.

Um, and then we sort of really got under the hood with the marketers as well. So I've spent hours talking about contact policies, engagement policies, explaining everything that we're actually doing and demonstrating the shift around what we had when we had a very heavy outbound, um, driven capability in terms of communications through to what we were trying to do and deliver with the 1 to 1, um, with 1 to 1 decisioning. Um, we effectively work through a re-engineering of our ways of working and our entire take to market process. So in terms of boundaries, um, my team's responsible for everything decision related, and then I pass the decision outcome through to another team in the value chain that actually then effectively brings those experiences, words, pictures to life in terms of what customers actually then consume. And that requires a value chain. Of people to actually bring to life. So we have the. Lisa's being very humble, like actually helping explain the shift of 1 to 1. Like instead of starting with a campaign concept of I want to send out X number to X customers on X date, starting with a.

Well, actually we're going to start with a single customer. What is the right thing for that customer at that given moment? It's a massive mindset shift to the marketers. And so I think by lifting the lid on how decisioning works, they didn't see it as a blocker to be able to get, you know, their stakeholder, what their stakeholder needs they were able to actually articulate to the stakeholders. Hey, guess what? You're going to get a better outcome over time because we're going to be communicating to customers at the time that they have a need, not at the time that you want to send a communication. But I think you're being very humble because you're underestimating the amount of effort. She's not kidding. When she spends 50% of her time doing that and 50% of her time delivering.

I don't know how you sleep. Um, but yes, that's a that's a massive culture shift. But it's really important when you crack that. That's when that that powerhouse of data analytics, decisioning and marketing come together to really create good outcomes for customers. And that's where finding focus and ways of working almost come together, like we actually work, particularly in the home lending space. We really tested the new ways of working within that environment and that particular marketing squad to start with. Then you create some champions and sponsors as you start to roll it out to other squads, because people have actually experienced something positive as they've actually gone through the shift as well. And that was really important. And now everyone's banging down your door.

That's the problem. Um, so I thought it would be, um, actually, yeah. I've got my slides mocked up. We're going to talk just very quickly. We thought it would be very, um, useful just to share the timeline, I guess, around what we're doing. We are moving at a rapid pace and it's something that we're really proud of, but we move at a rapid pace with two considerations. It's not just about getting actions out the door. It's really important for us to actually be building capability at the same time. So you can see at the top of the slide here, we released our first action in April 2023.

And what a year later we've got 121. The guys are in a release yesterday. We're now at 150. So we are building actions at pace, which is awesome. And we're now building actions across all the domains. Um, so actually after we went from home Lending and Digi, we very quickly moved into the business banking space, credit cards, etc.. Um, and under the bottom there you'll see some of the capability drops that we did. So we are scaling using 1 to 1 ops manager. Um, we use it every day.

My team are always on it. We love it. It's great. Right. Um, and it's a really effective way for us to scale, um, the, the mobile on Ibe I talked about it a little bit before, and we've just gone through an Infinity 23 upgrade. Just touched on journeys as well. Like we want to make the most out of this product. And we're really, actually really excited about what journeys can unlock as a capability. Helps us rethink almost from a campaign mindset through to a bit more of a journey based connection as we're thinking about actions as well.

But we also will be looking at exploring other features within CDH as well as we go forward into our journey as well. I'm most excited about Value Finder because it helps you find underserved pockets of customers, and I find that a lot of times we talk about, um, the types of customers and when we shouldn't be talking to them, we don't talk about customers that we should be talking to that aren't. So I really like that kind of concept that's coming as well. Um. We thought we'd share a little bit about some of the actions that we've, um, launched just to demonstrate, actually the breadth of what we've been doing. Um, so I'll just touch on this briefly. Um, but the first one is around business banking use case. So business banking actually historically hadn't had a lot of traditional outbound marketing. And so they really stepped in and embraced this, this space.

It's a very simple use case. We're following our customers who have abandoned a form. Um, but we hadn't had access to that digital data easily in the past. We fed that through into the brain. And we delivered seven use cases for business banking that are driving really strong conversion, and we're encouraging customers to come back. So, um, that's been really great in terms of actually generating momentum within the business banking space for some of the things that we're actually able to do and deliver through the brain. And then from a customer experience, it's great because instead of the business having to call their banker or having to pick up the phone and go, oh, I just needed a little bit of help on this one little part of the form. The bankers can proactively reach out to the customers and go, hey, what's happened? How can I help you?

And I think that 1 to 1 relationship is really important. One of the debates we always tend to have is what actions go in the brain and what doesn't. And I think this customer data refresh is a really interesting use case, and perhaps a nontraditional one in terms of something that might go into CDH. So it's a regulatory requirement in Australia that we have where we need to actually keep customers details up to date. And it's a really interesting use case because I need to get 100% response rate to actually meet the regulatory requirements. So we're actually using Pega to drive up a move for customers to update digitally. And we're doing that through a multi-channel approach, and we're using Pega to help us decision on what's the right channel treatment strategy and the right communication strategy to actually try and drive that conversion through. And the one I. The thing I love about this one is Josh will always say, like, I'm not helping somebody get their home, like their first home.

I'm actually asking somebody to keep something up to date. It's not the best experience, but what we can do with the brain is connect with customers. When they're already thinking about their finances. They're already in the app doing things. It's not making them, trying to force them to do something outside of their own kind of day to day. So I think it's a really nice, seamless experience. That one. Um, the, the next one we wanted to talk through was around our home lending banker lead. So it's just sort of touched on we've done a whole host of work with our bankers, and we've done a whole host of work with our analytics team around how do we actually, um, drive increased value, uh, with the leads that we send for our bankers to outbound call our customers.

So we've got a really large workforce who are outbound calling customers, connecting with them. And we want to make sure that that call is as valuable as possible. So we actually re-engineered a whole host of new use cases off the analytics work that we had done. We did a lot where we looked at triggers in our data and how they could actually drive value. Um, and that's led to sort of the numbers that you see here. So 50% more conversions and three times more opportunities have actually been delivered off the back of of the use cases. There were other shifts that came just in terms of how we actually delivered, um, these leads into bankers before. So, so before the brain bankers were getting leads on a weekly basis and sometimes monthly for some of our triggers. But actually what we were able to do through the brain is we're delivering them leads on a daily basis.

They get a limited number to call. They're able to action them within a day, and that's actually helped drive the conversion through as well. So the whole experience for the banker, it's not just about getting something that's better quality in terms of how the targeting is done or how the decision is made, but actually about the experience in the delivery side as well. The last use case is one of my favorites that we've delivered, because this is one that's really valuable for our customers and keeps them safe. So we've done a lot in the scams and fraud space in terms of communicating and educating our customers around how to protect themselves from scams. And this use case is one where, um, within our mobile app, you can actually, um, you have usually a standard setting around how much money you're able to transfer out of your mobile app a day to another, um, to someone else in another bank. And usually that sits in at around 5000. So what this does is if someone needs to make a payment larger than that within the mobile app, they're actually able to change it themselves. But a lot of time our customers don't change it back, or they're calling our call center to ask how to actually change it back.

And the potential impact to the customer is they leave themselves more susceptible to frauds and scams if they don't actually decrease their payment limit down. Um, and and and protect themselves by doing that. Um, this is one of our most engaged actions. Um, and we have seen I've saw some stats last week and it has actually, um, financially reduced the exposure to fraud for some of our customers. So there's a huge customer benefit there, um, that we're seeing off the back of this use case. Um, so to finish off, I need to read these. But Jess mentioned it in her keynote, and one of the things we're really proud of with our implementation is just the impact that we've actually had with our banker network. Um, and what I wanted to do was just read some quotes from bankers around the leads that we had specifically been delivering to them. Um, we've got a, an internal, um, workplace site where, um, actually unprompted, we had bankers.

We sent a new batch of leads out, um, with some of our new triggers. And these are unprompted responses that are coming back. And I think that's actually unprompted. Feedback is always the best feedback, because these people have gone out of their way to actually write this. So I thought I'd just read a couple of these things out just to, to close. And then we can sort of move, move through to questions. Um, so, um, powerful leads actioned by amazing bankers. Love this team. My favorite theme coming through was how timely these these leads are.

Looking forward to doubling down on more of these types of leads in the coming months. Well done! Such a fantastic opportunity for our bankers and our customers are loving the Proactiveness at the time they need it most. Well done guys have have observed these conversations live and the conversation flow is fantastic, surprising and delighting customers with supporting them in their home ownership journey. Nice. Well, thank you very much for your attention and we'd love to open for questions. That was quick. Okay. First one up.

There. This is Doha, Qatar, from citizens Bank. I actually have a question regarding the one of the journeys about the transaction account payment limits. So because the transaction data is so frequent and I was curious, are we able to use that on the NBA runs because I mean inbound we can use real time. But for the NBA runs it might take a lot of time. And I was curious, among all of the other NBA runs you have, what's the longest time that each NBA runs take for like the 10 million customers you oversee? I think we can answer the first question and then the second one you can. I don't think we have that information to hand. But, you know, as we have we have daily data about our customers.

And with Pega you can ingest real time data. So data really latency Shouldn't be a problem. You have to take into consideration what's the right thing for the customer, and what makes the most sense. So near real time can be really important in use cases. Um, overnight batch. If we can see something happened on the account the previous day, that's usually the right time to then reach out to a customer the day after. Thanks. And one last question. Um, it's actually about QA testing side of it because like, I'm pretty sure you heard this a lot.

Like you have more than 800 adaptive models already in place and also the decision rules. And I was curious, what are the breakthroughs and the challenges you have experienced during this journey from MVP one to the current structure on QA testing? I think, um, when you look at the approach that we took to to QA and we took that approach to data, like Lisa said, we actually took a step back. And we're not treating each action as just an action in isolation. We actually look holistically at what's the data that we need from a customer perspective. And a lot of that QA testing is making sure your data is coming through properly because you would never want to use like data that wasn't correct around your customer experiences. So we do a lot of that work up front. And then obviously anytime you release anything to customers, we've got a process that we test the end to end with the channel experience, to make sure it's showing up the way we want it to show up. And usually how those.

Sorry, I do want to let other people ask questions. So thank you very much. Please go to the mic. I forgot to say sorry. Please go to the mic with questions. Um, so the recording will get it. Okay, I'll go first. I'm Angelica Snowden, I'm a journalist with The Australian newspaper. Thank you for your presentation this morning, Jess.

You mentioned, um, there was a 40% increase in customer satisfaction. Customer engagement? Yeah. Are there any further statistics that you can share around what has happened with customers, particularly since you've introduced the Customer brain? Yes, I think the customer engagement stat is the one that I love because it's actually like you say, it's customers reaction to what we're doing in the brain, but we do also track the outcome. So did the customer do the thing that we wanted them to do. So in the transaction limits example, we actually looked and say, did customers decrease their limit? And then what's the impact of that? So each of them are kind of bespoke to what we're trying to get the customer to do.

But at the highest level, we look at our customers responding to what we're putting in front of them. And is there any any further stats that you can share today about that? Not at the moment. Thank you. And if I can ask one more question on on generative AI, um, there's huge potential for that. Um, CBA did a presentation yesterday about potentially using it in the future. Will nab look to do something like that with this platform as well. And will it benefit? We're not looking at this platform at the moment, but we are out in market with other gen AI use cases.

So we're exploring all opportunities, just not at the at the moment. And can you, I guess, talk about how it would benefit customers. From an AI perspective within within Pega Pega is probably the best person to answer that. But I do think there is something to be said for, you know, you can create really great experiences driven by data, but what's the tone of voice that you're using to engage your customer? And I think from the marketing background, that's equally important to make sure that you're not using the same tone of voice for me, that you would everyone else, because we might respond quite differently. So I think there's really a lot of value in power in it from a marketing perspective. Thanks. Yes. Bill Goodwin, a journalist with Computer Weekly.

So following on from that, the last question I'd like to ask what the biggest technical challenges were for you in rolling out this project, and also what the biggest project management challenges were for you. I'm laughing because the the person that did that is sitting in the in the front row. The biggest like the biggest challenge with with technically with anything is around kind of integration, right? We are a bank, we have legacy systems. And so how do you coordinate this Customer Decision Hub to be integrated into all of those channels? It takes a lot of wonderful people and effort across the organization. But I think when you've got the buy in of people because they understand the impact that it's going to have, they're more than helpful to, to to work with you to do it. But, um, yeah, it was quite a bit of work, but for some good results. Thank you.

Um, just just to follow up, is there anything that you would do differently if you were to do the project over again? Any lessons that you learned on the way? Probably lots of things, but nothing that comes to mind. Lisa. There's the one I took through yesterday. It's like, to be honest, I. Think picking the right actions and getting the right coverage is actually incredibly important. So we've done a proof of value before we actually did our production implementation. And I think if I had my time again, I would have chosen actions differently for the proof of value.

That's a good point. Yeah. And you built on that yesterday Lisa, by talking about scale. So how do you balance something that's really bespoke to somebody but also reaching enough customers? So I think getting that balance right up front is absolutely right. And that's why finding focus is a real key. That's one of the key things I would call out. Just don't try to be all things to all people from day one. Just pick a focused area and nail it and then roll it out from there.

Good advice. Thank you, Lisa and Jessica for the presentation. Um, it's great to see your journey. Um, I have a couple of questions regarding this journey. Um, I know you implemented 121 actions around April. Uh, how do you quantify the ROI for that 121 actions? Uh, before and after? Uh, you went live with that. So in terms of the way that we do our reporting, we've got a very nice power BI dashboard that tracks engagement for all of our actions.

And then it actually connects it through into business outcomes. Some of those are a little bit trickier than others. So putting a business outcome on a decreased payment limit just takes a little bit more time for for our, our team to work through. But we're looking at a balance from a performance perspective of both what's happening with the inputs. So once we put something into the system, our customers engage with it. Is it working? And then we'll also look at the outcome and the intent of the actual objective of the campaign as well. So we'll do a balance of both. And I think that's really important.

It's balance. It's around helping customers. So I don't have to I'm not putting an ROI on everything because how do you put an ROI on helping the customers and making sure that we're, you know, talking to them and educating them? Like, I'm not going to put a hard number on that, but I will look at are they engaging with it and do they get value from it? And the next question is, um, how do you prevent the false positive? False negative from the model? Um, are there any key challenges you've gone through in terms of the model? Um, I'm assuming the model is within your ecosystem, not in, um, Cloud or Large language model yet, is that right? We've got models that we feed into the, the brain, and we've got Pega adaptive models, and there's tooling within Pega, um, that can help you maintain and manage how those are performing.

So are there any resiliency practices that you want to share, like, um, how did you solve major, uh, false positive, false negative outcomes? No, nothing. At this time, a lot of the stuff we do is also really trigger based, right? I can see that a customer is on our website doing something. I can see that there's a need. Um, that's that's not a, you know, a false positive. That's just something where we have to respond to that customer in the moment. Thank you. Thank you.

Appreciate it. Anytime. Um, I think my question was kind of on the models. You guys spoke about how you have, like, different flavors of actions like sales, service, etc.. Um, how have you actually set up those models? You know, considering, like, every action has a different outcome. You know, something could be sales, something like service focused. Are the adaptive models set up accordingly based on those specified outcomes? So the adaptive models are really focused on identifying the propensity of a customer to actually engage with the message that we're actually putting in front of them.

So that's what exists is the dependent variable across the whole suite. And that's how we're treating all of those. We've got our own models that we will bring in and use as part of targeting criteria for programs of work. And from a model development perspective, we have really put a lot of thought into the feature set that we're actually using within CDH to actually build the adaptive models as well to ensure that it's the right set of features to enable the models to be as predictive as possible. And we've got a pretty sophisticated model governance process at NAB that we use just as part of our standard. Okay, great. No follow up questions. Thank you. Hey, that was the first round of applause.

Andrew Birmingham I'm a journalist with Mi3 Australia. Um, what's going on in the background is you guys are rolling this out. Is that the federal government is looking at a pretty major overhaul of the privacy legislation, including, um, wrapping up automated decisioning. I'm not going to ask you about the legislation, but I am interested in the extent to which you've had to factor that into your planning, whether you've had to, um, whether you've been getting updates from the kind of regulatory folk in terms of if they make a significant change and they're talking like it's going to be a significant change, what the implications are and how you can how flexible you are, you have to change quickly. Yeah, absolutely. I'd say first and foremost, like we always respect and adhere to customer's Customers consent and privacy. That is a non-negotiable for us, and we've got an amazing privacy and ethics team that we engage with. And so the privacy and ethics team, we engage with them quite regularly because as you can see, we've got quite a lot in there. And so for us, every use case we go to them, any kind of new use case, we'll engage with them and they'll really challenge us to go, are you thinking customer led?

Are you thinking future led on the where the policy is going and making sure that we're doing the right things for our customers? So great question. Thank you. We did a deep dive session with the data ethics team on the ethical bias capability that actually exists within CDH as well. So we've done we have tried to sort of go, these are the the tools actually at our disposal as well. We want to make sure we're using these so we can keep our customers data safe. Yeah. Thank you. Well thank you very much everyone.

Enjoy the rest of PegaWorld. Thank you.