PegaWorld 2026: Lloyds Banking Group: AI-powered Commercial Banking Transformation with Pega

Hey, everybody. Thank you very much for joining. I thought we'd get a good audience. So yeah, really good to see everybody. So thank you. So I'm Tom Stewart, the client director for Lloyds Banking Group. Um, and it's my absolute honor to introduce the presentation. They're going to share with us on their general use case in commercial banking that's delivering lots of real world value. And also on a Pega award yesterday. So yeah, really, really looking forward to that. Lloyds have been a client of, uh, of Pega for nearly two decades. We've supported them as the Workflow Automation partner of choice. And now they're in the process of implementing CDH as well. So really, really great history working really, really closely together. And yeah, really honored to, to introduce the team here. So thanks everyone for joining and I'll hand over to the team. Thanks, Tom.

Thanks so much for having us at PegaWorld. It's been a fantastic few days so far. You've been absolutely brilliant as the rest of the team have as well. We're going to introduce each other. So this is Papu. Manu has joined our lab from as we set up the LTC Technology Center in Hyderabad a couple of years ago. He has 20 years of Pega experience across different clients and has bought a huge amount of knowledge of the application and technical oversight to the team. Been a pleasure to work with you. Thank you Harry. Thank you and thank you, Alan and everyone for hosting us here. It's a fantastic two days. Um, very energetic, very energized and very eager to share our journey to you. So Harry is our lead product owner. He's Bristol based. He's very passionate about the business, very enthusiastic and passionate about Pega technology as well.

So he aligns very closely with what we say from the technology side. So that helps us move forward and stay current with what Pega is rolling out. So let's hear out our journey from Harry. Thanks, man. All right. So we've introduced ourselves. I feel like we should introduce Lloyds Banking Group a little bit. We don't usually have to do this in the UK because we are quite a big household name there. But this is an international audience, and I think it probably makes sense to take a step back and lay the road. So Lloyds Banking Group is one of the leading financial services organizations in the UK. We have 65,000 employees and 30 million customers. We have reach across retail and commercial banking as well as insurance, pensions and investments. The group is just over 250 years old, so that's about ten years old in the United States.

Um, and I represent the commercial bank. So that's business banking for businesses and entities from the very small to the very, very large. So we've got small independent window cleaners, retail firms, manufacturers, right up to large multinationals, Other, banks, etc. I've even got a big presence in the agricultural and farming market. So hugely diverse set of customers, really big organization. That scale really matters. And I'm sure many of you realize this and know this from your own organizations, but in a really big organization with lots of people in it, small incremental changes can make a massive difference and deliver substantial value quite quickly. So the area our application supports is commercial banking operations. So these are the teams that service that really diverse client base from really quick, simple transactional sort of service types. So data maintenance, online support, mandate changes, etc. through to support with specialist lending products trade, finance, asset finance, right through to moments that matter, moments of truth where we really make or break a client relationship.

So things like bereavement and complaints also manage within our application. So really diverse clients, really diverse set of services and products that we run. And therefore the application needs to be big. It needs to be scalable. It needs to be flexible. So before we get on to the GenAI elements, I want to take you through just a few slides that sort of set the scene of how our application has grown over the course of the last ten years or so. Um, that transformation and that growth has gone hand in hand with the transformation and growth of the commercial banking operating model. So I'm going to take you through both sort of side by side over the next literally 4 or 5 minutes, and then we'll get into the GenAI elements. So back in sort of the mid 20 tens, the operating model looked a bit like this map of the UK on the left, which looks a bit sickly with lots of green dots all over it.

Each of those green dots is a localized service center, offering a huge range of processes and services to the local clients in that area. Now, the problem with that, as many of you will cotton on, is every site offers everything and they all do it slightly differently. So there's very little standardization. It's very hard to implement sustainable, lasting change into a model like that. On top of that, we've got front and back office separation. So the people in the servicing centers who are talking to the clients are not always the people who are pressing the buttons to make that request a reality. And in our banking systems, lots of other problems attached to it as well. We did have Pega in this operating model. We had a version of Pega six, a really early build of our app. But it's pretty basic, right? It's not doing that much more than storing and tracking the work and logging that.

It's happened over the course of the next 2 or 3 years. We invested a huge amount of time, both from a business perspective and a technology perspective in. uh, streamlining that model, consolidating the sites down to centers of excellence in some key locations. But it's not really the consolidation that's most important. What really helped us drive through change was that we created specialist teams in those sites. That took us sort of vertical slither of that process backlog. The service catalog. And they became experts in doing that particular process. So an account closure that comes in in the new model is done by one team. And they do it the same way for everybody. And that gives us the opportunity to really start to implement change into those processes and yield great results at the same time. With that, we're introducing Pega seven. So a rebuild of the application utilizing some of the components within um that application.

So we bring in Getnext work. We have parent and child in place for the front and back office handoffs. And we also centralized our or created a centralized indexing team who were looking at all the emails coming in. So emails are still quite a big part of the commercial business. Our clients like using them and categorizing them and sending them out to the various teams to get the work done. We also started at this point to introduce some bespoke flows. So rather than just a generic Pega screen that everybody looks at, we started to introduce orchestrated bespoke flows for some of the core complex processes. So account closures were one of the first to get one of those. We brought in some robotics side by side and it became more automated, more standardized, more efficient. Right. If we move forward another few years in this iteration, um, the last part of the, the last slide was about sort of site consolidation.

What this one's about is flattening that model, getting rid of that front and back office separation. So we empowered our teams to work processes end to end. So the person who gets the request is the person that fulfills the request. And then the person that speaks to the customer. So they pick something up, they execute it until it's done. At the same time, we start to bring in. Pega. AI. So. Pega Platform. Onto the Infinity. Um, application. And as we bring that in, we're leveraging more and more of the um, components and widgets that Pega gives us, right? So we start to introduce auto indexing via machine learning that reduces that indexing team. I was talking about a moment from 40 people down to about 15. We build out more and more bespoke flows. Um, so we introduced at this point a lending flow, a trade finance flow, an audit letter flow.

So more of our teams are getting specialist treatment within the application and getting flows that are really suited to, to them. So where are we now. Right. The commercial banking operating model in 2026 has gone global. Um, we have sites now set up in several cities in India, um servicing the same customers, same processes, but we spread out obvious drop in OpEx by doing that, and it's no exaggeration to say that the stability that the operating model changes we did over the previous five, ten years really set us up to do that. There's absolutely no way that model would have worked five years prior to making these changes before we made the changes. It's a stability and the knowledge that as something comes in, it's immediately categorized, goes to the right person, and they've got the tools to handle it in the right way every time.

That makes that model a success and lands it properly at the same time. We've continued to invest in Pega, bringing in new features, taking in Infinity upgrades when they're available to us, investing in enduring teams to continue to build out new workflows, new bespoke, um, orchestration for our teams, new components that they can all use. Final slide before we get on to Roche and GenAI elements of the conversation. So this is where we stand today. Um, so our principal is one Pega app with multiple reusable features. On the left hand side, you can see a selection of reusable components that have been built and configured in the application. Uh, I'm not going to talk through all these. You'll be glad to hear. What I will call out, which has been really fundamental to the success of the application note is um, self serve config sort of delegated rules.

Um, we work incredibly closely and have done since the app's inception with a dedicated embedded team of citizen developers in the business who have been absolutely fundamental to the success of the application. So they keep those sort of config updates, you know, new drop downs, new questions, etc., into existing flows. They can make those they know how to do it. They know how to do it safely. And it keeps our developers free to do real value add transformational stuff. It's been absolutely fundamental to the success of the app. We've then got integrations and services, so we're really deep across multiple bank systems in terms of being able to pull data and push data back to systems. Um, the principle we tried to work to is anybody looking at one of our Pega applications or our Pega application, one of the flows should only need to look at Pega.

Everything else should be brought to them to manage and should be able to be pushed back. Now we're not there yet with every journey by a long means, but that's where we're looking to go, and that's the principle we work with when we launch a new journey. And then finally, the process flows. So I've talked about a number of these. So these are the bespoke journeys that we've launched within our application for a number of teams. Um the one thing I haven't talked about though, and again, a key part of the success is right at the top of that. You've got the CB case flow. Now that's our basic universal case type. It's super simple pre-processed process execute. But it allows us to onboard people onto the app really quickly. And the fundamental thing here is they have access to all those integrations, all those components, pre-configured, ready to use within the app.

So it allows people to get onto the application, get value out of it really quickly. Some of them may never leave that flow if they're simple, you know, we don't need to do anything more complex for them, but sometimes we'll onboard people onto the app, get to know them a bit, understand how they want to use Pega, they learn a bit about the app, and it enables us to learn enough to build a bespoke flow. So that's where we are in terms of the application, the sort of size and scale of it. Um, we're right across commercial banking ops now, so there's not much further to go in terms of width. The next place we can go is adding more depth and adding more value into those journeys. And that's where AI comes in. And she might take us through the journey there. Yeah. Thank you.

Thank you Harry. So thanks for setting up the history of the application and the scale of it. So we have 80% of our operations team on business and commercial banking. Working on this application and close to 50% of corporate institutional banking back office operations teams working on on the same application. So what I'm going to talk about today about the, the AI journey that we have is may not be new for all of you because today everybody knows that AI can do summarization. AI can do do content generation and all. It's it became a normal thing. So what we wanted to do is a quick that the journey that we had gone through, just let's take a quick time travel to one year ago. So last year, April to May, we were on Pega Infinity 23. And we are rolling out processes after processes. And then we had this plan of upgrading to 24 along with the PegaCloud three migration.

So then we took that as an opportunity to not just migrate to Pega 24, but you also leverage the most beneficial features that we can to improve the productivity of the workflows that we have. So that's where our journey started in April, May last year. So first we identified what are the problem areas where we can improve, where there are friction points, where the the journeys are not scalable in terms of servicing, right? We have people dependent when we get new customers and a lot of new servicing journeys, how can we scale? So the basic principle at that point in time is giving time back to the colleagues to focus on most important things. So that's where I'll talk about the different problem areas. So the next step that we did was, I mean when this migration upgrade was happening, we also had probably 3 to 4 good Pega engineers doing the capability assessment of all the Pega GenAI features that were available at that point in time, uh, doing an assessment in a sandbox environment and then work through it, mapping it to the business, the processes that we have and coming up with some recommendations, taking that to the architecture Enterprise Architecture Forum.

And then we had a set of recommendations and guidelines for that. So that's when, I mean, we took that in August to September. As you know, the enterprise architects, it takes time to go and, uh, talk about all these things, right? So that's why you see, it's a two month journey, but ideally, it's probably 3 to 4 meetings in that two, two, two month journey. And thanks to Pegasystems in, uh, explaining us the deep architectural elements of how, uh, the Pega GenAI is configured or set up beyond our VPC, uh, explaining every single detail of how the data flows, how it is governed, guardrails, uh, in the GenAI gateway, uh, that they have. And then the actual development cycle started in October. That's when we finished our Pega 24 upgrade. And the scale of application also took us a lot of time for testing this application.

So the actual GenAI development started in October with only two developers and one Kiwi, and we were able to roll out the first use case in November 2025, which is on the data extraction. So the use case we picked up for extracting for taking it to production is we want to go with a low impact use case first to see how it scales, understand the accuracy of it, and then eventually scale it to, to, to the larger areas. Can you move to the next slide please? Yeah. So the problem space like I said. So we create. So this application creates 2 million plus cases, uh, every year supporting both BCB and CIB business segments with 5 to 7000 emails. Emails play a large role in the intake where the emails are coming either from MBS or whether from the customers directly or from a different operations team.

And there are 2000 outbound communications that go to the customers directly from Pega. So that's where most of these, um, in 2023, we started off working up with predefined letter templates for all of this, uh, customer communications, but not, not scaled enough, but very few bespoke workflows. But then, uh, I mean, that's the scale that we, that, that's the scale where a lot of manual effort is required from the, the operation teams. Right. A lot of copy pasting the content from the documents, a lot of copy pasting from the emails that we get right. A lot of reading of the email and understanding what category this particular, uh, service or particular team this has to be routed to. That's where the indexing part comes up. The 15 member team who sits in Mumbai. So they read through each email and say, okay, this email belongs to that particular category, but before it goes to the 15 member team, we also have an email bot which has a machine learning model.

So we have trained the model with a lot of samples of emails and the accuracy that we were seeing was around 80%. And sometimes it fluctuates it probably depending on the context of the email. So it may drop down. So it is not consistently going up, but it is fluctuating between 60 to 80%. That's where the Reindexing team will come. And they sit through all the day. And each member of that team operates around 150 cases. Because if you see there are 7000 cases getting created, so 150 close to 100, 150 cases will be re-indexed and routed to the right team to service that. For example, if if the if it is all about account closure, then this person will go and reroute to that account closure work basket. And that's where that team will go and pick up that journey. So if you see there is a lot of friction here, like high handling effort handoffs between the teams.

Uh, and then the other point is about the classification errors, which I talked about. How do you classify after you extract the intent from the email from the text analyzers, how do you categorize and map it to the right set of predefined categories. Uh, yeah. So the barriers to scalable transformation is basically we have an objective to be as fast as we can in terms of servicing. That's where a lot of effort is kept in the self-serve digital digital servicing aspect. But when it comes to commercial banking, uh, it is not so simple to deal, to push everything to the self-serve. So there is a lot of complexity involved there. That's why human in the loop is human servicing is always required for that. Uh, and that's where the longer resolution times are also leading up to the application is really, really good. But the point is if you add more customers, more servicing journeys, the time to serve them is dependent on the number of people you have in the operations group, right?

So and the longer resolution times. So that's where we want to leverage AI to see how we can scale that part. So that's one of the problem space that we looked at between April to May. You move to the next slide, please? Yeah. So as part of capability assessment, we have looked into seven different categories. This is pure play engineering stuff. We want to at that point in time, the engineers need to understand what each offering from Pega is all about. So we mapped that to the understanding of the email, the classification, extraction of the content from the email, structured content generation, not just the templates, but on the fly, structured content generation, unstructured content generation for the emails, for the emails going outbound Other of course, guidance for the colleagues to deal with the cases, and also for this operating procedures and all and the coaching part.

So this was like probably 4 to 5 weeks of effort that the engineers have put through. And we worked with architecture team. And for each purpose, we have a recommendation that if it is understanding, we are good to use. Pega. GenAI. Right. So proceed with embedded summarization if it is classification. What we realized is Pega, the GenAI component is not rightly categorizing based on the content. The accuracy levels are not as much as we have in the ML model. So we thought we will discard that for now. We will come back to that point later, but we'll continue using the IBM model for the classification part. Then. Then the extraction part is all about the structured forms that we get, which is going to talk about, and also the unstructured PDFs. And all our focus was more initially on the structured forms where we get an asset finance form with a table like structure could be in an email body, or it could be an email attachment.

So Pega will read that content, send to GenAI, get the fields extracted and put it in the case. So that saves even if it is a one minute save for the colleague. Uh, it's still, it's a massive thing that we wanted to try and roll it out to n number of workflows that we have and followed by a unstructured content generation. So we are still evaluating, uh, at that point in time. But now we are ready. So we are about to release few more use cases, which includes the unstructured content generation, which is anyways going to talk about it. The guidance on the Knowledge Buddy and Coach, we have put significant amount of effort in understanding the Knowledge Buddy building the knowledge repository within, uh, Pega ecosystem. But then as an enterprise, uh, there is a conscious conscious decision made that there is another, uh, knowledge management system that has been built for the enterprise.

And that's where all the system of operating procedures and all have to be pushed, uh, and then read from. So that's why we can still use Knowledge Buddy, but it has to integrate with, uh, an enterprise knowledge management system we have within Lloyd's. Uh, we haven't picked up that use case to production yet. That's one of our priorities for this year. Can you go to the next slide? Yeah. So the key challenge here is we have a small set of developers who can Pega will help us move very fast on that. But being in a very regulated and compliance specific industry, we have to go through a control plane, basically the architecture forums, to make sure that what we are, what our use cases that we are picking up are safe enough to do us to use GenAI. So the, the left portion of it, you see control plane, when you say control plane, it is purely a process thing that we went through with the architecture risk and compliance teams to evaluate how the model performs, right?

The entire AI architecture that Pega has put up behind the GenAI connect rule and how it calls LMS, what models are available at that time. So all that manual review of the architectural elements was done during that that policy. And we were also Putting up this human in the loop rules. Of course, every GenAI call that we make. We have ensured that there is always a human in the loop. It is still not a straight through processing, not completely relying on GenAI, but then the confidence for the users is increasing day by day based on what we have on the execution plan, it's more about for the engineering team, they learned because they're all Pega developers. The AI terms are very new, so they learned about prompt engineering. They learned about how to do masking using. I mean, there is default masking that Pega does.

But behind the scenes there are. When we thought about it, there are 150 different elements that needs to be masked up. So for that, the keyword based masking and then the text analyzer based masking, masking, and then the router based masking were all implemented for the at that time. Um, and then a lot of filtering, logging that Pega applies. So the other challenge that we had is we worked with Pegasystems to see the Pega guardrails are aligning with the guardrails, and most of them were aligning. We didn't had any challenge there from the architecture team. So the, the reason we are calling out this is because we started evaluating the entire landscape, but we went very slow. We picked up a very small use case to start with. Last year we went to production. We received close to 98% accuracy. But once we got that into, it enabled this architecture assessment for the rest of the business units as well.

So now if consumer business wants to do IP and has to do they just follow the template that we have done. So all the timeline that we have gone from August to October, that is all. Not required now. So people can just go and enable GenAI in their VPC and then start rolling out the feature. So you'll see more and more GenAI being used across the US within Lbg. and this is all about it. This is only about data extraction, which Harry will explain. But we also have moved past that, and we were now solving the problems of case summarization, which includes summary of case data history, Pulse notes, related cases, attachments which are stored in some other repository. Of course, there are engineering challenges over there. We can't send a big context assembly to LLM. We are seeing some performance issues. That's where we are engineering to see how best we can summarize the content, and then send the summary of summaries to LLM and make sure that, uh, the rest of the application is not impacted by one single thread, which is waiting indefinitely for the LLM to come back.

So we are working out on the context assembly and all that stuff. So that's a very good, uh, engineering experience that we are getting. We moved away. So where Pega helped us accelerate in, uh, building the GenAI with the prompts, but there is a lot of engineering also that needs to be done to make sure that the performance is met when we are scaling the GenAI use cases across a number of processes and workflows. Great, thanks. Okay. Right. We're going to talk about a couple of those use cases in a bit more detail now. So the data extraction use case that was just talking about was deployed into our asset finance business at the end of last year. Um so asset finance was on our application already, but it had one of the basic builds on our universal flow. Um, and we knew we could do better than that.

Right? They're a fairly complex business with a fairly complex flow. And it's something that we wanted to improve. So this wasn't GenAI for GenAI sake. This was a genuine business problem, something we were working on anyway. But we saw the opportunity to go a little bit further with the GenAI. Um, you know, asset finance business 12 process steps, not really an orchestrated flow, lots of checklist questions, lots of swivel chairing to forms and systems. Minimal data, process insights, etc. etc.. So what do we do to help them? We did what we've done plenty of times before with lots of other areas. We sat down in a room with them. We talked to them about their flow. We talked to them about the problems they've got with it. We did a bit of re-engineering of that flow, and once we were in a position where we thought, that's the best way to execute this process, we designed a Pega flow around it, structured workflow, automated comms, full MRI, starting to source data via our integrations from source systems.

Where we saw the opportunity with GenAI, however, was data extraction. So the asset finance team, every deal that they did, they get a PDF in with key data in it that they were manually rekeying into the application. So here you can see probably not that clearly, but here you can see a asset finance form at the back left with key data in it. You can see that that's transposed directly on the back, right into our Pega screens. And at the front there is the prompt that's generating that data extraction. So results are in for asset finance. As Manuel said, this is starting relatively small, right. Um, but it was a pathfinder a foot in the door to prove this capability and moving faster. You can see we've got some really good feedback from the users. Um, and then in terms of the, uh, the metrics around this, um, this is a slightly out of date slide.

My apologies. So it's over 5000 cases created now, over 500 hours of manual effort saved and 99% accuracy in the data extraction. So genuinely exceeded our expectations in what we were expecting to see from this process and has been trumpeted from the rafters to BCB. Right. So it's one of the first gen AI use cases. Live in the bank, got a huge amount of coverage. Um, and yeah, a real pathfinder, right. So where do we go next? We've definitely gone bigger for the next use case. So a couple of reasons why I felt guilty being here this week. The first one is I told my wife I was going to a software conference in Vegas on her birthday, which went down really well. The air quotes are hers. Um, and the second one is we launched our complaints Pega application on last Monday. Right. So it's in model office right now.

Very low number of cases going through but huge moment for us. Um, really, really complex flow. It's probably the most complex thing we've built in our application. Um, it took about a year and a half all in from discovery through to implementation. Um, you can see here like multiple handoffs, lengthy email chains and docs to review processing time takes too long. So again, redesigned case flow. I could stand here and talk for hours about complaints, which I'm not going to do because Tom will kill me. Um, so, you know, we, we built a new flow for complaints. I wish it was that simple, but we have um. But where we saw the opportunity for GenAI was in a few places. So there's three use cases here and I'll try and sort of talk through them relatively quickly. The first one is initial complaint summary. So when a complaint comes into the team um it can be really complicated right.

It's got uh notes from the. MM. It's got a customer email. It may have an email chain attached to it. It's got various notes and data from the systems that it's hit before it hits the complaints team. The first thing that the user needs to do is to get on top of what the complaint is and understand it. That can take them up to 30 minutes or longer on complex cases. We've fed all of that information directly from our app into, uh, the LLM via Pega, and pulled back a summary in the case window that summarizes the complaint in a structured way for that case handler to get on top of it really quickly. Not only is it there when they first open the case, but it's also there every time they open it, which if you imagine your complaints handler with 1520 cases on the go, serves as a really good reminder every time they open the case.

This is what this one's about, this is where I am, etc.. On top of that, though, from a summarization perspective, we realized that because of the breadth of our app and the fact that most of commercial servicing sits in there somewhere, and often the thing that's gone wrong that's caused the complaint has a Pega case for it somewhere in our application. So another service in case from another team. So we've not only given the complaints handlers the power to summarize the complaint, but we've given them the power to summarize any case attached to that client within our application. So within the same window, within the same complaints case, they can navigate through the client's historic Pega cases, hit summarize, and get a structured summary of that case to help with their investigation to work out what's gone wrong. Absolutely game changing for them. We hope it's going live and feedback so far has been really positive. at the third use case in complaints.

And we're going live with all this stuff in the MVP. By the way. Right. So it's quite a quite a challenging time but should deliver a lot of value was letter content uh, being drafted via Gen AI. So at the end of a complaint you issue a formal response letter. A lot of that is templated, but there's a sort of paragraph or two in the middle, completely free text. This is what happened. This is what we did. This is any compensation, etc. that the complaints handle is tight for every single complaint. All the information needed to be able to draft that is captured in the case somewhere. So we realized if we send that through to an LLM, we can generate that content directly into the Pega screen for them to assess, add to the letter and send it out. Now we've got full human in the loop for that.

We're not trusting GenAI right now to write to our customers, but it creates a draft for them to, to look at, to review, to update. And, you know, everyone knows editing a draft is a lot easier than coming up with something from scratch. Hugely excited about complaints, it goes fully into live full ramp up from Q3, and the results we're expecting from it are, um, really quite big. Just to add one point, the draft also includes the case data. It's not just a plain bunch of words, but it also includes the context of the case data and what, what exactly is required at that particular stage or step. Uh, it understands that context as well. That's how the prompt was written. Absolutely. Um, I've got one slide just showing the, uh, the summary here. Again, it's going to be a bit too small to see, but you get the point, right?

There's multiple bits of case data going into this summary. And then a summary is really structured, right. We've worked through with the complaints team. What do you want to see. What do you care about what's important to you. Um this is a fake complaint. It's not a real one. Just so everyone knows. Um, and um, and it will provide that for them on every single case. Okay. Right. Um, I'm conscious of time to get to the Q&A, Tom. So I'm going to go through the last couple of slides relatively quickly. Um, this one is just a visual really. This is where we've done an assessment of our signature journeys within our Pega application and where we see the GenAI components adding value to the app. And you can see it's really busy, right? We've got backlog here probably for 2 to 3 years, um, in terms of delivering GenAI into our existing journeys.

Super excited about it. The more we can prove it, the more we'll get the confidence to roll it out. So just want to show the, the visual there. And then final slide then, um, what have we learned along the way? Um, so first thing here, I don't mean this to downplay all the really hard work and the complexity that managers team has worked with, but I think the point I want to land is it has been quite simple and intuitive to use the components, right? We've not had to hire entire new teams of data scientists and new Pega developers. It's is our existing teams that are working with us every day and delivering those use cases into live. So I think, you know, fantastic. Um, you know, that Pega has managed to make this in a way that we can just use really quickly. And the second point, and the one I'm probably most passionate about is the collaboration that's gone into these.

These are absolutely brilliant use cases for getting around, getting into a room with business users, developers and Platform people and talking about what they want to see in a GenAI summary, because you can alter it and tweak the prompt so quickly. We've had real success just getting people into a room for a day. We'll walk in with an example summarization, say this is what we think it could look like, and then get their feedback. You know, what else would you like to see? Where would you like the paragraphs? You know, which order do you want to see them in? What headings do you want? What bits should we make important, etc. we can then send them off for a cup of coffee or some lunch. Get them back 45 minutes later. We've updated the prompt. We've regenerated it, right? We can do that 3 or 4 times during a day and come out with something that is, you know, pretty close to going into testing for us.

And we've seen great success in doing that. And it also gets great buy in from the business as well. They come out energized. They come out feeling like they consulted in in the delivery. Final part of this slide and the end of the presentation, we're probably slightly over where we wanted to be. Um, is around feedback. So we've built into any generative content in our application. The user will need to say thumbs up or thumbs down. Right. Was it helpful? Was there something wrong with it? If it's something helpful, fine. They don't have to do anything else. If they do a thumbs down, we've got a list of things that need to tick through what wasn't right. Did it hallucinate? Was the content not appropriate? Did it? Um, you know, was it unhelpful, etc.. We'll be monitoring that feedback every single day via live reports so we can keep on top of how those prompts are performing.

Now, obviously things like hallucinations, we'll look at immediately what's going on, what's the problem there, etc. but that feedback is really rich and really useful over a course of time as well. So it will guide us in terms of which prompts are performing well, which prompts need a bit of a tweak and enable us to continue building them? Cool, right? I think that is us. Sorry, sorry. Yeah. So I just wanted to call out one thing that we have understood how it works. The engineering team understood how it works. The business teams understood. Now we are going to scale and operationalize operationalize it across the application. In terms of all the use cases that you see now, where are we heading into? That's a very good start this morning on the Agentic AI and the Agentic Process Fabric. If you see the number of flows that we have in the application, we are really looking forward to.

And again, we are our upgrade cycle is coming again for Pega 25. We are going to reassess to see if we can directly go to 26 and then see some of these embedded into large number of agents that we are building on GCP and how they get connected to the Pega workflows here and make it more predictable. And that's where our path will be from now on till next year. Hopefully, we'll be successful in doing that with the newer versions of Pega. Yeah. Thank you. Yeah. Huge thanks to Harry for taking us through that. And they mentioned how well heralded that use case has been on BCB, but also from a Pega side as well. It's hugely well regarded with the work they've done and the teams have really, really enjoyed and really helped be driven by Lloyds as well and what they've done. So thank you.

Um so we've got some time for a Q&A now. So any questions from the audience. Ah. Oh two. I think you're you're closer to the. A. Great presentation. You talked about the iterative prompt tuning for complaints a little bit. So that helped. But just curious about, you know, your, your experience with different models, the language models, LMS right. So did you find a particular one better for summarization and context building of the complaints, like a GPT versus a Claude or something else? Speed versus accuracy, etc.. Yeah. So we picked up we picked up one model to start. Can we repeat the question just because. Yeah, the thanks for that. So the question is about it. We try different models to see what brings more accuracy. The comparison of models we really wanted to do. We initially did that in our sandbox environment with GPT and Gemini.

And we also have a an in-house Gemini based models already in Lbg. But the use cases that we have picked up in the. We. We went ahead with the GPT four to start with. Now it got upgraded to 5.1. Um, so we really haven't evaluated the different model comparisons, but we have evaluated with the temperature settings within the model to make sure that it is bringing the right accurate information back to us. But then that's a that's that's in our mind. We'll definitely evaluate with the when GenAI is being open up with different models. Now in a 25 version, we're going to compare it for sure. Um, but, but that's, that's in our thought process and we haven't done that yet. Thank you. Thanks. Hey there. Casey Augustine from Bank of America. Uh, so working in the commercial banking space as well. So a lot of what you said, especially about emails, we get it like that is a lot of what we deal with.

So I, I, I have about a dozen questions, but I'm gonna keep it to two. Um, the first one would be when you are doing the channel to get into the agent or to go through the gen AI process. That is an internal workflow. Do you have anything exposed to clients to get into that workflow yet? No, we've not got a client facing application. This app isn't a client facing application. Okay. Yeah. Okay. Um, and then the second question with the data extraction, if you were going through that. So there's different models within KYC. Different jurisdictions have different items depending on what branch you're in, different things like that. Did you run into with how different forms are structured of how you get that data back? So account opening can be very specific when it comes to certain regional jurisdictional issues or different areas. How did you all combat that to get that 99% accuracy?

Because that's very impressive. Yeah. Uh, so I got to talk to this in more technical detail, but, um, I think the advantage we have with the asset finance use case is the forms the same every time, right. Um, what we found in the POC was structured data, um, particularly when a form is static and same incredible, incredible accuracy we're seeing on that better than what we were getting from OCR. Um, for unstructured data, it probably drops off a little bit. So I think some of the Doc agents in 25 were really interested in seeing, because we think that might give us the power or confidence to go into more unstructured forms as well. But yeah, it was it was a structured form. Yeah. So there are two aspects to it. But the use case that we picked up is a very structured form, either in an email or an attachment.

So it does really well whether you get an email, the structure in the body or structure in the attachment. So that problem is solved. Uh, but again, the I echo with the, the need that you have, we do have similar need, but we are waiting for the, we didn't progress much on that. We wanted to doc agent to be available in 25 and start leveraging that rather than investing heavily on that. If you see our journey, we were progressing more towards the compliance journey rather than the data extraction. Data extraction was the starting point for us. And then for the last six months, we were doing complaints. But then going forward, when we upgrade to 25, that's where we are going to use doc A, and there is a lot of, uh, architectural assessment is happening on whether to use doc AI or an in-house built extraction, because data extraction is a very common use case across enterprise.

So there is a large number of AI engineers working on the data extraction. And even there there are different specializations required there. So we really have to tap into. I don't think our application has that kind of a need, but we have different applications like KYC, you said are going to pick up those use cases in the coming six months. Yeah. Just with commercial banking being still so paper heavy, I was just wondering if you guys had figured some more of that out, but that makes a lot of sense of just the one form. Yeah, we're just waiting for doc agent to come and solve the problem. Hi, Nicole Crosier. I lead it for silver Rock. We're an insuretech operating in a multi-tenant family of brands. I was wondering, Lloyds being multi-tenant. You have Halifax, I think Scottish Widows as well. How do you account for that within your Pega environment and particularly these workflows?

I think you mentioned you kind of consolidated into one API. I imagine that adds some complexity. So let's have close to 25 different Pega applications across the different business units. And these are isolated into different VPCs. We have close to eight VPCs and they are categorized into VPC for commercial banking, VPC for consumer for insurance and all. So they are isolated and separate by themselves. But we also have a lot of VPC communication that we have use cases, especially for bereavement, for account closure, that the applications in Pega talk to each other for creating the cases and exchange the data. But as an enterprise, we also have a rule that VPCs cannot talk to each other without knowing the context. So what happens is the VPC will go to an AWS instance, which is sitting within a Lloyds hosted AWS tenant. And that's where then it comes to the other the other VPC, wherever we wanted to go.

Your multi-application within the same environment or your multi environment. We are multi application. We are multi environment also. Yes and yes. So if we have a if you take our commercial banking VPC there are four different applications sitting there and there is another VPC for LLM KYC where you have an onboarding journey, Offboarding journey, the fraud, uh, smart dispute and smart investigations are all there sitting in. So they are all well categorized based on the frameworks that are, uh, that the applications need. And yeah, we are multi application multi environment. Thank you. Yeah. Thank you. I have a very quick question. Yes. So I had a question around the data residency about the GenAI use cases etc.. Right. So I'm not sure what Lbg environment is, whether it is on prem or on PegaCloud. But let's say when you are posting the case data to a to an LM model, right?

So that can include sensitive PII information assets, right. And knowing LPG has, you know, high compliance and, you know, the risk assessment as such. Right. So how do you make sure that you are parsing the data or sending the data in a secure manner? And wherever the data goes, let's say whichever LLM, whether it is OpenAI or anthropic as such, right? So whether these LLM providers are basically securing the data and maybe not sharing it more widely with other groups as such. Right. So how do you ensure ensure the data residency part of it and the processing part of it? Yeah. So, um, what we are relying more on is Pega. GenAI. Right. So from the VPC that we have, we are writing the prompts and we are making a Pega. GenAI. Correct. So as part of doing that, we are masking the data. I mean, the problem that you said, the running through the architecture, that's what I was talking about, right?

Running through the architecture, understanding the masking part. So the data is getting masked using either keyword or router based masking for the sensitive data, which we do the masking. After the masking is done, it's all the Pega Cloud we are on PegaCloud the first answer, right? So Pega is taking care of a lot of those things for us. So the call goes to Pega GenAI service. It's all TLS based. And then from there the guardrails are all applied. When I when I spoke in the beginning, the guardrails of Pega that Pega maintains in the LPGA are almost very similar. So it's, it's very isolated. Uh, the, it's stateless predominantly. So the call that goes to GenAI. Nobody is tracking that. That's what Pega says. And we believe that's what is happening. Uh, so a lot of trust on the, on the Pega side, uh, and we also have our own internal summarization, uh, APIs within the, uh, so when we are sending any very complex or very heavily PII related data, we will, instead of using Pega.

GenAI. Uh, GenAI calls to Pega. Open API. LMS, we can use the LPG LPG ones actually. So the security part and all is taken care by. So that's where that's where we went fast because we trust and we trust not just by the name, but we careful evaluation. Uh, when I, it took two months for us to go in detail of the question that you asked to understand each element of how things gets stored, things, whether they are stored or not, and then how the things are passed and response coming back. Uh, yeah. Thank you so much. Thank you. Perfect. Well, thank you everyone. I think we probably could facilitate questions for another hour or two, but I think we've timed out now. So I think the teams team will be around for a few more minutes. And we'll be around for the next couple of days.

Um, but yeah, really appreciate everyone's time and engagement and yeah, enjoy the rest of PegaWorld. Thank you. Thank you.