Pegaworld 2025: Transforming Legal Operations: Full automation with GenAI and the new era of legal operations

PegaWorld 2025 - Transforming Legal Operations

It's a great honor to be here to share with you Santander legal case in Brazil. Hi there. I'm Ricardo Miguel, responsible for digital transformation workflows at Santander. And great to be here with you.

Let's begin showing an overview of the size of Santander in the world. Santander Bank is one of the largest and most respected financial institutions in the world, with a strong presence across Europe and America, and we proudly serve 171 million customers around the world. 17 only in Brazil. This vast customer base is a testament to the trust of a reliability that Santander has built over decades.

In 24, we achieved a remarkable financial Firestone like €12.5 billion in net profit 3.2. Only in Brazil. These results reflect our diverse business model, our commitment to innovation, and ability to adapt to changing financial landscape.

But beyond the numbers, what truly sets Santander apart? Your ability is our ability to manage complexity, especially in the areas that are often invisible, often invisible to the public eye. And one of the most on the most complex and high impact areas of our operation in Brazil is a judicial order, a critical area that ensures compliance with legal demands, guarantee the answer inside the terms and and the and protects the rights of customers institution.

And let me give you a sense of scale of the size in Santander. This process, involved 1000 legal process per month and involving ten up to 100 pages, per document and multiple business whole with a manual signatures, verification of the bank products up a fiver years in in the in the lawsuit Wheat and specialized workflows.

Our legal operations follow a structure value chain that includes, request intake via email, physical documents, or API's with channels in government and and others. Document checking, screening to determine the banker role in this in this process, registration into the legacy systems, to our government systems. And that preparation includes evidence guaranteeing, and legal rezoning, consolidating issuance of the final response and signature and submission, often requiring a legal authorization.

In 23, this process involved overtime employees across various stages. But didn't stop in here. In 24, we made a significant strides in automation. Over 200 cases automated. Here were our objective was ambitious. We had to automate all this flow here and here, all the FTEs we have and every step of the journey on Pega. And our objective was clear to reduce, accuracy, in raw accuracy and reduce, the SLA. Time to response to the judge.

Here is an example of, unstructured documents. As we know, OCR usual tools don't have the the power to read an unstructured document because it's a positional solution. and with GenAI. When GenAI becomes available for us, we start thinking how we can ask for the the machine to give us these answers without having someone to reading the document. Because, a median the average of people reading a document is one minute per page, and we're talking about almost 200 pages. It was really spend really time here to, to change this, to read this document.

Here is a video. It's already showing the connector GenAI. This is the size of the prompt. All the information that we are extracting here from the fields for Pega our version of GPT. Here a little bit of the rules we write to the prompt, for he understands what we want, all the fields that we want.

And here is the historical occurrence that we had when we begin with ChatGPT 3.5, it was 75%. It was really lower that when we were expecting. But we have a goal at least 95% because humans and doing this the validation was hitting 95%. So we need at least 95 to to make the automate successful.

Here we have also help with the Pega connectors to help us understand and communicate with other systems to have all the information we need. And then, when was launched GPT four, we had to 70 to 80 7 to 96.7% was began to to give us an accuracy. Very good. We do another prompt changing. It was drop 11. 3% and another one with users giving us feedback about the fields. We hit 98.5 98.7% of accuracy.

When we do this, we enable to all entrance of the flow become fully automated. The first interaction with the with someone with the the flow was after doing all the analysis, all the checks. What is the information about the judge? Information about the value of the the lawsuits. Information about Santander is not part of the of the lawsuit, and was an enabler for us to start to automate the rest of the the flow.

Here is another video. This is just let me back here just to say this is the actually prompt working. We upload the documents and show you the how great was this tool when we first begin and start. So he will upload the document and the prompt will be called and answer all the fields. This is a small document, have 50 pages But all the fields are fulfilled here automatically. And in the document we can see where these information were. Where he extracted the information to fulfill all the fields. Look at a unstructured document with a very high complexity because it's a writing of of judges. So it's a great result for, for for our automated here.

Yes. And here we have, after we start to automate everything, we put some RPA here and start to document a document consolidation. We have this goal to end up to be fully automated. The the workflow. Here we have an ongoing process. It's already automated. The the part of the signature in 2004 2005. We have to finish these two steps. Signature is already digital and answer to the judge giving the the response or the better response for that judge for that kind of of document that that kind of lawsuit to give the better answer, put GPT here also the same connector and ask for him to give him an automated response. And the lawyer will only have to read the response if it's feasible for despair to dispatch with the judge.

And looking ahead, our vision for the future is even even more ambitious. We are moving toward a fully digital legal process, with us using agent keys to to document checks and legal opinion generations. We decided to begin using these keys and RPA boots and to to reading to to legacy to registering in in legacy systems. Digital signature workflows integrate end to end and real time monitoring and quality assurance. With our dashboards. This is the future of of complexity operations in in Santander with Pega is leading this way of, of a legal process and other complex processes of, of a core banking.

And we see today in PegaWorld presentation that we have now available agents to do this, these steps that was manually, we put OCR, but now we can evolve this and get even faster and put agents in every step of the way in a management agent for we can fully automate the workflow.

This one, we interpreted more than 2 million pages of unstructured document like you see in in in example of, Ricardo, show us the SLA was less than 30% and now with the GenAI is 99% of case delivers Wi-Fi. We deploy the GenAI to interpret the legal documents and RPA to document to automate the repetitive tasks and OCR occurrences at the screening stage reached over 98%. This transformation is not just about efficiency is, it's about quality and compliance and scalability in scalability.

And in conclusion, Banco Santander is more than a global bank, is a technology driven, legal, robust and customer centric organization. Our ability to manage complexity at scale, from serving 171 million customers to processing thousands of legal cases of month of month, is what defines our leadership. That's it. Thank you. And open for for questions and Soon and doubts. Thank you.

Q&A Session Thanks very much, guys. I think you've honestly had an insight into the future there. What Santander has done. Honestly, incredible in terms of the amount of manual, complex work as well. Not simple work. Complex work. It's very important to the bank that you've, re-engineered, automated, with incredible results, like 99% service level from what you said, 30%, 30%, 40%. You got to 99% service level. Brilliant. Before the all the automation scale, with, put the the new features with GenAI and others arrive in 99.5% of. Honestly, I think another round of applause for the incredible results. Seriously. Thank you. Thank you.

So open to the floor for questions, or comments, thoughts, anything? If not, I'm dying to ask one and I'll open it up. But please.

Question 1 - Eric Kent, Northwestern Mutual: Eric Kent here from Northwestern Mutual. Really impressive. Like, again, being able to take all the complexity and unstructured data. I was wondering if you could talk a little bit about the process to refine your prompt, because it was only up there for a little bit, but it looked like it was about 18 paragraphs long. It looked pretty involved. Can you just kind of talk through how you started, how long it took you to refine it? And I mean, was did it end up being kind of that that big and interested in the process and going through that?

Answer: Good question. Yeah, we spent like three months, making the prompt and we haven't tested them until we go to production because of course, in homologation is where 100%. But when we started, the production was 77 something 7.2. And we start to use the the users to tell us if the field was right or was not.

One of the fun facts we have we had to ask for GPT, stop inventing answers and try to search on web about information about the case. We have to explicitly said do not make an answer. If the answer is not there, leave it blank. We broke the the prompt in each question that we want to be answered, and giving specific instructions for each task for each question.

So if he needs the name of the judge, we ask, is the one responsible for doing, the the judge's job? Is only one person. Cannot be more than one. When we have to. If Santander is part of node, we have to to say that we are in this place of the the document or if they are just asking for information about someone. So we have to specify for each field what we want and resume for a word or two, because we want to be automated, so could not be a long, long text. It must be a word or two, for we advance on the flow automatically.

In the beginning, for 3 or 4 months, we need for for the help of analysts to correct it in the, in the, in the screen and and with a control the prompt engineer turning the prompt and turn again to to rework and and increase the the the occurrence of the process. So to arrive in in 19 8.5%. In in 2 or 3 months. With GPT.

Follow-up Question: For just a quick follow up from me, I saw that it's amazing. You got to 98% accuracy. And you were saying people were 95% were your. When was it that your risk or legal teams were happy to release and use it full in production? Did you have to get to that 98% or was it before?

Answer: No. We started to give a a percent of the cases directing by for GenAI. So they are using doing manually, and we start to direct a few cases for the user for he start to reviewing and see if it was or not. And this is the the Pega to has this rework information that how many times the case was going forward and backward. And it was 95%. So when we show it that this small amount of cases was doing 97, 98%, it was better than people doing. And they have to accept that is much better. Here is the data. And it's 95 against 97. There is no need to to go against it. Very good.

Question 2: Other questions. Comments. Please gentlemen. Very impressive use case. Thank you for the presentation. A quick question, around the topic of the do you continuously measure the accuracy of the generative, and the, the, the way how the data is taken out and also how you generate the. The if we continuous, company because we have a goal to hit Reach 99.19. But now everything we make is it's easier to go down that go up in the percentage. So we continue monitoring. We are doing some prompted changes and we want to automatize the the test, the test case for granted. That is never going down, only up on the accuracy. And the other question was sorry.

Do you take random samples out like, I don't know, 50 out of 1000? Then measure manually and compare that against what the AI does. Or we we build our rule set on Pega that we upload the same document A randomly on the flow, and we validate if this answer is the same every time. So when he changes something, some some answer, we stop to review if there were any change or any deploy. Who has broken the prompt? Interesting. Thank you very much.

You also have people doing some of that checking processes as well as the Pega workflow. Also, randomly each 10,000 5000 cases, we we put someone to open the work desk and look if the answer is right. Just for for be sure that it's nothing different because, if our judiciary in Brazil changes some some information, we have sure that he is already still are grabbing the the right information.

In the future. We think, using another model to validate the response of this actual model. So this we we projected this this function in the future. But now the the analysts help us to adjust to these these changes, these little changes in the process.

Question 3: Just one more question for me. What was the hardest part of the process in terms of the change management exercise? What was the most difficult.

Answer: People. Okay. For sure. It was really hard to convince them that this is a tool that works. It can be measured because everyone says that this is something. I don't know if this answer is right, but we had Pega flow to show us if it's right or if it is wrong. So we have to bring data every time that we made a deploy to show that still this information is still right now the flow is answered correctly. There's a messenger dispatch directly from the judge automatically. So they was really anxious in the beginning, but after a few times they start to consummate with the idea and it's working. And now we pretend to do more, more things to automate the rest of the flow.

Yeah, and I'm thinking with results like these that you've had, surely your management is looking at you and say, can you please do some more?

Yes, we have to do on the core banking information because OCR is not for only for Juridico. It's I believe it's one of the hardest. But we also have in any kind of money. Loan that we we do, we have to check the identity of the person, the documents. If he's the owner of this apartment, if he's the owner. Probably the next to flow that using GenAI the flow about, cash process. So the implement of caches. It's a very difficult, flow, with, Thousand people working there. So the next, flow using this same. The same solution of, of a GenAI. Santander. Very good.

Question 4: Any other questions? No. Well, I just wanted to. Oh, sorry. There's another. Question. Great presentation. I have one question. When you upgrade to a newer version of the model, how do you test it again? Or how do you measure your accuracy when you upgrade to a newer version of the LLM?

Answer: That is an issue we had in all versions, that we have a flow that is 47 case types, and we have to test the flows manually. But the the GenAI we are extracting in real time. So every time the the user changes something on the floor, we put the the accuracy to last one and then start to grouping everything. So when it hits less than 97%, it gives us an alarm to to watch the solution that he probably has some problem.

Question 5: I have another question, if you don't mind. Sure. Have you put any guardrails in input and output, to stop any kind of, hallucination or any kind of, like, security risk?

Answer: Yeah. The the the two of, GPT has his own guardrails in the video. Headers. We are finishing headers on LinkedIn for any other information that we need that we we couldn't answer or you couldn't. A thank you here. But we have them in the in the video, it shows that there's a person of imagination that you want to own the on the floor. So you have to be to 100%. You want to be creative or 0%, you don't want anything to be created. And we specifically said on the prompt, it's the first line, please do not imagine things and answer only what we are asking. Thank you so much. Thank you.

Closing: Anything else? Look, I'd just like to finish with thanking you both again, both for traveling here and sharing your story and honestly showing us an example of what's been talked about on the main stage in action that's been done in the last 12 months. So brilliant. Congratulations for that. And thank you for the presentation. Thank you. Thank you, thank you. All.