PegaWorld 2026: Sun Life Delivers Instant Answers: Empowering Agents with Knowledge Buddy

Good afternoon. How is everybody? Did you have lunch? Okay. Thank you. Today my name is Nirban Sanyal. I'm a director of business architecture in Sun Life, Canada. I have with me. I'm Supreet. I'm the development director from Pega or from Sun Life for all the Pega applications. Okay, so today I'm going to talk about Sun Life and Knowledge Buddy. So the use case that I'm going to share is basically very simple. How do I give the right information and right answer to an agent at speed? Now, before I go dive into Knowledge Buddy, I want to give a brief introduction about Sun Life. Uh, Sun Life is a Canadian insurance company. I've been here for last over 150 years. We operate in over 25 countries. Uh, we have 70,000, over 70,000 employees. For the last 150 years, our motto has been really very simple. How do I.

Help clients achieve lifetime financial security and lead healthier lives? Is that okay? And, uh, so as we are a global company, we operate in over 150 countries. As we scaled, the real bottleneck we saw was accessing knowledge was the absorption of knowledge for the agents who are working in our operations, in our contact centers. So let me tell you about what the agents are actually facing in Sun Life. So the agents we because we are old company, we have a lot of knowledge systems, but all those knowledges are dispersed into different systems, SharePoint, confluence, and other sources. What we don't have is how do I present those information to the agent at the right time? Users used to refer to different sources of knowledge, multiple tools, knowledge sites. They have to refer. And what we typically call the swivel chair activities. Then again, our knowledge articles were not that good either.

Knowledge articles were built for completeness, was dense and not built for speed. Not built for speed, not built for speed. And that's. Fine. Now, as you imagine, with all the fragmented sources of information, there was a lot of impact to the businesses. SaaS were impacted. There was a there was a lot of escalations because of the lack of knowledge. Think about as an insurance company, when I'm handling a case, I might need a specific information for a specific client for a specific product. If I don't have those information, I might be doing something wrong. And that's why there was a lot of escalations. And, you know, at the end, employee morale was also getting really bad. Then, uh, back in 2024 and 2025, as you have noticed, uh, the large language models started to come up in the market and become really popular. And that gave us an opportunity to ask the a question about what if.

Instead of searching for answers in the. Knowledge articles, we actually give the users the ability to ask a question and get. A trusted answer back with confidence. And that's built the foundation of our solution. Funny enough, I was actually thinking about. I should have had some Knowledge Buddy or a GenAI capability to ask the question on, you know, how to enable this thing or how to make this thing work. So that's what probably all our agents would be looking at. How do I ask questions when they're processing the case? How do I ask question? Or how do I ask the relevant question and get the response, which will help process my case faster? So like I was explaining, typically it's a swivel chair activity. When they're processing a case, they don't really, you know, know immediately as to how it has to be handled. They'll have to go through all the knowledge articles, understand for this particular policy or for this particular relevant claim, this relevant step.

What do I do? So there's a lot of focus on going through the articles and doing the manual wheelchair activity, which is where Knowledge Buddy actually comes in. And the great thing for us was that we are already invested in the Pega ecosystem. We had a knowledge management application, which has roughly around 50,000 plus articles, and we were able to harness all those articles, feed that into the Knowledge Buddy application, plug that Knowledge Buddy application to our existing Pega applications. So that's what gave us the advantage of having the knowledge articles or access to those knowledge articles through the question back and forth for the end users, the end user can literally ask the question while they're working on the case. How do I process this? What kind of data is required or what kind of validation is to be done, and it will be seamless for them.

They don't have to swivel chair and go into a separate knowledge management portal to go through all the bulk of the articles. So that's what actually gained a lot of process efficiencies. Now this is just the first step in achieving that kind of efficiency, right? So the first step is to give them the access to the knowledge. So this knowledge which you cannot access easily is probably knowledge lost. And we want to give them that seamless ability to accessing the knowledge. Like I already highlighted it. It's an AI powered system. What Knowledge Buddy brings in, it brings in a rag tool where the data gets vectorized in the back end, and you do have ability to prompt engineer it, configure all the prompts you want in the back end. So that will be additional padding, but you could also have the users ask the questions at real time and in the back end, with the additional padding of the prompts, we can confine it.

We can limit it, and we can also make it more efficient. So that just to give an idea about how much change, how much benefit it was for the users. Right now we will talk about that. We scaled up to 1000 users. We plan to be around 2000 users by year end, but usually before Knowledge Buddy came into the picture, it used to take about 1.5 minute to search for an article, locate a particular section, find the information, go back to the screen and do it. So we are processing around. We are handling around 3000 4000 cases maybe per day. And that's an average if I most of the cases will need some kind of knowledge search that translate to be about 2000 3000 minutes per day. Saving. So that's really what really drove the user adoption. First, when we introduced, we had to build trust in the system people were still using directly going to the site.

The great thing about Knowledge Buddy is that it comes with some out of out of the box reports that you can track which users are actually big users of the application. We noticed few users are really super users, and most of them did not even use the system in the first couple of weeks. So we actually tracked back to the team leaders and made an effort to use incentivize the users to use AI system. One of the other things we noticed is that when we launched, we had an approximate acceptable accuracy. I say acceptable means we are in the insurance business. We have to make sure the information is correct to the level of risk that you're taking. So it was about 92%. We quickly raised up to 98% accuracy level. That's when we decided to to roll out to the more users. One of the reasons, you know, it went up, we worked with the users for better questioning.

That's one thing. And the second was highlighting was, uh, prompting is a big portion of your investment. Whenever you are using AI to, uh, you rate your answers to your knowledge base. Prompting comes in handy for, especially when you have multiple languages in a country like Canada, you have to provide some answers in French as well. Yes. And the beauty of it is actually, while in our effort to get to the accuracy, we identified that a lot of it is dependent on the LLM, which is being used by everybody in the back end. And the beauty of it is that Pega gives you the ability to configure the LMS, which you can write and there's a data residency part of it. There's a regulation part of it. So we also have to work with our security teams to understand whether these LMS can be used in Canada or not.

But yes, Pega gives us that ability and flexibility to configure different LMS. Again, if you don't like the outcome of that particular LLM, then you can choose another LLM. It's not vendor bound. It's not really bound by, uh, you know, the actual product limitations. So that's the flexible part. If I were to do it traditionally, you know, every single LLM which we are going through would probably need a little bit more coding to be done, a little bit more elaborated evaluation to be done. But from Pega perspective, the ecosystem already exists. Everything is going through the same security protocols. Everything is being governed by Pega in the back end as well. So we didn't really have to change anything around it. It was very simple to switch from one LLM to another LLM. We did have a few hiccups in terms of how we are going to enroll this one out, especially in terms of the governance.

Now, Sun Life and Canada, we both are highly regulated ecosystems. So how do we roll this one out? So Pega buddy was probably the easiest thing here from the Pega perspective, Pega buddy was probably the easiest thing for us to do the prototype. And we had our architect do the quick POCs and socialize it, you know, give all the necessary energy to building it forward. But then the security and the risk governance aspects of it needs to be vetted out. And that's what took a little bit of longer journey for us to get it rolled out. That's one of our key lessons. So if we are implementing any AI based solutions, the primary, you know, aspect would be to look into the governance and go through that journey of how is it going to look like and how is it going to behave in production.

Most often than not, we undermine the complexity of deployment. Prototyping enGen, AI with Pega or with any other tool is pretty easy. Even Blueprint is easy, but taking them to production is a little difficult task. Not that it is impossible or not that it is overly complicated, but we do need to cross those hurdles, right? So yeah, just want to add that one of the things you want to keep in mind while extending the AI features to the users is cause less disruption to the user. So basically integrate the user, integrate the AI into the platform, into the system, not a separate tool, not a separate signature task, but into the system. Contextual and AI initiative answers. Correct. And that is why our primary effort was to bring AI to the existing ecosystem, which we have. Yes, the Agentic AI processes or the evolution of Agentic AI solutions is something which we are evaluating and we'll be taking them forward with Pega too.

But then the first attempt was to bring AI to the existing user base, and get the feel of how they are going to interact with the system for us. And this is one important learning, which I personally had when we originally launched the pilot. Everybody was very, very happy with it. They were like, oh yeah, I can get GenAI feature in the Pega itself. So that's great. But by the time we reached 2026, now everybody is very familiar with the chat bots or the chat of the world, the clods of the world, and they are coming back and asking for new features. Okay, can I get this little thing? Can I get this little thing? Why do I have to type every time? So we started working on a few enhancements as well. Asking or training somebody to ask the questions is difficult. So we gave them an FAQ list which is pre-populated for them based on the context of the case, which they are in so small things like that.

I think incremental value is what we focused on. That way we increase the adoption as well. Next slide. So key learnings, like I already said, you need to iterate through these, you know, solutions. We need to get the feedback from the business users. AI is great. You know, all the features it brings in is really great, but the adoption is what is the tricky part. The user should also be in a position to actually consume it and consume it more meaningful ways where it can actually bring in the efficiency which we are calling out for. I think that was one of the, you know, important things, which we kind of focused on socializing the overall GenAI solution. Then pilots, I think we started with pilots like Nirvana already highlighted. That initial pilot gave us the kick start with the ten users and then scaling up to 100 users, 101,100 users.

Plus. Now we want to scale it to 2000 users. Governance is you have to build the trust with the governance team as well. And we have to explain things very, very methodically of how Pega is doing things on the PegaCloud side, as well as how our applications are interacting with each other. So the ecosystem we have is that we have the knowledge management application on prem. We have our Pega application consumer applications on prem, but the buddy is sitting in PegaCloud. And all these have to be integrated and all these have to work seamlessly. And that is where I think the explanation to the governance teams or the security teams is really important. One thing which we realized is, yes, you can give buddy. You can give any ChatGPT or GenAI kind of feature, but it all depends on the actual content which you have created, right?

So we have to understand the knowledge article complexity as well. What is the quality of the articles which you have created, which is what we unearthed after implementing body that. Yes, some of the responses are bad, but they are accurate. They are actually fine based on the content we have created. So that's when we went back and we are like now, reevaluating the articles we've created, improving the quality of those articles. We also have a process wherein we are trying to rewrite those articles using GenAI as well, so that they are written in a proper templated format, which can be usable by body. So that is a huge learning because people will be complaining that body is not really providing the right answers, but the right answer, like the LMS themselves are giving the right answers based on the context, based on the content which is sitting in the articles.

Right. And then when we actually take that back, it will be an advantage for us to refine our quality of content, which is also one of the features in buddy. Buddy has a feedback mechanism and we extended that feedback to accommodate or report based on the actual feedback from the user as to what is not working in this response. So that way we can feed that back to our knowledge management team, and they can update their articles based on the feedback we've gotten. The feedback is really important because we initially had feedback of good or bad, but we also wanted to collect what is actually bad. So we rolled out a new feature to identify where they could select a preselected option of what actually is the article wrong? Is a buddy not returning the answer, and we could look into ways to fix them.

Correct. And like I already highlighted, understanding and developing an AI governance model is really important. So working with your security teams, working with your architecture teams and emphasizing on how different layers sit together, and what kind of governance and what kind of logging or monitoring can we put at each layer is very important. It's a good thing for us because we were already invested in the Pega ecosystem. So we had the luxury of using all the tools which are at our disposal, and also the PDC logs and everything. So they're all well governed by Pega and they're very transparent and they meet our security guidelines too. We did have concerns or, you know, issues around the restricted data and things like that. So we've not subjected any of our restricted data through those knowledge articles. The knowledge articles were cleaned before they were put into any kind of LLM models.

So the PII and restricted data scanning was done prior to us doing any kind of evaluations. We have to make sure that we are actually explicitly mentioning or having the guiding principles of how we are going to implement your AI, whether it is single question, whether it's a conversational back and forth, how effectively somebody is really using and how is it relevant within your Pega application or within your ecosystem, which is sitting. Like I said, I think the primary focus for us was to bring those gen AI solutions in the existing ecosystems, have them as AI assisted solutions, so that the AI can provide you the right guidance, right assistance. The decision making is still with the humans, but it can provide you the right suggestions and recommendations based on your case context, based on your knowledge content. Can I go to the last slide?

So where are we actually going next? We want to keep the Knowledge Buddy roll out to many more users. We are, we are. We are 6500 operational users for Sun Life for Sun Life. And you can think if we can roll out to 2000 users, pretty almost 30% of the population. It's a pretty big area. Um, we we have a lot of Pega applications. Pega built on the Pega platform. So we want to use the surface area that we have to utilize Knowledge Buddy where required. We, um, we want to, um, we are working on different use cases where we can provide more contextual conversations. Right now we have a question and answer format. Not a lot of recent conversation history. We want to change that. We want to maintain a history. Users are actually excited. They actually want us to help with the communication, drafting and communication.

Just like you go to ChatGPT or someone draft communications, we could use, uh, Knowledge Buddy. Um, we could use, uh, Knowledge Buddy for those kind of different purposes. Uh, but right now for 2026, our goal is to low scale disruption. Um, increase the surface area for the Knowledge Buddy, consume more knowledges and also extend to other knowledge sources like SharePoint, like confluence. Uh, right now we are very focused on the knowledge management system because it's easy to fit them to the body. But now we are moving on to other different sources as well. The foundation is what we are currently setting the stage for all the future innovations which we can bring in and help the user, not only just the user in future, if at all. There is agentic AI processes who wants to harness the knowledge articles or the content knowledge articles. I think that is what the stage we are setting with this particular initiative.

One additional thing which I do want to call out is that it is really great for us when we are having a knowledge management solution, because if you are authoring and you're updating your knowledge articles on a constant basis, there is sync connectors, which are available from Pega, which automatically feeds into body. So you don't really have to worry that the information is getting stale on the body side. This automatic sync connections work beautifully well with the ecosystem which we have on the consumer side. The plug in components which are provided by Pega. They also work very well with the existing applications we have. And you could embed those Knowledge Buddy plug in components within a case, or it can have a separate pop up however you want it. We've done different kind of implementations and that's what our evolution will be. Right. So how do we really make use of Knowledge Buddy in more effective manner?

Right. Currently we are in a situation where we are having users type the questions, but the next step is to auto populate a response based on the case context already, right. So you have a case context. If your case has the right data then you can go back, make the queries from body, retrieve the content, have that pre-populated for the user. Yes, they can still go ahead and ask the questions and we'll give them the response. But the pre population would give them that one step further. In addition to that we have email bots already implemented in our Pega applications. So email bot does all the NLP logic and it extracts the keyword constraints keyword content from the email. And it does the case triaging case routing for us. We want to actually we were working on it. It's probably live already. So the email bot solution plugs in with the Knowledge Buddy.

We extract the content from the email, go to Knowledge Buddy, get an automated response back, and we pre-populate the response for the user. We are not really automated that end to end process flow yet because we are doing that field test if the responses are good, if the content is right. So there is that manual intervention still. But the future could be that all the inquiries which you get, you know, what's my withdrawal limit? What's my contribution limit or am I eligible for this claim? AM I not eligible? So all those inquiries and the load from the operational user could be taken away. And they could focus on the actual, you know, good financial works or where there is more decisioning required and that agentic or end to end process flow. I believe we can achieve that very seamlessly. So right now we are just giving them the pre-populated response with our email bot solutions.

Like I said, conversation or sorry, context aware knowledge assist is where we are focusing on pre-populating that data from the case perspective. There is a great use case for us in terms of Voice AI as well. So we do have a call center application which we are plugging in with Pega Voice AI content, but then we can seamlessly integrate the Voice AI with the Knowledge Buddy as well. So the Voice AI engine will do the NLP and extract the text content, we will go back to the knowledge management, understand what has been spoken during the call, and it will give you the suggestions, the recommendations within that calls context so that the customer rep, when he's talking to the customer, he will get, oh yeah, this particular customer is talking about so and so policy. So these are the type of actions you can do.

So there will be case suggestions. They can do some script adherence as part of the Voice AI feature as well. But the biggest win would be accessing the knowledge content real time without having to type understanding the voice, and then provide that suggestion as the call is actually happening live. So that is something which we are looking forward to build as well. So we started the POCs. We started the groundwork on that, and like I already said, no disruption. So we don't want to do Big Bang. We don't want to impact all of our users. Change management is always a constraint or always something which we have to deal with. So plugging in knowledge assist feature wherever it is required, whichever application it requires. With Pega without Pega applications, we do have, you know, other non Pega application users using Knowledge Buddy as well. So and it is seamless right.

Like you can get the responses via APIs. That's the advantage of having the right solution from Pega. It can integrate seamlessly with APIs. We do want to expand our content sources like already highlighted. Currently everything is fed from the Pega knowledge management application, but we are working on bringing in the content from SharePoint directly. There are loaders which are available from Pega which we can seamlessly bring in. So we are also bringing in data from confluent, and that will go to Pega. And we can serve up those knowledge responses as effectively as possible. I think that's it. We have. So that's mostly it. I think we are, you know, happy to answer any questions if you guys have. And you know, we can have more discussions also. Please go ahead. Mike's. Just a question for you about the content that you had before you installed Knowledge Buddy.

Um, what was the shape of that content? Was it pretty, pretty well structured, pretty clean? And what activities did you have to do in order to get it to where it needed to be so that you could go to that first pilot? So some of it was clean. So we picked up the ones which were cleaner, I would say, for the initial pilot. Um, so there are ones which are not clean. So the adoption is slower on those side. And we are working with the teams to refine the content. But yeah, when we did the initial analysis, we picked up the, you know, section of articles which are much cleaner, which are more precise and which probably give us the more value as well. Yeah. It's a, it's a continuous process. So we had choices of waiting for the content to be cleaned. Or we could go live early with the content that we have and parallelly work in the background to get the content better.

We chose the first option, go live earlier. Um, we have an Excel we had in mind an acceptable accuracy level. If we fell below that line, we would not be releasing to the rolling out to the more users. So we, we always paid attention to the accuracy level. We conducted accuracy. We calculate accuracy level per week to see if we are dropping below 92% or something, which was an acceptable risk. Right. And that was the important part, right? Like, I think the primary part was that if we wait for content to get refined fully, then we will not really see what value this is going to bring in. So you do definitely have to start with a pilot. So identify the small set of, you know, teams or you know, content which is cleaner. And that will drive the adoption because that will also encourage other people to refine their content and then they'll come.

That's what that's how it happened for us too. Once they see that there is value being generated and they have the right content in place, it encourages other people to create their content or update their content in the right way. Just to add to that, there was instances where users reported incorrect answers. We actually found out the knowledge articles were incorrect. So in a way, Bardi started exposing the gaps in those articles as well, which was very difficult to do manually. Going through like our initial load was something around 30,000 articles almost. Yeah. The other thing, which we realized was that people some people don't really complain, right? Like they're already used to the ways of the working styles today. They already know that these articles have some gaps. They already know that. Okay, I need to ignore this particular section in the little article, or this article actually takes too much time to load.

So. But with buddy, there is a comparison happening saying that oh yeah, buddy is responding this way. The article has this way article loads in like 10s 15 seconds. Buddy provides a response sooner. So that's the kind of actually healthy competition is what we realized with buddy bringing in that kind of a feature within the ecosystem, people will automatically realize, yes, there is an opportunity for improvement. Everybody is working towards it, so they can also contribute towards it. Yeah. Um, you spoke about accuracy, right? So I have two part question around that. So what were some of the parameters or methodologies that you guys used to measure the accuracy? And second is like what was done besides retraining or, you know, refining the articles to improve the accuracy of the answers. Some of those. So we launched a feedback. Obviously there is a feedback option in the body.

You can provide feedback and the reason for the feedback. If it is wrong. So we tracked those metrics. We had, uh, you know, we, we have different kind of transactions, as you would imagine in your companies. Also, there will be financial transaction, non-financial transactions. We paid a lot of attention to financial transactions because those are money involved, non-financial transactions. You know, we are okay. So we we score them properly. So we, we, we ran reports to how many articles were searched, how many answers were wrong, how many answers were particularly for financial transactions versus non-financial transactions. And we wanted to have a 90%, uh, you know, accurate answer. The simple metrics is the article needs to be accurate, provide the information that was necessary for the user to process the case. There are other things as well. It is taking us around 10 to 12 seconds.

Um, that was also an acceptable number to to return an article, but that's a huge improvement from like the seat for article to be written for 10s. But they used to sit for one, 1.5 minutes. So it's a. So those are two, uh, you know, right or wrong. And then the how much weight you are giving every second will count ultimately. We did a few adjustments, right? So like I was saying earlier on that the prompts need to be adjusted. They need to be padded appropriately as well. So we found that by doing appropriate prompt engineering, you could improve the accuracy level. So that's the first again dig at trying to improve the accuracy. Now what Pega does is that Pega takes the knowledge content, chunks the data into smaller content, and vectorize that Content and then it reaches out to the LLM. Right. So we've realized that with working with.

And James earlier too, that the knowledge content needs to be updated and rewritten as well. So we worked with the Pega folks. We also worked with our business folks to distribute that knowledge article into smaller articles. And that way you could get more accuracy because the more refinement in the article, the better your responses would be. Typically, what has happened in the past is that everybody, whoever has come into the system, they've created one huge article. They keep adding more and more stuff to that same article. It becomes easier for them to look at one single article, have that open and navigate through that. That's that's the typical human nature. But that is what causes the problem, right? Like when you're trying to access a huge article, if you're loading it on the knowledge management portal, it is a memory hogger. But on the GenAI side, it really consumes more tokens because now it has to go through the huge article to give you the right answer.

You have to find the sweet spot between the chunking and your paragraph in the articles. Once you find that sweet spot, the answer starts to flow up automatically. AI likes those answers to be in a single paragraph, like if you are dealing with something. If those are in the same chunk, you will get almost a good answer. If they are dispersed into different chunks, not going to get very good answers. One additional thing which we did was to keep an eye out on how frequently or how much time that user really takes to resolve the case to write. So there is a possibility that I could be answering or asking the question wrong, and then it could be giving me a response, a bad response, or a different kind of a response that would lead me to ask more and more questions. And then hence, more and more time is spent in processing the case.

So we started looking at that as well. Like with the GenAI capability, is it really getting better in terms of processing or not. So we had the additional feedback loop coming in, the reporting coming in from Betty, but at the same time from the processing aspect, also, that monitoring helped us bridge the gap or identify how long somebody is really taking to process something. Good afternoon. Good afternoon. Yeah, I have a couple questions. Um, you mentioned about structured and unstructured data, right. We can get response from structured and unstructured. Can we get answers from video files? That's an interesting question. I don't think we have implemented um responses from video files. Um do you know. Yeah. Um so video file is something that, uh, we have not been able to extract yet until past releases, but now we do have one solution in place where we can extract the content with the help of LLM.

So there are some LLM models which are capable of extracting content from videos as well. So those we do support with Knowledge Buddy. Today there are some limitations. The LMS context window is limited and but it works. Yes, I'm I'm product manager for Knowledge Buddy. Yeah, yeah. Okay. You can definitely come to Knowledge Buddy booth later as well. But in this session, if you have any specific question for Supreeth and Nirvan. Yes, let's use the experience. Thank you. Thank you. But we did, uh, I think, uh, accidentally uploaded a dog's picture or something like that. Yeah. Video file might work, I don't know. Hello? Hi. I'm from Canadian Tire Bank. So. Okay. Hello. Um, I, we are actually in the process of also piloting Knowledge Buddy. And I'm just wondering, Do you have multiple buddies or do you have a single buddy like multiple by department?

Glad you asked that question. So we have multiple buddies and that's another great feature from a buddy standpoint, right? So you could create multiple buddies. We have created multiple buddies based on the different line of business we have, and they have separate data source mapped so that again, the data is segregated. The buddy responses are segregated, but more importantly, the prompt management for each of the line of business would be different. Hence, having separate buddies gave us that advantage. In addition to that, we are also using multiple buddies for testing purposes, development, and testing purposes. So what we do is we create a test buddy for testing, you know, things like that. And whatever prompt management changes we want to do. And once we have finalized it, we kind of do a pseudo promotion and we update our stage buddy, or we update our production buddies appropriately so that multiple buddy is very effective for us both from a development standpoint as well as from a production standpoint.

So one of the things to keep in mind is that we had this question, but I don't think we implemented, um, we ended up implementing those, uh, restricted articles like some users may only see some articles if you ever run into those issues, multiple bodies probably is the way to go. You should not be mixing restricted and unrestricted materials in the same body because the body doesn't, you could train them, prompt engineering them, but it might look like some restricted data. Correct. And that's also something because for restricted content we are not allowed to go out from Canada. Right. So all the LMS. So we can only call the LMS which are hosted within Canada. And if you do those multi bodies, you can control which body to call from your front end application. That's a pretty standard configuration mechanism, but it gives us that flexibility that if you're accessing body from a restricted client case, then you could go only to the restricted LLM, which is hosted in Canada.

Right? And then if you're accessing it from, say, a regular case, then you're probably better positioned to access the latest and greatest elements which you have access to. Thank you. My other question is just around the handle time. Have you found a reduction in handle time in the with the teams that are using it? We are actually running some analytics right now. There is a Product Owner for for buddy and we are constantly. There has definitely been some reduction in the handle time just because we needed to search a lot of articles. One of the other things we noticed that, um, obviously trust in the system is very important. We are still working through a lot of users who are kind of not, um, going into the buddy and asking questions the way we would like to. Um, we'll Will continue, I think, to work on those ones, but I, um, definitely some improvement in the average handling time as well.

One of the, one of the features which we implemented, which actually helped us there is the FAQs, the standard set of questions, which anybody would probably ask in that line of business is what we gave them. And in that way, they don't really have to write or type the questions. So that really helped us a lot. Yeah. So did you say something? I'm sorry. So with the accuracy of the answers being dependent on how the question is asked, what's the most effective way you went about educating the users? And it's kind of a two parter. So like the more tenured users versus new users. What's been the more effective ways of training them or educating them on how to better ask the question the right way? So we actually, um, did a lot of demos and, um, I did not call luncheon learn because we did not provide lunch.

So it's a lesson learned. So a lot of question and answer sessions were there where we demonstrated actually people were using something like to ask if a Kansas City, right, just Kansas City and Barry would spit out something and they would do the same thing with the product. And we, we demonstrated what to do in Kansas City when the soccer World Cup is going on would be a more preferable answer to the body. So we started demonstrating that the articles, the accuracy of the articles, the accuracy of the results would be significantly improved going forward. Right. And, uh, one of the things we did targeted, uh, you know, manager interventions as well, uh, some groups were really using a lot. Some groups were not. So we specifically went to the users and ask, um, ask around like what may be the limitation? Trust is a big factor.

Technology actually worked pretty well. Like in one month we are up and running and we could have been up and running in the production in two months. But building the trust, getting all the reports ready builds the trust. You provide those reports back to the managers who feel confident that they can trust the system to work on a financial transaction. What also worked for us is contextualizing those questions. Now, the user may not contextualize it, but you do have the ability to contextualize things in the back end. So Pega does provide us the ability to pad those through the prompts, like filtering. Yes. So you can filter questions. We can filter the questions. We can add tags. We can, you know, provide those additional features for the users, like I was mentioning earlier. FAQ. So FAQs list will give them the way the questions are supposed to be asked, and that will encourage them to ask the questions in the same manner.

Right. But like I was telling in one of the previous responses to the prompt management changes, which you could do with the padding. That also helps a lot. So rather than just relying on the user to do the right thing, you can also try to influence by doing the right things, by putting the constraints around what that question is. Somebody could ask a generic question. This is also something which has happened. Somebody could just randomly go ahead and ask a generic question, which is not really relevant to that case. So but with the right tags, which are configuring, so then that restricts the response to be limiting your filter search criteria for that particular line of business for that particular case context. So you have to build the trust. I think trust is the most important factor of adoption. You have to trust the answer of the AI do I trust the answer for my financial transactions or not like personally?

So we'll come to that trust. Yeah. Thank you. Okay, thanks. Just one more question for you. So you talked a little bit about it as far as like how to build trust and some of the ways that you showed the end users the value that the that Knowledge Buddy brought. But did you have other change management strategies that you used and how did you measure adoption? At the end of the day, that people are actually using Knowledge Buddy and not bypassing the system and going through their old ways. We actually had a had done some reporting on how many knowledge articles typically people were accessing per case basis, um, which came out to be about 1 to 2 articles per per case. And so our, one of the metrics was if I'm using buddy, I should have the same kind of questions as to the buddy.

So when we first rolled out, obviously the number was way, way low. We over, we thought we overestimated the number of times that were going into the knowledge articles. So we measured against the baseline that we set ourselves that 1 to 2 hits. Um, and obviously, um, that will be not the same as before because now with buddy, you should be expecting to get the answer right at the first time. Previously they were probably going to the 2 or 3 different articles. So that is one measure, right? Um. So Pega also provides you with a seamless way of checking the questions which were asked. The response time taken for each of the questions and the response which you have received. We've also introduced a case ID, an additional parameter from a reporting standpoint. So we work with the Pega team to introduce that. So that way we can understand what kind of questions are being asked for a particular case.

So that really helped us really. Helpful for the businesses because business I'm not I'm not too much familiar with certain products and others, but businesses know the users know best what they are looking for And if we can tie them up with the case that they were working on, and most of its 80/20 principle, 20% of those responses were actually causing most of the problems. So you could fix those. So we had 2 or 3 different metrics. The performance metric 10s under acceptable accuracy level. And then the number of responses we are supposed to get per day as we roll out more users. Correct. And the addition of the case ID so that it will be contextual for the case they are working on. Okay. Thanks. I think we are over the hour. Maybe we can take one last question if anyone has. Yeah. Hello.

Good afternoon. Um, you told us you built upon an existing Pega knowledge knowledge base. Um, not all other companies are in that position. How long did it take for you to build up that knowledge base. And would you recommend to do that before, uh, using Knowledge Buddy? So we were in a better position because we had the knowledge management application in Pega. But Knowledge Buddy itself is not restrictive. So you can ingest any content from SharePoint, confluent, any kind of pages. The loader is giving you that flexibility to go to body directly. So I don't think that's a limitation. It was an advantage for us because of the auto syncing capabilities which came along. And I think there are job schedulers which will go ahead and do that syncing mechanism as well. So I don't see that as a limitation. It was easier for us from an adoption standpoint because it was positioned as an extension for existing knowledge management application.

Thank you. Okay. Thank you. Thank you everyone. Appreciate your time.