Ir al contenido principal

We'd prefer it if you saw us at our best.

Pega.com is not optimized for Internet Explorer. For the optimal experience, please use:

Close Deprecation Notice

PegaWorld | 38:29

PegaWorld 2026: Unleash AI Safely: Secure, Monitor & Govern Your Pega GenAI Ecosystem

Build trust in AI for mission‑critical solutions! Hear about a modern framework for AI safety, performance, and governance, demonstrating how probabilistic evaluation, and continuous monitoring, can enable trustworthy, auditable, and scalable AI adoption at enterprise scale.

I am Brian Feinberg. I'm director of product management and strategy for cloud and AI at Pega, and we're going to talk about how we can unleash AI safely across the organization. Before we dive into it any deeper, though, I want to get a sort of feel for where you guys are in your journey. So by way of hands, how many people are maybe in sort of a development or POC kind of stage with a genetic solution, not just Pega, it could be any framework that you're leveraging. What about in a production capacity? Yeah. That's fantastic. That's kind of what I expected to see. One of the things that I think I'm hearing consistently from customers is how do we build to a level of trust in a genetic solutions where we feel like we can truly unleash them to a widespread types of use cases and workflows across my company?

And I think three questions come up. Most commonly, people are thinking about how do I know that the solution is working? How am I going to continuously monitor and observe these solutions in the wild to make sure that they're continuously working the way that I expect? And when they have issues like any software solution will, what do I do to go about fixing them? So that's the topic for today. We're all going to be talking about how we're going to build trust into our genetic solutions. One of the things that I learned from listening to the keynote earlier, and I think it really resonated with me and I sort of folded into this presentation really well, whether it was MetLife or whether it was Kerim or whether it was Alan, there's no debate that Agentic and autonomous Agentic solutions are absolutely incredible. But in many ways, trusting them is still quite difficult.

So I want to think about the two points at the top, what I consider the trust problem. Okay, you have the same prompt, and every time you run that prompt against an agentic solution, irrespective of the model that you may be leveraging, you get a different output. Maybe you can't reproduce that solution and it becomes difficult to audit in that way. Okay. And then from a control perspective, when you're thinking about this at a runtime perspective, when an agentic solution operates, and maybe there's multiple chained agentic solutions together, decisions can land and actions can be taken before a human could potentially intervene. And so I want to talk about the four traits of enterprise trust that may be broken or at least fractured in some ways as a result of autonomous agentic solutions without proper orchestration and governance. Okay. One of the things I think about first is bolted on safety.

We are rushing to ship technology and solutions as fast as we possibly can. These models have made the velocity of delivery Absolutely incredible. So we're shipping as fast as possible. But as we're releasing them, we find that there may be an issue. Okay. So we think about it. Okay. I'm going to go and put an escalation path here. I'm going to add a gate. I'm going to add some human in the loop governance to solve these problems. And so we're kind of finding our way into this whack a mole, fix it, find it, fix it, find it, fix it, add governance on the fly. The second core trait that may be damaged in this process is the determinism problem. These large language models, by their very design, do not deliver deterministic results. And so in many scenarios that may be perfectly fine in a heavily regulated situation or in a healthcare situation, or something that requires strict determinism, it simply doesn't work when you're writing test cases that may pass today and the same test cases fail tomorrow with the same prompt that begins to erode trust.

Okay. And then the third trait is what I called undefined boundaries. What I like to think about is the microservices architecture pattern that we've known for 15 years. Okay. When we were decoupling systems and breaking them down, we were always trying to figure out what what did this self-contained unit really look like? Okay. What was its function? What did another function look like? And how do we sort of break them down into logical components? We're doing the exact same things with what agents can do. They're scoping what tools they have access to, what they can change, what they can modify, and so forth. And so it's not an insurmountable problem, but it's a problem that we have to consider and we have to solve for. Okay. So an undefined boundary problem again erodes trust. And then when you think about undefined boundaries and you extend that a little bit forward, what is the cost of an error?

Okay. You can have a significant blast radius. There are stories and I'm sure you've heard them. I've read them. An agent goes out, deletes a database. It deleted what, I don't know, 17,000 records or some number like that. And then the hilarious part was, is that it did. What a human would have done is that it replaced the data set and then lied about it. Okay. And so the cost of an error in these types of situations can get really, really high. And when the cost of an error is high, what do we do as software developers to go fix it? Well, we're going to sort of prune down what it can possibly do. We're going to take away maybe write access. Okay. You can't do that anymore. We're taking away this tool. We're going to change. We're going to modify behavior. And then ultimately you end up in a less effective solution because you've pruned down the capability set for that specific agent.

Okay. And so trust isn't something that you bolt on. Okay. It's not something in the agentic world that you bolt on. You've got to do it from the start. And that brings me to the first point that I'm really going to make, which is the concept of predictability by design and predictability by design, will make trust a whole lot more easy and effective for us. I want to start with a little bit of an anecdote. Everyone has heard of Security by Design. That is the first thing that came to mind when I've been talking to customers and hearing the issues that are coming up with fully autonomous solutions. Security by design. 25 years ago, I don't know some number like that. We we were doing the same thing. We were shipping solutions and then we were bolting on security after the fact. Now we're shipping agents and we're bolting on intervention and gates after the fact.

And so the same pattern is repeating itself. Okay. So the problem with bolting on security into our agents, the solution 15 years ago was security by design. Do it in the SDLC, do it in the design process, do it in the architecture. And we're sort of here today where now we're thinking about, okay, human in the loop escalation patterns, gates being bolted on after the fact. But instead, if we think about it doing it in the design phase, that's going to give us greater predictability. Let's start walking through each of the four tiles. The first one I want to cover off is thinking about domain expertise. OK, a large language model is not a loan origination officer. It is not a health care worker. It does not have the sort of domain expertise that we know is required to handle these specific workflows, either in heavily regulated industries and so forth.

Okay. So the definition of what does good look like or what does correct mean needs to be decided by humans. And that's doing it in the design phase within Blueprint, as an example, is a highly effective way of introducing that expertise. Okay, let's think about determinism. I don't think anyone would advocate that you don't want more autonomy within your genetic solutions. You want reasoning. Absolutely. We've talked about it on the main stage a little while ago. But how do you how and when do you decide where you're going to introduce more discretion or more determinism? Again, I think the answer is doing it at design time and then thinking about authority. Similar in nature to determinism. But what can an agent in an autonomous scenario decide to do alone, versus asking, in coordination with a human in the loop, or having an escalation pattern being constrained and confined within the case execution process?

That's where you're able to decide on what authority looks like within an agentic solution. And then at the design time, you're also able to control blast radius. Okay. We talked about this earlier with the cost of an error. If this agent goes wrong, who gets hurt and how badly? Right. And I think one of the things that I hear pretty consistently, if these are agentic solutions that are customer facing, people are increasingly worried about the brand damage that can happen with a conversational agent that can say and reason and think in many ways that acts sort of like a human in the way that it can speech and deliver. Okay. And so I think all of that, or the vast majority of it is solved by a design time scenario where you're able to take Blueprint and the fantastic capabilities that you saw with Infinity studio earlier today, you're able to capture human intent up front.

Build that into the design. Clearly define all of your decision boundaries. OK all the constraints that you're able to implement in the ideation phase, and then also implement all of your escalation logic, all of your human in the loop, all of the gates that are required to make agentic solutions predictable and trustable. Okay. And then at the bottom, you know, you can't fire an autonomous agent. Okay. And there's kind of a provocative way to think about it, which is if I go to work and I think about making a mistake and the possibility of being fired, I'm thinking about the material impact to me and my family. And I don't think agents AI agents feel the same way. Okay. And I think it's similarly true with retraining them. There are concepts of training models. Absolutely. But and maybe we'll do another survey. I don't think the vast majority of us are going out and training or fine tuning these foundational models.

We're taking off the shelf. Gemini. We're backing it with logic. Maybe we're using rag, we're doing things like that, but we're not retraining these models. We're taking Cloud, we're taking Gemini, we're taking OpenAI, and we're leveraging it out of the box. And so you're not, in fact, retraining these large language models. So I think that's a core pillar. How do we add trust through design time. And now we're going to pivot a little bit. And we're going to think about how we're going to add predictability at runtime. Okay. And so I think everyone in the room well knows we've been doing business orchestration platforms for a long time. We are, you know, have been rated very highly in that category and a fantastic solution for that approach. Um, and I don't want to think about Pega as a means for just Pega centric agents. I want this to extend into third party agents.

Okay, we're talking about land graph frameworks, Lang chain crew, AI your own frameworks that you may be building. And what does an orchestration platform at a case management platform like Pega mean to those types of solutions? Okay, so let's think about a free form autonomous Agentic workflow, 100% reasoning, no strict rules, very powerful, but they can also be quite brittle and out of the gate. There's no shared state. There's no replay of it. Okay. And so I think you'd of course would say, well, I can build that. I can build a shared memory store. And you probably would. Okay. But if you simply wrap those solutions in Pega, you get the case, you get the process workflow, you get predictability, you get agents running inside of a case, steps, stages, and so forth, and an inherent level of determinism out of the gate. Okay, think of it a little bit about hallucinations.

That's been a word that has been at the forefront of everyone's mind as long as large language models have been pervasive. But in these scenarios, if you're leveraging a model. And we would. To determine which tool to call. Okay, that can lead to cascading and propagating impacts in any autonomous agentic framework, we likely have dozens or maybe hundreds of systems. Okay. And the impact of one impacts the other. And that can lead to massive cascading impacts across the estate in a, in a sort of thinking about this in an orchestrated way, in a Pega way, you've got your tools allow listed inside of the, the, the case, you've got validated steps, the workflow can gate each step and scopes the tools access. The error stops right there if there is one. And the scope of the impact is dramatically reduced. Okay.

And then finally, and this is something that I hear all the time, and maybe I should have put this first because it's the thing that maybe is top of mind for the majority of our customers is that. um, if you've got an, a genetic solution, everyone comes and they want, they want observability, I want large language model, I want tracing, I want metrics, I want logs, I want to be able to put them into a system and review them. Okay. And I think the thing absolutely, you should continue to do that. But those calls and those agentic reasoning answers are not by themselves business decisions. They are representative of it. And maybe you could piece some logic together, but they are not inherently that OK in every Pega solution, you've got an audit of every step. You've got your token use, you've got your tool calls, you've got rule versions that show all the logic that was being implemented.

You've got a full replay for RCA purposes and compliance purposes. Okay. And so at the bottom, every enterprise needs a boat Platform in some way to orchestrate it. And I think some are deciding maybe an agentic rules that they're building these solutions as they go. And I think the easier path, um, is leveraging an orchestration platform to solve those types of platforms or problems, to get determinism, to get auditability, and to know that you can do it at significant scale. So that's the second pillar. We've talked about design. We've talked about runtime. And then the third one is there is trust that we build that we talked about. And there's trust now that we test. And that's the thing as we go forward and Ryan will come up and show you a really fantastic demo of how that comes together in Pega. But I want to break it down into three pillars.

Let's talk about reliability and stability. How do we regression test agent behavior before a model or prompt updates or ships? How do we catch the sort of silent degradation, the silent changes that may happen as they're in production? When you think about safety and governance and this is going to be a huge issue or a huge sort of surface area for us is you've got prompt injections, you've got jailbreak attempts, you've got patterns of misuse in any sort of agentic solution because they're conversational in nature And and then the second the second one is how do we keep agent behavior unbiased on policy and audit ready? Because we know with hallucinations and we know with the way models work, that you may not be able to get to that level of consistency easily. And so you've got to test and monitor for it. And then the third piece of it is performance and change.

These are foundational. Everyone wants to know how quick is it. Is it responding effectively? Are the customer satisfaction issues that may happen from slowness cropping up? What does that look like? And then maybe you want to address when to leverage a model that may be faster. We like to think about them as smart models or fast models. Sometimes the right solution to a job is maybe a fast model because it delivers a better experience and the results are where they need to be. Okay. So I want to break down in the next slide talking about testing strategies for an AI world. I think everyone is very used to what exists on the left hand side. We've lived this world for as long as software development has existed. And so the old way of testing and the old way in which we thought about things was we would hard code a set of assertions, we would pattern match, and we would have rigid logic.

And if the test passed, it passed. And if it didn't, it didn't. In today's paradigm, things are not nearly as deterministic as they used to be. And so we might want to leverage something like an LLM judge or an LLM eval. The only thing that's capable of sort of representing what a human might interpret from spoken speech, and look at it that way in the old way of testing, going down the list, we would test in isolation, we would stub out components, we would mock up services, and we would test. In the new world, we have to think about a large, complex, interconnected system, maybe of 100, 100 agents in my estate. I've got third party agents, I've got Other SaaS agents over there. You've got this big web. And so live orchestration, testing and regression testing of against the full suite is the world that we live in today.

And then the third one, we got better at this over time. We used to say, okay, once, once it was done, we would throw it over the fence and that was shipped. We got better about that as SaaS came along. We weren't done. When the product was done, we continued to monitor and deliver, but there was still roots in that today. Okay, but we need to think about continuous monitoring in a slightly different way. We need to think about drifts in prompt changes that may happen upstream or downstream of our systems. We need to think about hallucinations, things like role break and degradation. All these new sort of concepts and metrics to look at become critically important for us. And then let's talk about edge security for a second. We used to create a huge perimeter. That was the sort of the Great Wall. Nothing can penetrate it.

Now. We need continuous guardrails for adversarial testing, prompt injections, off topic queries, toxicity bias, and so forth to look at it from a continuous monitoring perspective. This is what the world I think looks like today for us in a predictable AI world. Okay. We're going to establish our design time. We're going to establish our runtime gates, and then we're going to sort of test in this world. We're going to break it down into these two buckets. We've got probabilistic tests on the left hand side and deterministic rules based tests on the right hand side. Rigid, rigid logic still exist. We're just going to leverage leverage it in the right way. And these are the metrics that we're going to be thinking about later when Ryan goes through it. Okay, so I'm going to hit them pretty quickly. We're going to talk about things like role adherence.

Is an agent adhering to the persona that we think it should adhere to. Okay. In the same way that we would train a human agent to speak in the brand that we want, we're going to do it for our agents as well, and we're going to validate it. Things like knowledge retention as an agentic solution takes multiple turns. Does it have recall to the first step, or the third step or the seventh step as you go through a journey. Something like conversational completeness. Did you complete the the intent of the client's inquiry? Was it answered in the way that you thought it should be answered? Did it factually make something up in the context of a hallucination? And then on the right hand side, thinking about deterministic logic. Are we calling the right tools with the right payload? Are we adhering to the latency and performance metrics that we set out?

If we if we want a response in 500 milliseconds, are we getting to it? Are we deviating too far? A business case life cycle another critical metric. If we had a journey that needed to go from A to Z, did we take all the proper steps to get along that life cycle of the proper journey? And then step agent accuracy. Did we call the right tool? Did we do the right thing? Did we get the right response back? And then a little opinion from from us on where we think these fit from a frequency perspective. The question you might be asking yourself is, okay, great, we'll run these tests, but how often or how frequent? What does that really look like? And here's a sort of blueprint for it. On the left hand side. And I won't go through the metrics. We think about something like latency and performance as how we think about synthetic monitoring.

We're just going to continuously check. Maybe we'll run it every ten minutes. Every 15 minutes. We're going to look at the system in a live basis. Okay. Are we delivering the core business logic? Is it function in the way we want? We're going to run that on a continuous basis against our live agentic solutions. And then because agents, as they're called in workflow are so critical to the orchestration process, are we getting the right triggers at the right points in a process? And then on the right hand side, maybe daily, weekly, we're going to do this in our pipelines. We're going to think about are we adhering to the right role? Is the persona of the agent the way that we expect it to be? Is it sounding the way we want it to sound? Are we factually incorrect if we have what good looks like in a representative example?

Are we getting that continuously in these daily or weekly tests? And then things like safety metrics, toxicity bias and so forth. Let's run those in a, you know, a lesser cadence than an ongoing way and do that in a daily or weekly kind of fashion. And since I know that everyone is getting ready or excited for the demo, I just wanted to lay the foundation for what you're going to see. So we're going to have a Pega Agents solution, and we are going to take what we're going to call a golden record. This is just a representative session of what a good journey looks like for this customer. We're going to have that sort of as a recording, and then we're going to replay that journey against the Agentic solution. And in the evaluation process, every turn that is taken, we're going to capture all of the metrics that we talked about a little earlier.

Okay. And then in the design phase, the report is going to produce. Whether you're adhering to it, there's going to have sort of a scoring system. Did you score above the threshold that we set? And you could do this even in an automated way. Maybe I'm just going to automatically accept the solution. I'm just going to release it, or I'm going to have a human go through the process. But that's where we're going to see. And I'm excited to bring Ryan on stage to show you guys what it looks like. All right. Thank you, Brian. Okay, so let's actually see what Brian was talking about in terms of a demo. And so with this demo, we're going to be able to see a Pega Agents. And this Pega agent has a number of different tools. So we can create cases such as a complaint, resetting a password.

It has access to a Pega knowledge agent. It's got the ability to create a home loan. So it's been in a retail banking scenario and it's got a number of other different tools. And what we're going to see is we're going to use an open source framework called deep eval to actually observe and test this Pega agent. And so through this, we're going to see a custom built UI that allows us to easily talk to that Pega agent and talk to deep eval and to be able to observe this agent. So before we get into the demo, I want you to consider this, this input here. So the customer is upset. So so we put this into the agent that the customer is very upset. They tried to log into their account to check their home loan statement, but their password wasn't working. Now they're locked out and they want to file a complaint about the online system being unreliable.

So just take a second and think about what you believe the agent should do in this scenario, because there's a couple of a couple of things based on the tools that that I just mentioned that the agent could or should do. So just take a second and think about what you think the agent should do. Okay. Now we've all thought about it. Remember this because this is going to sort of form the basis of, of what we're going to pass to our Pega agent. So Brian did a great job of summing up what we're going to, what we're going to see and all the components of it. But let's talk a bit about how the data actually flows between all of these components. And so we've got three main components that we're going to see. We're going to see the Pega application where the Pega agent lives.

We're going to see that deep eval application. So that front end. And then we've got that model judge. So what deep eval does. It leverages a separate large language model. And that is what acts as the judge to see between all those different terms. Turns. Are we hitting the thresholds or are we obeying all of those different metrics? Right. So for all the things that are that are non-deterministic, we're going to leverage that external model judge to tell us what, what, whether or not the agent responded in the right way. And so when we create that golden data set or that, that golden record, that, that, that correct conversation that we want to, to use to test against, um, this application is going to actually go into Pega request a conversation and then bring that back and create the golden data set from that. So we can create these conversations in Pega.

And then this application can go retrieve them and then create the golden data set from that. And then when we're ready to evaluate the agent, we again go turn by turn from the golden data set. We pass the input to the Pega agent using an API. The out of the box API. We send that. The agent sends that response back to our application. We then take that response, as well as all the other information in that golden data set and the information about the tests or the metrics that we want the model to evaluate. We send those off to that external LLM that scores are that scores all of the different metrics for that turn, and then returns those to the application and then continues on until we finally get a final output of of pass fail for each of the different metrics. And so the specific example that we're going to see is that we're going to create that golden data set.

So we'll see creating how that's created. We're going to evaluate that agent. We're going to get the score. And then we're actually going to see a fail. So we're going to see an instance where the agent does not pass a specific test. We're going to go in, we're going to modify the agent, we're going to test it again. We're going to score it again. And then we're going to see how that fails. So we're going to sort of see an end to end scenario of how we're going to be able to test this agent and and remediate it so that we get a good result to show an example of that observability. And so now I'm in my Pega application and, and what we're looking at here. I promise I know how to work a computer. And so what we're looking at here is I'm in my Pega application and I've just, I'm in a dashboard that shows some of the metrics around the agent.

The important part around this is the bottom part. So we have the list of all of the AI conversations that, that we've, we've had, and we're going to go to a particular conversation. And so this is the conversation that I want to use as my golden master data set. And what we're looking at here is what's called the agent tracer within Pega. And so this is out of the box. And it shows the log of everything that the Pega agent did. So it shows all of the different tool calls. It shows all the inputs and outputs to those tool calls. It shows the user input, it shows the agent response, it's got all the latency, the latency times, and the token counts. And so, so with this, this is a complete agent interaction. And this is what I want to use as my, as my golden data set.

Or is that that perfect conversation of how the agent should be responding to all the different, all the different inputs that I want to test. So what we're looking at here is the deep eval tool. And so this is this custom built front end that allows me to, um, sort of look at my Pega agent identify first the golden data set and then evaluate the agent. And so the first thing we're going to do is, is create the golden data set. And so I point it to a Pega instance. And that's what that project configuration is. We'll see that in a bit. And then it will load the conversations. And so what we may have missed is that there was actually a conversation ID associated with that agent tracer or with that, that agent case or that agent log. And so I can pull the, that specific log and then create the data set from it automatically.

So this tool knows how to go, given that ID knows how to go and get that log and then turn that into a golden data set. And so that's what we're going to see here. So to do so we'll name it so that we can use it for evaluation easier later. Click on that caption Golden data set button. And then we can see here that the golden data set was captured, that we've got four turns and then now the next step is to actually take this and evaluate it. And so when it comes to the evaluation, we again select that, that Pega agent that we want to evaluate. We select the golden data set and we can here preview the golden data set so we can see all the different turns, all the latency, we can see all the inputs, the outputs, and these case types that we're expecting the agent to create.

So all the information that came from that agent tracer log has been captured here. And then finally we select the metrics that we want to evaluate against. So Brian did a great job of explaining a lot of these different metrics. Um, you know, so we can define the metrics and the thresholds here that we want to evaluate against and the score that it has to, to that, that it uses as a threshold, whether or not to determine whether or not that metric passes or fails. And so in this example, we're going to use for metrics we're going to evaluate against hallucination, against bias and toxicity. And then this business case adherence. So essentially did the agent create the right workflow for the right input. So once we once we determine our metrics, we hit that evaluation button that goes against, again all the different turns that are in that golden data set.

And then and then measures and sends that off to the LLM to measure those different turns against what's in the golden data set. And we can see here that three out of the four tests passed, the one test that didn't pass was the business case adherence. And so when we start to scroll down here, we've actually got a log of all of the entire test. And so we'll be able to see that we can start to take this entire test. It's got all the different turns. Everything that the agent did, it's got the the outputs of all the different tests, which is what we're seeing here. So for bias and hallucination in the toxicity, those all passed. And then we see the failures and what failed and specifically what failed. So if you if we go back to that prompt that we thought about a few minutes ago, um, what the agent tried to do was so it, it was expecting a reset password case it created, but it didn't actually create the case.

So we can actually go back into the agent tracer to see what happened. Um, and to see this particular output. So back in Pega, I go back and I can see from the agent tracer what happened. There's the input and we can see that it did try to do something, but essentially it got caught in analysis paralysis. It did recognize, hey, I need to reset a password and I need to file a complaint, but it left it up to the human to decide rather than being proactive and actually starting a case. And so to fix this, we're going to go into the agent in Pega the agent configuration. And we're going to add two lines to our guardrails. So any time that we want to that any time that a request can be solved using a case, then start a case. And then also if a user's request contains multiple issues, always start that case for the highest priority issue.

So in this case someone couldn't log in. That's probably a higher priority issue than filing a complaint. So so let's do that. Let's let's get them logged in and then worry about the complaint. Okay. So we'll go we'll we'll save our agent configuration. And now we'll go back and we'll, we'll reevaluate the the agent just for that business case adherence since that's the one that failed. Okay. So we'll run our evaluation And what we'll see is that now our business case adherence passed. And again, the way that we're going to take a look at this is go into that Agent Trainer and we'll be able to see that we actually created that that password reset case when we when the user when we got to that input. Okay. So, so what that showed was we were able to, to test and observe that agent using that LLM judge for those non-deterministic tests or those non, those non-deterministic things had the external LLM evaluated using the deep eval framework.

Then going back and, and fixing, fixing and remediating for that test. And so the way this was, was all set up was again, through through this application here. So in the top right, there's a configuration here where we can, where we can define a number of different things for how deep Val works. So we can define the LLM judge here. So we can define the model where that model is coming from. And so we can define what what we want to use to judge. And certainly because LMS are good at different things, we may want a different LM for for different judges. And then we can define a configuration for for Pega. So what Pega instance we want to connect to, we can even define the agent type that we want to evaluate against, whether that's a conversational application level agent or what's called a step agent.

So within workflows, we can have agents automate workflows to do things like research or summarizing. And generally when we do those step agents, uh, that needs the context of a case so that it has the appropriate data to be able to evaluate against. Um, so when we define it, we can define that step agent and then provide a case that we can use to seed that agent with data before we evaluate it. And then we define the agent identity. So this helps for that. This defines that role. So when we start to get to that role adherence we can describe what the agent does. The domain. Um any guidance that we want to provide the agent. We can define sort of the workflow steps. Again this is taking a bit of a, also a view of, of looking outside of Pega as well. And then also look at context for hallucinations.

So what kind of things, um, or factual statements that the agent should adhere to to make sure that it's not hallucinating and going outside of those to make up facts. This will also has a rest API. And the purpose of the rest API is, is that now you have the ability to start to run these tests to define these golden data sets, as well as to pull these tests, these different tests Um, programmatically. And so you can imagine taking this tool and now this can become part of a CI CD pipeline, right? So, you know, in your pipeline, one of the gates is, you know, call, call the DP val framework. Do my agents or evaluate my agents, if any of them come back as a fail, then we're not. Then that gate fails and we're not going to move on.

Um, or even, you know, if you wanted someone to get access to all the different tests and see, you know, now, now that as you start to evaluate the agent, we're keeping those logs and you can start to build up that, that library of, of previous tests so that you can see over time how your agents changed and have the record of all those tests available to, to see how your agent's been performing over time. Um, and it is secured with OAuth, so we can give it, give it a client client client registration and then it will, will provide that for you for for OAuth. Okay, so you can actually download that tool that I just showed. So I literally just got approval for it 1030 minutes ago. Um, but it is available live now. Um, so yeah, feel free to, to play with it.

Um, fork it, have fun with it. It also does include another tool to actually sort of a look at Pega Agents and sort of test them as a, as a more of a test harness. Um, but yeah, have fun, have fun with it. Also there is an AI expert circle post on what Brian and I talked about. So certainly feel free to, to engage with that post, um, and ask any questions that you have. And we're happy to, to answer those.

Recurso relacionado

Producto

El diseño de aplicaciones, revolucionado

Optimice el diseño del flujo de trabajo, rápidamente, con el poder de Pega Blueprint®. Configure su visión y vea cómo se genera su flujo de trabajo en el acto.

¿Listo para crear un Blueprint?
Elija el motor de reinvención que se adapte a sus necesidades.
Para flujos de trabajo y diseño de aplicaciones

Reimagine sus procesos y convierta cualquier flujo de trabajo en una aplicación lista para su implementación con total confianza.

Pega Blueprint®
Para el diseño de estrategias de marketing y de experiencia del cliente

Visualice los recorridos del cliente y las estrategias de engagement a través de todos los puntos de contacto y actívelos.

Pega Customer Engagement Blueprint™