PegaWorld iNspire 2024: Operational Agility: Accelerating DevOps Adoption at FBI with Accenture Federal

Um, let me apologize. I woke up this morning with the sniffles, so I'm trying not to sniff, sniff, sniff into this mic. So I'm going to apologize up front because that's probably what's going to happen. But I am the section chief for the Business strategy and analytics section of the Human Resources branch of the FBI. That's really buzzwords for, um, you know, I know enough to be dangerous. That's really what that's a buzzwords for. But lucky for this group, I bought some really smart people along with me to kind of walk us through what this DevOps is all about. Let me get this clicker and I always do this wrong. So, um, I'm not going to walk through this full agenda, but just know that we're going to do a quick overview of DevOps.

Then we're going to talk about how we adopted it in our program, and then we're going to talk about how that has been impactful to our program, because I think that's the most important part of this. I'd like to read our mission statement, and I got to read it word for word off of this screen. One, because it's a lot of words, but two, it actually really does describe our section perfectly. So what we do in business is help hrb managers review their processes and re-engineer them to be more data driven and efficient. The section focuses on leveraging modern technology like Pega, centralizing important hrb data, and providing high end analysis and consultation to FBI leaders who are making strategic decisions. And I will say in earnest that the Bureau really has moved to looking at data to actually drive its decisions. What's not on the screen, though, is our vision. And our vision is actually to make things simpler, faster, smarter. That is what we focus on every day in our job.

Simpler meaning if there's any manual processes, reducing those manual processes in the steps in the process as much as possible. Faster automating whatever we can, and only when it makes sense because sometimes it doesn't make sense to automate. Smarter again, using that data to actually drive decisions. But before I hand this over to Corey, I just want to revisit history just a little bit. So I came to the Bureau in 2009, and like most of you, I'd probably seen a bunch of shows about the FBI, some movies. And so my expectations for technology was way up here. Way up here. I expected virtual files, virtual file cabinets. But that's not what I saw.

I actually stepped into the bureau and realized that our HR system was a mainframe, custom built system from 1975 , and this is 2009. I hope nobody in here from the FBI who actually helped build that. Um, I want to keep my good government job, but that's actually what I walked into. Um. We're talking if you didn't know function keys, you were not pulling any data. You were not circumventing that thing. But the people in the bureau actually loved it. They really loved this system. And when we talked about decommissioning it, the devastation across the full enterprise, that we were actually going to do that.

And I often hear about it even to this day. But my job in 2010 became modernizing the OHI, the FBI's ERP enterprise resource planning tool. And we did that. We wrote out the first module in 2013 and we used the waterfall methodology. I know everybody in here is probably fully, um, you know, engrossed in that and what that actually means. And it was the way to do it back then. And it kind of went like, thank you, customer , for your requirements. And I'll see you in about nine months with the thing that I'm going to deploy for you, except for nine months later, it might not have been what they wanted, might not have even been what they needed, because in that nine months, things really did change, and I might throw it over to my ONS team and say, hey, it's your job to fix it because me new dev, I got to go now and build the new next thing. And then fast forward Covid.

Covid changed so much for us where we were working, how we were working, the tools we were using. Um, you know, the bureau having to be remote, that was a new concept for us. Luckily for us, we had already gone to Cloud the year before, so that made that transition a little easier. But we decided to actually switch from waterfall to agile at that very time frame. We did a pilot with one team, and over that time, over that time we were all out for Covid. We actually moved like 14 teams to being actually scrum teams, and that kind of takes us full circle to where we are actually today. And so I'm going to turn it over to Corey to actually walk through what DevOps means, but then we'll jump into like what it meant for us. Thank you sir. Yep.

Thank you. So I'm Corey Waters. I have worked. I'm an LSA. I work with Accenture Federal. I've been working with the government and Pega for 12 years now, and over that time, I've had the privilege to see a lot of delivery transformations across organizations. A lot of waterfall technique, moving to agile , big batches, slow processes, moving to fast iterative changes, and the hidden thing that you don't really talk about a lot with this is it puts a lot of pressure on your technical folks. A lot of overhead with those frequent deliveries, pushing code, merging code constantly. So that's where DevOps comes in.

We're looking to help our technical folks and kind of close the end of the gap on that agile process. So DevOps is the union of people, process and technology to enable continuous value to our customers. To put it simply, agile builds the right thing and DevOps builds it right. Um, so these are in very intentional in order. Uh, a lot of people you jump straight to, I'm just going to throw technology at my problem. And quite frankly, if you don't have people and processes in order, it's never going to work. Um, so let's dig into people. The big part of people is just how do we define value? And this is actually where DevOps comes from.

So as a developer I typically define value as what shiny feature can I get to production as fast as I can. Um, on the other hand, my ops folks, they're going to say, well, my I provide value by having a stable environment that's available for everybody. Reliable. Um, and if you think about it, every line of code we push, it's an opportunity for a problem. So those values kind of go against each other. So your ops folks are going to be trying to slow down your devs, and your devs are just going to be getting pretty frustrated. Um, so what we're going to do is we're going to kind of reframe the whole problem. We're going to define value as actually talking to our customers, figuring out what they need and then getting it into production, having them use it and get feedback. That is how we provide value as an organization, not as individual teams.

And once we redefine that, everybody's working towards a common goal and we can actually start moving faster. Next is the processes. So you have a small team. You can usually keep up with this kind of stuff. But as you scale and as you start having these government systems that are affect tons of people, it gets complicated and you need to have standardization. You need to have automation so that you can keep up with all these changes. And everybody knows what's going on without having to be experts of the whole system. And then finally, technology that's honestly it's the easy one. You're at PegaWorld, you already have great technology and it's an open platform.

If your enterprise needs other tools, you can integrate them with Pega. So like I said, that's going to be the easy part. And it's funny because you jump straight to it usually. So now that we know what DevOps is, let's talk about kind of the goals of DevOps, what you can expect to see in your organization as you adopt these principles and you start living them. So first off culture, typically when we get places we see a fixed mindset. We've done this the same way for 30 years, and we're going to keep doing it that way because it works. Um, risk aversion again, you're doing it the same way. You're avoiding risk by doing it the same way. Um, and then the way you think about your product, you think, this is what I think the customer needs.

So that's what we're going to build and we're going to push it to them. And we need to move away from that. We need to have a growth mindset. The world's changing. We can't be on mainframes anymore. We need to we need to move forward. We need to tolerate some risk. And that can be a little scary. I don't mean let's just code in production.

Um, but let's do experiments. Let's have feature toggles where we can get limited groups of users using stuff and get feedback from them. Um, and that's how we're going to grow and that's how we're going to get better. And then finally design thinking. Let's not think that we know our customers and what they want. Let's actually go talk to them. Let's find out what they need and what will help them and what will make them want to use the system. Next is leadership. Uh, typically we see top down decision making in silos.

Um, you're wasting a lot of potential if you do it that way. Um, I do think that it's beneficial to have maybe a point person that can if you can't come to a consensus, make a decision. But really, you should be talking to everybody. Um, everyone brings something different to the table. They have different experiments or experiences with the customer, and they have a lot of value they can provide if you listen to them. And the next not or no common practices or processes across the system. Again, as you grow, that's going to be a problem. You need people to know kind of what's going on across the organization, across the program. And then finally, the focus on predictability, budget and efficiency.

Again, this is good. We want to be predictable, but we want to innovate as we're predictable. We don't want that to hinder our innovation. Um, next is talent and org. We have this extensive matrices of who reports to who. And everybody's really siloed off and specialized. We don't want to do that. We want to move towards cross-functional teams. Everybody working together.

We want our operations folks, our security folks, our business folks, our developers all talking to each other and figuring out really early on what we want to do, how we're going to do it, and the risks and the problems that we might encounter. We want to have T-shaped skills. Throw the buzzword in there. That doesn't mean you have to be an expert of everything. It just means you should be an expert of what you do. But you should also understand what your other teams are doing. I should understand as a developer how my testers are going to test what they're going to look at. I should understand how my operations folks are going to deploy my code, how it's going to how it's going to scale. I need to talk to everybody.

How do we make it secure? And we want to decide all that early on. Um, once you get to the end of the process, you're not going to be able to change it or it's going to cost a lot to do it. Um, and then finally, the process again, coming from waterfall phase gate, we want to get faster. We want to get that feedback from from our customers and be able to adapt our product and make it something that they love. Um, and then we also want to have continuous improvement. That feedback cycle needs to be constant. We need to be talking to our customers. What did they like?

What do they not like? What's bothering them with the system? It it's crazy. Every time we do a deployment now, we like to go talk to our users. What? What don't you like about this? What's what's breaking? You would be surprised. The stuff that doesn't bubble up to you.

That would be such a simple fix. Um, and then finally, automation. We want to automate these processes. Your developers, your your operations folks are very smart. You don't want to waste them just doing repetitive tasks that they don't need to think about. These are knowledge workers. You need to use them to do creative work, to do important work. Um, so now we're going to kind of shift into what have we actually done, how have we adopted this? And to set the context for everything that we're going to talk about moving forward, I want to start with the AFS Pega DevOps toolkit.

Um, so I think a unique position we have as consultants is we see problems across a lot of different organizations, a lot of agencies, um, and a lot of these problems are very similar. Um, so what we can do and what we decided to do was our center of excellence took the time to build recommended processes, recommended standards, ways of doing things, um, automation scripts, custom Pega integrations, custom Pega web services, all to enable us to kind of pull everything together and automate our delivery pipeline. Um, so yeah, we've got automated testing, built in change management, built in deployments, releases all controlled from Pega, and it's all just tied together in one clean package. Uh, next traceability. So I know across every industry, traceability is important, but the government it's extra important. We need to know at all times what everybody's working on, what they're merging when they're merging it, what tests they're running. What's the results of those tests? And we need to be able to go back and see the history of that, not only for audit reasons, but also to learn from it. What's not working in our delivery pipeline.

And then finally, customization. We can build something, but we can't build something that works for everybody. We realize that every program is going to have its own special needs. I think we can get probably 80% of the way there, but we opened up this toolkit for you to extend. We want you to make it your own. There's extension points all over the place. These aren't final rules that you need us to update. These are just things that you can go in and change based on the tools you're using, the process requirements, you have, things like that. And then the last thing I'm going to talk about here is orchestration.

So this is kind of like central to your entire pipeline. This is what's going to drive everything Pega offers deployment manager within the platform. It's an excellent tool. I think custom tasks were added a couple of years ago and that really bridged the gap. You can do anything with it now. Um, with that being said, we did choose to go with Jenkins, but that was more of an organization decision. We had the skill set already present. We already had pipelines being used. Um, and Pega, actually, again, it's an open platform.

It's awesome. That's one of my favorite things about it. They offer prpc service utils. You can hook in to a lot of dev ops out of the box services and tie them into whatever product you want to use. Um, and again, we, we offer common tasks. These were already within our toolkit. And also we had an expectation of added complexity over the future. So there's different tools that we're going to need to tie in different processes, different groups of people. And we felt that this was the best way to to make it so that we could extend in the future.

So with all that being said, I'm going to turn it over to Nick and he's going to talk about kind of the nitty gritty of what we really did here and the impact that we had. All right. All righty. As Corey said, my name is Nick Crockta. I'm a lead system architect for Accenture Federal Services. I've been working with Corey for about ten years now, and I started working with Tanya back in March of 2020, which was a bit of an interesting time back then. So as Tanya said, the the FBI at Air Force, we were deeply integrated, deeply ingrained in the waterfall methodology. But we were fortunate enough to start our agile transformation with our first Pega implementation at FBI Air Source. And I start with agile because one of the core concepts of agile is the concept of deploying early and often.

Right. And most folks in this room, they're probably familiar with CI, CD right, continuous integration, continuous delivery. But before you get there you need to have a solid foundation with release management. And what does that mean? What is release management? To put it simple, all it really is, is setting up a new release, right? Setting up a new baseline that you eventually want to push to production. Right. And it doesn't sound that complicated and it really shouldn't be.

But more often than not, you do find yourselves running into a couple of different challenges, and one of which is depending on the platform that you're using. Yes, we're PegaWorld we all use Pega here, right? You can have a multi step, cumbersome, maybe complex process to set up a new release in that software. Right. Fortunately, it's not too bad with Pega, but let's say you're starting up a new scrum team. You're trying to bring on new team members. You have three four senior developers, three four senior system administrators that come from a variety of different backgrounds. They're going to have different standards. They're going to have different setups for their releases.

Right. We want to be able to standardize that process so that we don't risk having human error when we're trying to create a new baseline or create a new release. And this may or may not have happened, but sometimes you have strict deadlines, right? Sometimes you're trying to get a lot of stuff done right before you go on vacation. You want to set up a new application. You want to set up all these new rule sets. You know, the password that you need to set everything to, but you fat finger it, you leave the next thing, you just block your entire project for a couple of days, maybe a few hours. And this definitely didn't happen with us. So how do we fix this?

How do we get around that? So incomes Jenkins incomes are DevOps solution. So one of the big things that we were able to do is we were able to take Pega out of the box lock and roll functionality and expose that as a web service. And before I get into the nitty gritty of that lock and roll service, it's kind of dependent on how you manage your releases, right? For us, I think we have a pretty good setup. We're deep in agile, as I said, but if we're in our 17th increment, our first Sprint, maybe our second release, we version control our applications so that it's 17 0102 seems simple enough, but it is actually super, super helpful and makes it a lot easier to hook in tools like this Lock and roll feature. So let's say we find out there's a production bug, or we just found out we need to push this high profile release right pretty quickly. All we need to do is log into Jenkins, log into our DevOps tool, enter a couple parameters and it's going to lock your rule set. So it's going to lock your application and automatically iterate over to the next version.

Right. So that seems great. And and all. What does that what does that really do? Um, so I used to be the guy that was responsible for manually doing those tasks. It took about 30 minutes for our application. And we don't have the biggest application in the world. Right. But if you're working on a large scale program and you have to do this 30 minute task per application that you have, that adds up, right?

And if you're doing agile as well and you're having, I don't know, 20 releases a year, 30 minutes times 20. I don't want to do the math right now, but that adds up, right? Uh, and then additionally, with these DevOps tools, they inherently come with an audit trail. Right? So audit history, maybe you've had a super busy week. You can't remember what version you just rolled all your apps to. You can easily go back to your deployment manager. You can easily go back to Jenkins and see who did what and when. Right?

What version am I on? Why did Gerry go back here and set it to this version? That's that's wrong. Let's let's fix this. Right. Adds a lot of transparency and then the benefits. We already talked about it a little bit, but time savings, time savings is massive. It simplifies it. And then you don't have to worry about the human error aspect of things to you don't need to remember what all these passwords are.

These variables are already going to be set in your deployment manager in your deployment tool. All right. Next up the fun and shiny part. CI CD continuous integration and continuous delivery. This is the whole goal of this is to try and get work quickly from dev to production quickly and efficiently. Right. And you're not able to do that unless you have a proper solid CI CD solution, right? And this isn't always the easiest. You have ways to do this manually.

You have ways to get your code from dev to test, from test to production, whatever environment that you want to send it to. But you'll also you'll often run into some people limitations or maybe some security limitations. For us, when we started, we had one scrum team. I think we had three developers, maybe two Bas a tester, but only one person on that team knew how to deploy code from dev to test, right? Maybe the other folks were a little bit too junior, but it was only one person's responsibility. And besides that, we also have what we call the separation of duties. Right? We don't want to have too many people having this power. Otherwise who knows what's going to happen, right?

So we need to make sure we follow that process. And then when it comes to deploying code from one environment to the next, I think a lot of us, or at least the developers in this room, can probably do it in their sleep at this point, but it is a multi step process. You have to find your product, you have to export your code base. You have to save it to a directory, maybe get whatever repository that you're using. You have to remember to use the right naming convention so that when you try and import in the next environment, you don't accidentally choose a stale, weak old version and then spend the next week trying to figure out how you can rip it out, right? Like all little things that add up all challenges that we can run into. And so how do we fix this? So for us and I know everybody can do this a little bit differently. You can have a different development process.

But what's worked successfully at FBI Air Force and a couple of different programs that we work with with AFS is we use branch development. And I know I wasn't a big fan of this when I first started. I think this guy had to drag me kicking and screaming, trying to get us to do it, But essentially when we get a new ticket, we get a new user story, we get a new bug that comes in, the developer is going to pick that up. The first thing they're going to do is log into our development environment, and they're going to create this branch. And this branch is think of it as just like a oh, maybe that's just me. Um, they have a branch that's isolated to just their code. Right. You don't need to worry about this in process work. You don't need to accidentally check in something that's going to break 15 other people in the project.

It's isolated. Right? So super, super helpful. And it gets into the concept of working in small batches, right. You don't need to have everybody working in this main code base. You don't need to worry about 15 people checking in, potentially the same stuff at the same time. Branch development helps a bunch with that. But how does that how does that come in with DevOps? How does that come in with our toolkit?

One of the neat things that Pega does with branch development, it has it actually provides a couple of hooks for you to add some custom customizations, some configurations. So we fully took advantage of that. Corey already mentioned the Prpc service utils, but what that does is it exposes some of those common APIs that maybe we didn't know that existed. Right. And two of those APIs is exporting your code. And then you can probably guess it, but it's importing it into the next environment. Right? So what we did is we combined it. So now a developer, they finish their their user story.

They say hey can you can you review my code for me that's secondary. Developer now has a couple of different options. They can say yes, this thing passes. I want to merge this to the main code base. But now with our toolkit, they actually have a couple different fields on their screen where they can determine, do I want to automatically deploy this to the next environment? Do I want to send it to multiple environments . Where do I want to send it. So we're combining separate steps all into one. And we're going to touch on it again in the benefits.

But how does this how does this help. Um, let's say you have a scrum team that's super, super strong and they're delivering three, 4 or 5 user stories a day, but you still have that singular person that was responsible for moving that code from one environment to the next. This this was me at some point, um, and back then would I want to spend 30 minutes, 15 minutes, several times a day trying to get this code from one environment to the next. Or am I going to set a timer on my desktop for 5:00 every day and just grab everything that's ready and move it? Right. That's simple. I only have to do it once. But is that great? Absolutely not.

Because when you do that, you're you're slowing down the train. You're grouping all this work together and you're basically asking your testers, your bas, your QA, whoever's testing this. Hey, can you please now just work on these five things at once, rather than working on everything individually. And again, that just slows down the train. So benefits already touched on it, but costs are time savings rather. And one thing that we don't often think about is by giving our team, by giving more folks the ability to move this code around, we're we're empowering our people, right? It's pretty neat if you're maybe one two years on the project, but now you have the ability to say, you know what? I did that I moved it, right. It goes a long way.

Um, and the small batch delivery, we talked about it again. You just want to be able to push smaller sets of code from one environment to the next. And it is extremely helpful when it comes to ad hoc releases and hotfixes. And then for my buzzword of the of the day, it's going to be mean time to recovery. Um, sometimes when you push code, you have these strict deadlines. You're going to accidentally let a bug slip through. Knock on wood, it's not going to be that bad, but sometimes it is. Sometimes you're going to be in this high stress situation. Tanya gives you a call and says, hey, can you please rip all of this out?

It's breaking. Please write and do it. Do it yesterday, please. Um, it's. You're able to do that. You can do this manually in Pega. You have a smaller application. It's really not that difficult. But again, once you start increasing the size of your applications, if you have a larger scale program, a one step process then becomes, I guess repeated over 15 times, right?

You have to log in, you have to update your version, you have to point it back to the right one, and you better not accidentally point it back to a more stale version, because then you're just again causing more problems for yourself, right? Prpc service utils Pega has an API exposed to allow you to pass in another couple parameters to just very easily and very quickly roll back to a previous version. So I know some of the other technologies in our stack. It's not it's not as easy. But for us just enter one field, you click go. Next thing you know everybody's happy again. And the team can spend the majority of their time finding that bug, trying to replicate it in the lower environment and pushing a secondary release, another release to get this thing corrected. All right. So containerization externalization.

We added this one a little bit last second, but we kind of just wanted to brag about ourselves a bit here. So this past year we were able to successfully containerize externalize our Pega implementation. It was took a bit of time to get there, but I think we're all pretty happy with the results. And this has been a big push of pega's definitely in the last year and probably longer than that. But trying to externalize your components of Pega, right? Externalize calf externalized search, externalize Hazelcast right container, container container or switch to Pega Cloud. But depending on your programs, one might be better than the other. But we took the initiative. We were able to make this thing work.

Our infrastructure team is very, very solid and we were able to containerize our environment. So I'm going to drift away a little bit from the challenges, solutions, benefits and kind of focus on the benefits and hopefully give a couple examples here. But some of the benefits of containerization is, first of all, it's portability. So containers encapsulate everything needed for an application to run, abstracting away from the host operating system. Um, they can be consistently deployed across any platform or operating system, making them more portable than the virtual machines that a lot of us in this room are probably more used to. Secondly, security and maybe I should have said this one first. But containers provide isolation and when you properly configure them, you can increase the security and reduce your vulnerabilities. After that, we have the ability to have consistent environments and I can't stress this one enough. All of you probably have different numbers of environments in your stack or in your implementation, right?

You have dev, you have test. Maybe you have QA, you have training, production mirror, another mirror of production, right? Using containers allows you to have one consistent deployment that you can share across all of your environments. Um, and one thing that's super helpful for that is sometimes you're going to run into some issues in production. I know in the past we'll have we've had different variable settings and lower environments, which makes it just a pain in the behind to try and replicate these issues in that lower environment. So far we haven't had that problem. Knock on wood. Um, and then lastly, simplified deployments. Containerization simplifies application development by bundling dependencies and configurations.

And yes, that's just words. But one of the things that it really does is it allows your infrastructure teams to focus on the things that are more important. They can focus on the orchestration techniques that you need to to enhance your CI, CD pipeline, rather than spending day in and day out trying to manage these complex infrastructures. And yes, that all sounds good and I feel like I'm a pitch man for containerization right now. But I can give I want to give a couple examples of how it's helped us recently. Um, so right now we're we're going through this, this massive implementation, this massive release. We're trying to decommission one of our legacy systems and move it all over to Pega probably done this before, but part of that process involves data migration efforts. And what does that mean? Essentially, you have to take all your information from that old application into your new one, translate it.

Because oftentimes those two applications, they don't they don't speak the same language. Right. Um, and those those tasks can be super, super tedious. And one of the biggest things with this data migration effort is your production data is going to look so, so much different than your development, your test data. Right. Your team is going to be in there in dev and test entering John Smith God knows how many times, right? Sally Sue. Right. Whereas in production that's not the case.

It's real data. Depending on the application you're working on, you're it's a critical application, right? You're working with people's PII maybe in your dev and test environment. Everybody knows when you enter Social Security number it should just be numbers right. Don't put a hyphen in there. Don't put a period. Don't put a space. Yes it is nine digits right. But in your old environments they fat fingered some things.

Maybe they didn't have those same validations that you had. Maybe they allowed the hyphens. Right. So you can kind of see where you would have some of these discrepancies. Right. So how does containerization come into play? We we learned our lesson with some of our previous deployments. We had a couple headaches when trying to migrate that data. But what we decided to do is like, why don't we just replicate our production environment ?

Why don't we just whip up a new one? So I remember I called Yasir way back when and said, hey, it's it's 8:00. Can you just can you do this? Can you create a new environment? It's like, yeah, just give me two hours and next thing you know, he whipped up a net new environment with his team that allowed us to test our utilities, test our scripts against real production data. We found some issues, but it's a lot better to find those issues in this mirror, in this copy environment than it is in production. Then you don't need to worry about your cleanup tasks. You got rid of it ahead of time, right? And most of the stuff, it's not that minor.

Like if you do it right the first time, it's going to work. But where you find yourself losing some time is the cleanup efforts, and we were able to prevent that. Then one other example, it follows along the same lines of replication We go out on what we call roadshows out to individual field offices and try and show them the net new things that we're building. And when we do that, we're training individuals on the new functionalities, the new features that we're releasing to them. Right. And what we want to do is we want to get them hands on keyboard. We want them to get familiar with this. Net new system that they're going to be using for eight hours a day, right. And we were able to do this in the past.

We we took aside some of our team members, took aside some of our BA's, our functionals, our developers, everybody, and staged a bunch of data. But what data do you think they're staging? Right? Is it really going to be production life? Are the people on your scrum teams, on your teams, the ones that are actually using the tool as an end user? Chances are the answer is no, right? So you're not going to have that real life looking production data. So what do we do? We replicated our production or one of our Q environments again and just took the data, potentially scrambled it right, and moved it over to this net new training environment in the matter of a couple hours.

Right. And now we were able to share this with our end users. They're able to test to do their UAT user acceptance testing. We're able to train them against some of the data that they're going to be looking at tomorrow. It's pretty neat. All right. And then where do we want to go from here? What's next? We only fit five things on this slide.

But we could probably put 20 things on here if we really wanted to. And I'll try and go this go through this a little bit quickly. But first off Pega has unit tests. Most people in this room may or may not have used it, but unit tests allow you to kind of put scenario testing around specific individual rules. Maybe you have a really complex activity, a really complex piece of functionality that if this thing breaks, you're going to have problems, right? You can take a Pega unit test, whip it up and add it to your deployment pipeline. So now we already talked about the merge process. Why not add one more thing to it. You merge it to your main code base.

We're automatically going to push it to test. But right before we do we're going to execute this one unit test, these 15 unit tests. And if it fails block the deployment. Right. It might sound bad at first, but would you rather block this deployment from dev to test or accidentally forget about it, let it go to production production and then deal with the consequences later. All right. You have that ability to configure it. You have that ability to customize it however you please. And taking it one step further on the testing front, you can integrate these development or sorry, these DevOps tools with different automated unit testing suites.

So for us is it going to be selenium? Is it going to be UiPath? Is it going to be Pega scenario testing? It doesn't really matter, but now you can plug in those automated testing scripts so that when you again deploy to the next environment, kick off the very, I guess, common tasks that are always being run by your testers manually. Why not automate that ? And if it fails, block it. If it fails, don't , it doesn't matter. But you can test it right as part of your deployment process. Number three, this is probably my personal favorite because I think we do a pretty decent job on our project of documenting all of our code, from checking comments to keeping track of all the rules that we're changing.

And this, this might not seem that that beneficial, but when you when a developer makes a change and maybe they leave, maybe they get rolled off, maybe they decide to go on to brighter pastures, right? Somebody has to replace that person. If there's a bug in that code, it's going to be pretty difficult to try and find out where it is. But if you're able to configure integrate your solution with JIRA and just take all that information that's entered, move it, clean it a little bit, maybe if you want to, but move it over to Jira, you know, have another repository to find the changes that you just made. It's a lot easier to learn from that than it is trying to, I guess, go into Dev studio as a functional person, try and figure out why something happened, right? Because we know Jira, your ticketing systems, whatever you use, it's all pretty similar, right? Number four confluence. Same thing. Just another Atlassian tool.

But if you have release documentation, you can take that out of Pega and automatically send it over to confluence. You're just creating your own documentation tool, moving all of it over there so more people can have access to it. And lastly, Pega has a static assembler. And what is what does this mean? But if you have a bigger program maybe you're releasing this. Net new piece of functionality that's never been touched before. You could be the very lucky or unlucky soul that has who's going to be hitting that new functionality for the first time, right? When that happens, things need to crash, and sometimes you'll find yourself logging in. You're going to see that old blue Rubik's Cube that you have probably seen in Pega seven or Pega eight.

If you're on it, the little blue circle and you're going to be spinning for 30s waiting for everything to cash. If you don't want to provide that that negative user, I guess impact or experience, you can actually hook in pega's Pega's static assembler to take care of that for you. So when you deploy to production, kick this thing off and it's going to automatically cache everything with you for you. So that way the first user who logs in at 5 a.m. or whatever time that they do, they don't need to worry about that 32nd spinny circle. You save yourself one complaint and everybody just has a better user experience. So with that, I'll pass it back to Tanya. Thank you sir. These guys make me look so good.

I love it, so I'm going to run through this fairly quickly, because we want to make sure at least there's a little bit of time if people have questions . But this part is really important. It's probably the part we're going to spend the least amount of time on. But me as a system owner and having responsibility for some of this, literally it is the impact that has made the difference. So based on all of the information that Corey and Nick have given us, like we've literally been able to increase the number of deployments to production like we are deploying all the time. It's a little scary sometimes, but we are literally pushing code, you know, every three weeks over over multiple sprints, over multiple pies. And that's definitely very different than how we operated in the past. Like I said, we'd have that one big release at the end of the year and we'd be like, yep, did it. Sorry, you might not like it, but it's yours now.

And we've definitely, definitely flipped that reduce time for the development to production. More this last this next one is like really important. And Nick kind of touched on this. But the morale that improved for the developers feeling very empowered, like they're pushing that stuff, they're pushing code, they're taking some part in actually the deployment of it. And so that's pretty huge reduction of the bottlenecks I'm usually getting on this team. What are we testing. Why is it taking so long. Because sometimes that's the questions that I'm getting from leadership. Like what's taking so long?

I mean for us I feel like three weeks I'm pushing code is not that long. But sometimes for leadership, the, you know, the impatience there to actually get that end to end functionality done. Um, but this is one of the things that I like to highlight, because we've done a remarkable job of reducing the bottlenecks and again, achieved the shift to the left mission goals. We always talk about shifting to the right, but the ability to be able to say you've actually shifted that that deployment to the left and you're actually deploying on time and sometimes ahead of expectations is like a major goal. So I'd like to thank you all. I'd like to thank you all for taking the time to listen to us, and actually allowing us to come and actually talk about the successes that we've actually had. So I'm going to go to questions. Are there any questions? Oh that good huh.

And I'm okay if there's no. Yeah. Uh, funny funny thing, Nick and I, Nick and I have to mad dash. We've actually got a little bit of time just because our flight got delayed. We just did the last release of a major deployment that is changing the way that we do security background checks. And so we are literally heading to Huntsville from here. And so we have a little more time to ask some questions. So definitely leave it up to you all. So I guess I'll break the ice.

Uh, what kind of deployment strategy are you guys using? Using, like canary blue. Green? Like, what'd you guys end up choosing? Yeah. Um, I guess maybe I don't fully understand the question, but is can you ask it a slightly different way? Sorry. So, like, when you deploy to prod. Yeah.

Um, blue. Green would be like having a inactive and an inactive version of your application. So deploying your new code to the inactive one. Smoke testing and then swapping. So yeah like it's seamless to the end user. But you make sure that like deployment actually worked. Gotcha. Or like um Canary would be like restricting the IP range. It's only like a few people get it.

Gotcha. Yeah. So it depends on the release that we have. Nine times out of ten, uh, we deploy everything straight to production. So we don't use the blue green setup. That's something that we certainly talked about. But what we have done and what has proved super successful for us is that we use Pega's feature toggles or our own slight like customization or configurations of that. So we just had this big release, like Tanya said. And what we have, what we've been doing is we've been pushing chunks of it from dev to test to production for the last year or so.

Right. But it wasn't until we hit a weekend, right. When everything's in there, we'll actually toggle everything on, do some smoke testing straight into our production environment until we verify that everything works great, and then we release it to our end users the following the following workweek or work day. Yeah. So okay. I think one I had a little bit to that. I think one of the reasons we do it that way, because in your lower environments you don't have those interfaces there, and so you really can't test some of that stuff until it literally is in production, because you can't push it to the other systems that are actually really live and tied to your to your prod. Okay. Yeah, that makes sense.

And then just the last thing I'll add to a beta test or access group is a pretty good way to do it too. You can release the application, leave everybody on the old version, and then just bump it up for a limited set of users. You can get into a little trouble if you have database changes, so you've got to be careful with that, but that's a good way to do it as well. Okay. That actually kind of leads into another question I had when you guys were talking about rollbacks. How many integration points do you have? And like when you guys roll back from one version to the other, like, is that somewhat painful? Like if you're integrating with something that's already like in flight in another system, how do you guys handle that? Um.

I guess. Yeah. So a lot of the time, the rollback again, it depends on the release. We can automate database changes as well. So we can have rollback scripts for the database. But a lot of the time it can be just as simple as bumping it back down. You're setting your access group back down to the old version. Um, a lot of the time our integrations also versioned, so you can have system settings to say what version of that integration you're going to use. And we can build all of that into Jenkins as well.

We can update those system settings automatically as a as a rollback script. Okay. Thank you. That's really all I had. We're on the same team. We'll see. We'll see if we have the same questions. Um, mines around your production data replication process. Um, one, is it because you have your business storage in a separate repository?

And two, how often do you guys refreshing that production mirror data? Yeah. So I kid you not, this happened within the last month, so we're still kind of standardizing the process, if you will. But, um, yeah, I mean, we it's very seldom that we've had to do it. And we're still trying to come up with the long term strategy to make that work because we try to once before didn't work out so well, but we're hoping that we can use some of the AWS features since we're on RDS to help simplify that process for us. Okay. So your data repository is a separate RDS that you guys have. Yes. I'm just wondering how many times do you find yourselves releasing now and then, also what framework or structures that you put in place to engage with customers throughout this cycle so that you're actually building the right things?

If you are releasing quickly that you're actually building in that. So we released. Every three weeks. We have a Pi that runs every three months somewhere between really three months and 12 weeks. We are definitely following the agile framework in terms of scrum of scrums, um, requirements gathering with our customers. We've actually added some of our own things in there too as well. So we do this prioritization across, um, the sections of the various sections that are responsible for some of the things that we are doing for them. And so we pull the requirements from there, they prioritize it. Then we do our Pi planning, like our full team goes down for about two two days to actually do that.

Pi planning on the front end. But throughout that full release cycle we're actually meeting with the customer constantly. We're actually truly following that true framework. We're going to follow up on that. So you release something in three weeks percent of the time that you say you got right, based on the customer feedback. Right? And then what percentage of of work do you. Have to redo based on the feedback you get? I say we literally shoot for about 80 to 85% .

We go with that mindset that we know we're not going to have it at 100% . So I think we get truly at the release around that time frame, around that amount of time, we're somewhere hovering around 80 to 85% . And that is intentional. Um, and I do think we actually get that right. That's not to say that the customer is not losing their entire mind, because sometimes we literally are changing their business process. And to to take on that change is hard. The places I say we fail, if we are going to fail, we fail in two areas testing or the change management of that process. And so we are constantly having communication between us and our customer. And and I can't say sometimes it's wonderful and they love it.

The other thing we're doing is system demos. We do the system demo also every three weeks. And so before we were listening to the code, they're actually seeing the demo of it. Um, I'll say the challenge there is it's not always in context, because they're seeing pieces of the thing that you intend to release, and sometimes they don't get to see that full, that full end to end solution until the very end. And sometimes that is a challenge. Um, things that we've also implemented, though, even after that is we try to get hands on keyboard not only through the testing, but at the training. Um, before we go live, that's usually training. There is also on site support for the customer and what we're doing now. What we're about to go to is actually a road show.

We do this road show, this concept of a road show we've done since the beginning of the program. And I will say it's one of, I happen to think one of the best things that we ever did, because we know that we're going to have a touchpoint with that customer at least once a year, at least. And so we're doing feedback. We're training on whatever the new thing is that's coming out. And sometimes we are getting ahead of it. We may even do a design thinking session in that during that road show. And the thing that is supposed to be a training. So it's quite often the touch point is quite often, and I will say truly in waterfall, we didn't do that. We did road show all the time, but the touch point was not as frequent as it is today.

Thank you. All right. Any other questions? Because I'm about to do the mad dash to the to the airport. I'll be sticking around if there are questions that come up afterwards since they have to jet. Yes, definitely. Appreciate you guys willing to definitely share anything that we have done in terms of, um, releases, the stuff that we built. um, we do it definitely with other government agencies and kind of share notes and things. Definitely willing to do it with anyone that's actually here.

I think these folks here know how to get in contact with us if, um, if you need to or want to. All right. Thank you all. Thank you.