Click  to watch this demo in full screen.

View our "1,000 Change Requests (Un)done in Under a Minute" challenge!

Welcome to the Replayable database refresh demo.

1. Introduction

The current approach to executing IT changes is technology or department-oriented. If a ticket is for upgrading an application server, the work is routed to the applications team. If for a database, then to the DBAs. Our approach to change management automation is process-oriented. We believe the currently accepted department-oriented approach is obsolete. It was only necessary in the on-prem world when the administrator's technical expertise could not be codified and re-executed in YAML or JSON language. The technical know-how was only in the heads of the hired administrators. Naturally, the ticket needed to be routed to the correct department to avoid delays or reassignments. Each department had enough administrators on payroll to handle its peak load once a week or even a month. For the remainder of the week, the admins were either at rest or coasting. In addition, their expertise was not interchangeable. What one team knew and did, the next department was unable to replicate. For example, the Azure SQL Server department was useless at refreshing an OCI Oracle database and vice versa. We believe such department-oriented open-loop automation, meaning an automation method that requires any human intervention at all, hurts the business more than it helps, because its goal is to accommodate the administrator, rather than extricating them from the process as the slowest and weakest link in the automation chain. We believe in closed-loop automation, which eliminates any human intervention except for its design. Before exploring closed-loop automation, let's first understand how open-loop systems work currently.

2. The Current Way of Administering an I.T. Change

A request is opened. An admin does the work and closes the ticket. Tomorrow, he will repeat the task, then again and again, with or without the aid of scripts or decision support software. Either way, this is an open loop repeated by a person in perpetuity. Closed-loop automation is different. It frees administrators from having to execute the same task repeatedly. Instead, the admins design the automation logic once, and the automation takes over the process. Unlike open-loop automation, which cannot function without a human, and, by extension, allows a possibility of error or entry deviation, in the case of closed-loop automation, all steps and variables are predefined and locked down. Since they are locked down, there is no need to approve each iteration of the same process, meaning a specific run with its own assets, variables, credentials, and so on. Instead, we approve the automation workflow and how its particular variables are allowed to iterate or diverge within different technologies or assets. This would have been impossible in the case of open-loop automation because of the need for human intervention. Now that we have explored the difference between open-loop or incomplete and closed-loop hyper automation, it is time to apply both to a commonly used IT process.
Let's say an application owner opens a ticket to refresh all lower environments' Oracle databases from the production instance. This is a routine configuration change that IT organizations perform several times a year. The goal is always to have a fresh subset of data to code and test with. The following day, the change request is routed for approval, as all coding work will stop during the refreshes. It takes another day or two for the management to approve the change. On day three, the database refresh ticket in our example has at last been approved and placed in the appropriate queue. Since it is Oracle databases that need to be refreshed, the request is placed in the Oracle DBA department's queue. The department manager assigns the refresh to a particular administrator experienced in the tool or vendor specified in the request. The due date on this refresh request is a week away. But why wait a whole week to run a quick script? As the manager was once an excellent administrator himself, he understands that his team is already overloaded and may become even busier, so he should give them some additional buffer time to address more pressing production issues that may arise. Let's continue with our example of the status quo. The Oracle Database Administrator finds the time within the week his manager allocated, makes the 10-minute configuration change, and closes the work item as implemented. It then goes to validation by the development team, which takes another day or two. Now, for brevity purposes, let's calculate the total for all iterations of the refresh. Remember, three refreshes are running consecutively: Development, CERT, and UAT. The 10-day execution time was only for the development environment. There are two more to go. Here is the final scorecard for the omni-environment database refresh process in its current form.

The first measurement is duration. According to the scorecard, it took approximately 30 days to refresh all three lower environments consecutively. After all, administrators have other work to attend to. Besides, they work only certain hours a day and certain days a week.

Next, the cost of the business is the sum of hours worked by all personnel involved in the activity, from approvers to administrators, multiplied by their estimated contractual hourly rate. The direct cost of an Omni Environment Refresh may run to a large enterprise as high as $180,000 per year.

The error rate, or quality score. People make mistakes. That's a fact. A lot of them, in fact. The average human error rate is 4%. That means some projects will have to be relaunched, postponed, or even cancelled. Quality score is where indirect costs come into play. We can't attach a price tag to a substandard product or a long-delayed feature release. Yet it is quality hits like those that may lead customers to leave for the competition in droves, forcing you out of business. Quality matters.

The last metric is the scalability factor. Scalability refers to the ability of an IT process to expand or contract in response to demand quickly. The load a current team carries can be temporarily doubled for up to a week. To complete more work, you must hire additional DBAs. Otherwise, your personnel will eventually get overworked and will burn out.

As you can see, the current departmental method of delivery of configuration changes is severely lacking in terms of cost, speed, quality, and scalability.

3. Replayable Demo of Omni-Environment Oracle Database Refreshes Module - Behind the Scenes

Let's compare the same refresh with its closed-loop automated version. Keep in mind that the database refresh module is just one out of hundreds already developed, and thousands more modules are in the works, not just refreshes and not necessarily Oracle. Our software runs on a small VM or a physical Linux box with very modest minimum requirements. It connects to your ticketing system via an API on one hand and to the asset in need of maintenance on the other. The first thing you will notice about closed-loop automation is how quiet it is. There are no meetings to be held, no approvals to seek. Everything has already been discussed, approved, and locked down the minute you assigned this particular refresh not to one of your administrators, but to a Replayable module. Unless instructed otherwise, the automation software gets to work immediately. The ticketing system, such as ServiceNow, provides a high-level process description: what needs to be done, where, and when. The cloud tags provide the current metadata about the object of maintenance, vendor, version, cost center, and custom tags like last refreshed or stack version. And our software provides the hyper-automation logic in the form of workflows that all cloud vendors understand. The logic is pre-coded ahead of time, with error detection, rollback, delegation, etc. It is an answer to the question, what is the best possible way to execute the change? As we explained earlier, our closed-loop automation sees the task at hand from a process standpoint. That is because there are no administrators and no departments in this closed-loop implementation. Our change management process starts with the ticketing system, providing a high-level process description. For example, suppose the change request states omni-vendor and omni-environment. ServiceNow transmits that metadata to our software via API. Since the work item is omni-environment, all lower environments will be refreshed from PROD: UAT, Certification, and Development. Also, since the work item is omni-vendor, that means all the cloud vendor software will be refreshed simultaneously. AWS, Google Cloud, Oracle, etc. This demo is for refreshing an Oracle RDBMS in an EC2 instance in AWS, but it could also be used for changing the environment, updating the tags for all MySQL instances in Azure, or for another purpose. Next, our hyper automation software contacts the asset in the cloud to obtain its most recent tags. For example, the Replayable module gets all the EC2 instances' hostnames and Oracle database names and other metadata, and uses its internal templates to construct runnable scripts at the time approved and scheduled by ServiceNow. The fact that the metadata is obtained from a single source of truth (the cloud tags), and that is done immediately before the runtime ensures a complete absence of errors and the quickest execution time. It is at this time that the Replayable module transmits all the runnable scripts to the cloud assets. Each EC2 instance gets its own, highly customized, absolutely error-free version of the refresh script, including all the relevant validation scripts and SQL plan execution test cases. The script variables are different, but they all still follow the general process automation workflow. That is done at the same time for just 1 or 1,000 instances in parallel. As a matter of fact, all you have to do is change one line in the automation inventory file from LNX002 to LNX[001:999]. That is a closed-loop, automated alternative to hiring 1000 of the world's best administrators ever lived on a moment's notice and having the capacity to disengage just as quickly. That is infinite almost scalability. Let's get the final scorecard comparison. Open versus closed-loop IT change management: hyper-automation. Let's see who wins.