Demo: Hyper-Automation of Oracle Database Change Requests

 

Oracle OEM, Ansible and ServiceNow

Let's start with a scenario. You don't have to imagine hard, because if you've ever been on an Oracle DBA on-call rotation, you've lived it.

It's 2:47 AM. Your phone lights up. SIMPROD_TS tablespace is 97% full.

Tablespace Full 100% OEM Incident Manager
Oracle OEM - Tablespace Out of Space Incidents

You groan, fumble for your laptop, VPN in with fingers that don't quite work yet, and spend the next 25 minutes doing the same five things you did last Tuesday, and the Tuesday before that: check usage, check disk space, check whether growing this datafile is going to starve some other database sharing the same mount, run the resize, verify it worked, and then — the part everyone secretly skips — write a ticket note detailed enough that Future You (or worse, a coworker) can understand what happened.

Now multiply that by an enterprise with a few thousand databases and tens of thousands of alerts a month.

Suddenly you don't have a "wake up one DBA occasionally" problem. You have a full-time alert-triage department problem, a DBA-burnout problem, and — my personal favorite — a "why does this ticket say 'fixed, see prior notes' with no prior notes" problem.

When "just add more on-call people" stops working

At small scale, tablespace alerts are an annoyance. At enterprise scale, they become something closer to a natural phenomenon — like weather, except the weather is angry emails from your VP asking why Oracle Enterprise Manager (OEM) has 4,000 open incidents.

A few things start happening that nobody designed on purpose:

- Alert fatigue sets in. When a DBA sees the 40th "tablespace full" page this week, muscle memory takes over. Muscle memory is great for typing passwords. It is not great for double-checking whether that "quick resize" is about to blow past the physical disk capacity shared by six other databases on the same host.

- Documentation quality becomes inversely proportional to how tired everyone is. 3 AM ticket notes are... let's call them "abbreviated." Six months later, during an audit, "fixed" is not a sufficient answer.

- The same fix gets reinvented, by hand, thousands of times a month, by different people, at different skill levels, with different levels of caffeine in their bloodstream.

- Your best engineers are stuck doing the least interesting work. The people capable of redesigning your database architecture are instead resizing datafiles at 3 AM. That is not a talent allocation strategy. That is a hostage situation. None of this is a knock on DBA teams — it's what happens when you point a fundamentally manual process at a mechanical, repeatable problem and then scale it by 1,000x. The process was never the bottleneck. The math was.

So we automated the math.

The fix: let the systems talk to each other

We built a closed loop connecting three things that, until now, mostly ignored each other:

- Oracle Enterprise Manager — already watching every database, 24/7, better than any human could

- ServiceNow — already the system of record every incident lives in anyway

- Ansible — already the tool our DBA team trusts to run real database changes, because it's been doing exactly that for nine other automation projects The trick wasn't inventing new technology. It was making the technology we already trust hand incidents to each other automatically, instead of routing everything through a human pager.

Four systems, one closed loop

Four Systems, One Closed Loop

Here's the shape of it: OEM detects the problem and fires a webhook. ServiceNow turns that into a real incident, with real fields, in real time. A scheduled Ansible job — running every two minutes, unattended, forever — picks up that incident, logs in, runs the actual diagnostic SQL, checks whether it's safe to fix (more on this in a second, because this is the part that actually matters), fixes it, and writes down exactly what it did. Then OEM's own next monitoring cycle — not the robot patting itself on the back — confirms the fix actually worked, and that's what closes the ticket.

Not only automation did the work of a DBA while he was asleep, it did a better job. Usually, a sleepy Oracle DBA simply resizes the tablespace or adds a datafile. If he is an experienced professional, he will check if the storage mount where the datafile resides has enough free space for the additional space to grow. Usually. But automation is unusual. It is never sleepy. It never does a job half way. It knows nothing about the concept of "good enough". Here is what it does. It actually calculates the sum of MAXSIZE's of all datafiles in the Oracle container, all of its pluggable databases, and even all other databases mentioned in /etc/oratab. If it determines the mount will become more than 75% full once all datafiles grow to their maximum - even ten years from now - it immediately opens a ServiceNow change request with the storage team to add an additional disk space. At 3AM!

One more aspect to consider. Here is a pie chart of such OEM "Tablespace Full" alerts from an actual client (a monthly analysis).

OEM Alerts Pie Chart

Oracle Tablespace Full Alerts Break Down by Type

When an administrator sees these 737 alerts, he has to act on all of them, one by one (or at least starting with the production alerts). But he lacks context. Out of the 737 alerts, 379 are related to the same issue (directly or not), but fired at different times - you resolve one you resolve them all, yet the DBA still has to sift through every single one of them! Moreover, another 233 are stale alerts because the alerting works one way, they will remain in the queue until someone manually checks whether they are still valid. It will take an inordinate amount of time for a large team to sort those out to get to the actual, actionable urgent alerts that matter. There are only 125 of those. Less than 17% of total work actually needed to be performed.

Now, lets take a look at a high level automation flowchart (OEM + Ansible + ServiceNow):

the 10-step lifecycle of OEM alert automation with ServiceNow

The 10-step Alsert/Incident Lifecycle

Everything this closed-loop automation does adheres to vendor's best practices, it is ITIL aligned, with cost-containment and risk management.

Ten steps. Zero humans. One very good night's sleep.

OEM Tablespace MAXSIZE and AUTOEXTEND ON

"But what if the robot breaks something?" — great question, we asked it too

This is the part I actually want you to read carefully, because it's the difference between "cool automation demo" and "thing I'd trust in production."

Before the automation touches anything, it doesn't just check "is there room to grow this one datafile." It reads /etc/oratab on the host, checks every single database living there, adds up their current footprint and their theoretical maximum footprint if every autoextensible datafile on that box decided to max out simultaneously — and only then decides whether it's actually safe to act.

This is, frankly, more thorough than most humans manage under pressure at 3 AM. It's the annoying friend who checks the stove is off twice before leaving the house — except it does this instantly, silently, and writes down exactly what it checked, every single time, whether or not anything goes wrong. If the check fails, the robot doesn't get creative. It stops, writes down exactly why, and leaves it for a human. No drama. No "I'll just try something."

Show, don't tell: this actually happened

Talk is cheap, screenshots are not. Here's a real incident, generated by a real (intentionally triggered) tablespace alert, worked entirely by the automation:

ServiceNow OEM Tablespace out of Space Progress Notes
ServiceNow OEM Tablespace out of Space 2

ServiceNow Incident Opened by Ansible On Behalf of Oracle OEM

Notice this isn't a summary like "fixed the tablespace." It's the literal SQL statement, the literal command output, and the literal math behind the safety check — automatically, every time, with zero chance of "I'll write it up properly later" (we all know later never comes).

And because plenty of DBAs live in OEM and never touch ServiceNow, the same note also shows up as a native comment inside OEM's own incident — cross-referenced back to the ServiceNow ticket:

ServiceNow OEM Tablespace out of Space Incident Manager

The Same Evidence, Mirrored Directly into OEM, Incident Auto-Closed

Nobody had to remember to update two systems. The systems just... did it. Which is, generally speaking, the entire point of automation.

Okay, but does it actually save time, or is this just a fun toy?

Fair question. Here's the honest version, not the marketing version:

OEM Oracle Tablespace KPI Improvement

Legacy Manual Process vs. Automated Aler/Incident Management and Resolution Pipeline

The manual-side numbers are realistic estimates of a typical on-call incident, not a scientifically audited historical average — we say that plainly, because a business case built on made-up precision isn't a business case, it's fan fiction. But the automated side has one very real, very measured number baked in: 33 seconds. That's the actual wall-clock time the automation took to diagnose, safety-check, and fix a real tablespace incident, measured from an actual log file, not a slide.

Everything else on the automated side — the "waiting for OEM's next check-in" time — isn't the robot being slow. It's the robot being patient, because it insists on independent confirmation before declaring victory. That's a feature, not a bug.

KPI Difference

Estimated KPI Improvement Ranges

Great, but what about the other 19,999 alert types?

This is the actual pitch, so let's not bury it. Everything above is built to be generic. The webhook contract, the "claim it so nobody double-processes it" logic, the fleet-wide safety check, the dual-system audit trail — none of that is tablespace-specific. The only thing that's specific to tablespaces is the SQL that diagnoses and fixes tablespaces

Which means the next alert type isn't a new project. It's a new SQL script riding on infrastructure that already works.

phased expansion roadmap

Phased Expansion Roadmap

Archive log space, ASM disk group capacity, backup job failures — these are next in line, because they're mechanical, well-understood, and low-risk, exactly like the one we just proved out. Trickier stuff (killing blocking sessions, restarting listeners) comes later, once the first wave has its own track record.

The point

Nobody's job is "resize tablespaces at 3 AM." That's not a role, it's a symptom — a symptom of a manual process being asked to do a machine's job at machine scale. Automating it doesn't replace your DBA team's judgment; it gives that judgment back to them, for the problems that actually need it.

Screen Capture of OEM + Ansible + ServiceNow Integrated Automation Solution Handling 5 Artificially Induced "Tablespace Out of Space" Alerts in Real Time

If you're running Oracle Enterprise Manager, ServiceNow, and Ansible and you've ever thought "there has to be a better way than this" — there is, and it's more straightforward to build than you'd expect. Happy to talk shop in the comments.