Reduce Manual Repetitive Work With IT Automation

Free up time for value-adding jobs.

ANALYST PERSPECTIVE

Automation cuts both ways.

Automation can be very, very good, or very, very bad.
Do it right, and you can make your life a whole lot easier.
Do it wrong, and you can suffer some serious pain.
All too often, automation is deployed willy-nilly, without regard to the overall systems or business processes in which it lives.
IT professionals should follow a disciplined and consistent approach to automation to ensure that they maximize its value for their organization.

Derek Shank,
Research Analyst, Infrastructure & Operations
Info-Tech Research Group

Executive summary

Situation

• IT staff are overwhelmed with manual repetitive work.
• You have little time for projects.
• You cannot move as fast as the business wants.

Complication

• Automation is simple to say, but hard to implement.
• Vendors claim automation will solve all your problems.
• You have no process for managing automation.

Resolution

• Begin by automating a few tasks with the highest value to score quick wins.
• Define a process for rolling out automation, leveraging SDLC best practices.
• Determine metrics and continually track the success of the automation program.

Info-Tech Insight

1. Optimize before you automate.The current way isn’t necessarily the best way.
2. Foster an engineering mindset.Your team members may not be process engineers, but they should learn to think like one.
3. Build a process to iterate.Effective automation can't be a one-and-done. Define a lightweight process to manage your program.

Infrastructure & operations teams are overloaded with work

• DevOps and digital transformation initiatives demand increased speed.
• I&O is still tasked with security and compliance and audit.
• I&O is often overloaded and unable to keep up with demand.

Manual repetitive work (MRW) sucks up time

• Manual repetitive work is a fact of life in I&O.
• DevOps circles refer to this type of work simply as “toil.”
• Toil is like treading water: it must be done, but it consumes precious energy and effort just to stay in the same place.
• Some amount of toil is inevitable, but it's important to measure and cap toil, so it does not end up overwhelming your team's whole capacity for engineering work.

Info-Tech Insight

Follow our methodology to focus IT automation on reducing toil.

Manual hand-offs create costly delays

• Every time there is a hand-off, we lose efficiency and productivity.
• In addition to the cost of performing manual work itself, we must also consider the impact of lost productivity caused by the delay of waiting for that work to be performed.

Every queue is a tire fire

Queues create waste and are extremely damaging. Like a tire fire, once you get started, they’re almost impossible to stamp out!

Increase queues if you want

• “More overhead”
• “Lower quality”
• “More variability”
• “Less motivation”
• “Longer cycle time”
• “Increased risk”

(Source: Edwards, citing Donald G. Reinersten: The Principles of Product Development Flow: Second Generation Lean Product Development )

Increasing complexity makes I&O’s job harder

Every additional layer of complexity multiplies points of failure. Beyond a certain level of complexity, troubleshooting can become a nightmare.

Today, Operations is responsible for the outcomes of a full stack of a very complex, software-defined, API-enabled system running on infrastructure they may or may not own.
– Edwards

Growing technical debt means an ever-rising workload

• Enterprises naturally accumulate technical debt.
• All technology requires care and feeding.
• I&O cannot control how much technology it’s expected to support.
• I&O faces a larger and larger workload as technical debt accumulates.

The systems built under each new technology paradigm never fully replace the systems built under the old paradigms. It’s not uncommon for an enterprise to have an accumulation of systems built over 10-15 years and have no budget, risk appetite, or even a viable path to replace them all. With each shift, who bares [SIC] the brunt of the responsibility for making sure the old and the new hang together? Operations, of course. With each new advance, Operations juggles more complexity and more layers of legacy technologies than ever before.
– Edwards

Most IT shops can’t have a dedicated engineering team

• In most organizations, the team that builds things is best equipped to support them.
• Often the knowledge to design systems and the knowledge to run those systems naturally co-exists in the same personnel resources.
• When your I&O team is trying to do engineering work, they can end up frequently interrupted to perform operational tasks.

Personnel resources in most IT organizations overlap heavily between “build” and “run.”

IT operations must become an engineering practice

• Usually you can’t double your staff or double their hours.
• IT professionals must become engineers.
• We do this by automating manual repetitive work and reducing toil.

Build your Sys Admin an Iron Man suit

Some CIOs see a Sys Admin and want to replace them with a Roomba. I see a Sys Admin and want to build them an Iron Man suit.
– Deepak Giridharagopal, CTO, Puppet

Use automation to reduce risk

Consistency

When we automate, we can make sure we do something the same way every time and produce a consistent result.

Auditing and Compliance

We can design an automated execution that will ship logs that provide the context of the action for a detailed audit trail.

Change

• Enterprise environments are continually changing.
• When context changes, so does the procedure.
• You can update your docs all you want, but you can't make people read them before executing a procedure.
• When you update the procedure itself, you can make sure it’s executed properly.

Follow Info-Tech’s approach: Start small and snowball

• It’s difficult for I&O to get the staffing resources it needs for engineering work.
• Rather than trying to get buy-in for resources using a “top down” approach, Info-Tech recommends that I&O score some quick wins to build momentum.
• Show success while giving your team the opportunity to build their engineering chops.

Because the C-suite relies on upwards communication — often filtered and sanitized by the time it reaches them — executives don’t see the bottlenecks and broken processes that are stalling progress.
– Andi Mann

Info-Tech’s methodology employs a targeted approach

• You aren’t going to automate IT operations end-to-end overnight.
• In fact, such a large undertaking might be more effort than it’s worth.
• Info-Tech’s methodology employs a targeted approach to identify which candidates will score some quick wins.
• We’ll demonstrate success, gain momentum, and then iterate for continual improvement.

Invest in automation to reap long-term rewards

• All too often people think of automation like a vacuum cleaner you can buy once and then forget.
• The reality is you need to perform care and feeding for automation like for any other process or program.
• To reap the greatest rewards you must continually invest in automation – and invest wisely.

To get the full ROI on your automation, you need to treat it like an employee. When you hire an employee, you invest in that person. You spend time and resources training and nurturing new employees so they can reach their full potential. The investment in a new employee is no different than your investment in automation.– Edwards

Measure the success of your automation program

1 15 FTEs x 80k/yr.; 20% of time on MRW, reduced by 20%
2 15 FTEs x 80k/yr.; 30% project capacity, increased by 30%
3 25k/hr. of downtime.; 5 hours per year of downtime caused by errors

Automating failover for disaster recovery

CASE STUDY

Industry Financial Services
Source Interview

Challenge

An IT infrastructure manager had established DR failover procedures, but these required a lot of manual work to execute. His team lacked the expertise to build automation for the failover.

Solution

The manager hired consultants to build scripts that would execute portions of the failover and pause at certain points to report on outcomes and ask the human operator whether to proceed with the next step.

Results

The infrastructure team reduced their achievable RTOs as follows:
Tier 1: 2.5h → 0.5h
Tier 2: 4h → 1.5h
Tier 3: 8h → 2.5h
And now, anyone on the team could execute the entire failover!

Info-Tech offers various levels of support to best suit your needs

DIY Toolkit

“Our team has already made this critical project a priority, and we have the time and capability, but some guidance along the way would be helpful.”

Guided Implementation

“Our team knows that we need to fix a process, but we need assistance to determine where to focus. Some check-ins along the way would help keep us on track.”

Workshop

“We need to hit the ground running and get this project kicked off immediately. Our team has the ability to take this over once we get a framework and strategy in place.”

Consulting

“Our team does not have the time or the knowledge to take this project on. We need assistance through the entirety of this project.”

Diagnostics and consistent frameworks used throughout all four options

Reduce Manual Repetitive Work With IT Automation – project overview