Comprehensive software reviews to make better IT decisions
Agile Tips the Balance Toward Success for Most Projects (but It Takes Time and Effort to Become Agile)
Image courtesy of The Standish Group International, Inc., 2015
In 1994 the Standish Group published its first CHAOS Report, which famously reported that only about 16% of software projects were completed on time and on budget. Over the years, this staggeringly low success rate has thankfully more than doubled (albeit with a modified definition of success). It’s important to know that Agile projects have carried the lion’s share of this improvement, which is why many organizations are making the move to Agile (but be warned that an Agile transformation takes time and effort).
The 2015 edition of the report examined success rates based on whether projects used an Agile versus Waterfall methodology, and the results showed an almost four-fold improvement in probability of success (39% compared to 11%) when the project used Agile instead of Waterfall. And this multifold improvement continues to be supported by other sources. So it’s no surprise that Info-Tech member organizations are increasingly adopting Agile, which has gone from being a rarity to becoming a necessity.
But becoming Agile is not as simple as just training your teams in Scrum, SAFe, DAD, or any number of other established methodologies. This is true because becoming Agile (first and foremost) is a change management undertaking, and organizational change is hard. It’s therefore not surprising to see many first-time adopters of Agile struggle to achieve its promised benefits. This is because the first tendency of any organization is often to “do Agile” rather than “be Agile,” meaning they focus on doggedly following prescribed process steps without actually focusing on the intended purpose of those steps. A common example is sprint retrospectives that become blame sessions because organizational culture has not shifted toward collective ownership of both success and failure by the entire delivery team (not surprisingly, CollabNet’s 13 State of Agile Report says that “Once again, the survey responses indicate that organization cultural issues remain the leading impediments to adopting and scaling agile”).
Successful adoption of Agile is about organizational change, but many mistakenly see it as simply a process change. Following a process, like Scrum, XP, Kanban, or some hybrid, is an important first step in becoming Agile, but it doesn’t end there. Your organization will also need to change the way it thinks and operates to achieve the full potential of Agile. Some of the more difficult changes that need to take place include:
- Abandoning command-and-control in favor of self-directed-teams: Agile relies on the collective skills and motivation of the delivery team to make good choices for the organization no matter what twists and turns they encounter along the way. A culture that relies on top-down control and decision making will interfere with your ability to succeed with Agile. Your organization needs to properly empower delivery teams to make critical decisions quickly, which means that not only do you need to support and trust your teams, but you need to help them understand what good decisions look like.
- Becoming more willing to fail: Every organization suffers from “The Determinism Fallacy,” which is the mistaken belief that, by thinking long and hard enough about any problem you can determine the correct set of steps required to solve it. Unfortunately, software development often requires a trial and error approach to be successful, which can be extremely difficult for organizations to accept. Agile rejects the determinism fallacy and instead asks the organization to try, fail, and adjust in short bursts so that the impact of failure is minimized. Think of each sprint as a short experiment to see if something works. If it does, great! If it doesn’t, use the learning from that sprint as a course correction that will reduce the risk of future failures.
- Blaming the problem, not the person: Agile embraces the fact that things can and will go wrong. Fundamentally, each failure is an opportunity to learn and adjust behavior to reduce future failures. But this can only happen in an environment where failure does not lead to blame. Delivery teams must feel safe enough to admit what isn’t working and collaboratively take steps to eliminate the problem for the good of the organization. A culture that blames the person, instead of the problem, will fail at Agile.
- Adopting collective ownership of both success and failure: Traditional command and control organizations often make people responsible for “one small piece of the puzzle” (for example, making Business Analysts solely responsible for gathering and documenting requirements). Agile instead, requires each member of the delivery team to be collectively responsible for every aspect of the deliverable. They all either succeed or fail together, and this means doing work which might not be part of your traditional job description. This shift requires the organization to empower the delivery team, and equally, it requires the delivery team to step up to the challenge of collective ownership. An organization which hangs on to an “it’s not my job” culture will fail at Agile.
- Seeing change as good: Many organizations have been habitually trained to believe that mid-project changes (e.g. new or different requirements) are bad and need to be avoided. Agile embraces change as not only normal but good, especially when it is driven by learnings from earlier sprints. A delivery team needs to adapt its way of working to accommodate this need for change (e.g. don’t over invest in a design that may need to change once people start to see what the final product will look like). A delivery team that is resistant to changes will fail at Agile.
- Seeing baby steps as great progress: Agile advocates frequent demonstration of progress to stakeholders (e.g. sprint demos every few weeks), both to show progress and, more importantly, to gather feedback that is used for continual course corrections toward the final product. At the beginning of the project, the amount of progress that can be demonstrated will be naturally quite small and often will look very different from what the end product will eventually become. This can be a challenge to many organizations that are used to only seeing demos when a team has something substantial (and finely polished) to show. An organization who sees small incremental steps as disappointing, rather than comforting proof that progress is being made, will fail at Agile
These fundamental changes in organizational thinking and behavior will be difficult for your organization to accomplish, and so it will take patience, perseverance, and strategic thinking to help Agile take root in your organization over time. Here are some tips to help your organization reduce the risk of failing at Agile:
- Avoid big bang: Moving to Agile is difficult (and risky) because of the amount of change required by the organization. Don’t make the mistake of doing a “big bang” introduction of Agile to your organization. Start small, expect to make mistakes, learn from those mistakes, and adjust your approach as you go. We often call it an Agile journey because it is just that: an organizational journey that starts slow and improves incrementally over time. And just like any journey, what it looks like will be different for every organization.
- Pick the right pilot: Piloting Agile on a project is a great way to avoid a big-bang approach. Use the experience and learning from a pilot to increase your chances of successfully adopting Agile across the organization. However, it is important to pick your pilot carefully to avoid failures that can hinder broader adoption. Any Agile pilot will be risky simply because of the amount of organizational change required, so pick a low risk project. For example:
- Don’t pick an in-flight project and switch to Agile in the middle. This is a guaranteed recipe for failure.
- Pick a small project that will be relatively easy to accomplish in a short period of time (the delivery team needs to focus on getting good at Agile practices without being overwhelmed by a long and difficult project to boot).
- Choose a delivery team that is open and eager to try Agile. Ideally, you should seed this delivery team with people who have used Agile before (perhaps in a previous job) and are willing and able to share their learnings on the pilot with other teams.
- Pick your Product Owner and Scrum Master carefully. It is said that the most valuable contribution Scrum has made to Agile is creating the Product Owner and Scrum Master roles. Poor choices for either role will be detrimental to your pilot’s success. Both should be filled with willing, capable, and respected individuals who can dedicate the time needed to the pilot. Your Product Owner needs to be empowered to make decisions quickly and have easy access to subject matter experts who can validate decisions when needed. Your Scrum Master should have some technical expertise, but more importantly must be a good facilitator, mentor, coach, and effective leader.
- Train your teams: Be sure to provide your delivery team with high-quality training on the Agile methodology they will be using. Don’t assume that “reading up a bit” on the process will be enough.
- Understand that training is not enough: Training is an important first step, but the truth is that your delivery team will only get good at Agile by practicing it. And in the early days, they will need support, guidance, and patience to allow them to make mistakes and learn. Expect things to be a little rough at first as the delivery team works to put theory into practice.
- Leave yourself a buffer: Don’t assume your delivery team will be working at peak efficiency from the get-go. Leave some buffer in your Agile pilot schedule to account for learning and mistakes your team will almost certainly make. Your first Agile project will NOT be your most efficiently run project.
- Use an Agile coach: Bringing in an experienced and capable Agile Coach is a great way to increase your chances of success. Many of the problems, questions, and concerns that new adopters get stuck on can be easily overcome with the help of a good Agile coach. Pick someone who comes highly recommended by references and be sure to have one or more members of your organization work closely with the coach to learn their practices so that you maintain this skill set within the team once they leave.
- Automate as much as possible: Adopting Agile is far easier and more likely to succeed when you
incorporate use of automated tools to carry some of the load. For example:
- Use one of the many available software development management tools that support Agile processes (like Jira, VersionOne or Helix) to make your delivery team’s job easier and make it possible to automatically gather highly valuable metrics (like sprint velocity) without additional effort by the team.
- Use software test automation tools (like Parasoft) to make regression testing less labor intensive.
- Use team collaboration tools (like MS Teams or Slack) to improve team communication, particularly for remote team members.
- Get organizational buy in for the changes needed to support Agile: It is critical that your organization’s leadership both understand and accept the changes required to successfully adopt Agile. Many leaders are unaware of the true challenges associated with the shift to Agile and the role they must play to help change culture, thinking, and behavior. Similarly, if they are cannot accept that Agile is something that requires significant time and effort to adopt, your organization is less likely to be successful.
- Use Info-Tech’s expertise and offerings: Agile adoption is a complex journey with plenty of obstacles that can get in your way. Info-Tech offers blueprints, expertise via analyst calls, and workshops that can help make this journey easier and more likely to succeed. Consider using Implement Agile Practices That Work or the soon-to-be published Agile Transformation Accelerator to help you with your journey.
Take advantage of Info-Tech software reviews to see how individual tools rate based on unbiased customer feedback.
Thor, the Norse God of Thunder, tells Jane Foster, the woman he’s trying to impress, that on his home world of Asgard, the realm eternal, science and magic are two sides of the same coin. Had Jane been a part of the operations teams at Google (or other mature online service providers), she would have immediately realized we have a similar technology right here on good old Earth. We call the science site reliability engineering (SRE), and service level objectives (SLO) is the magic behind it. SRE is a powerful concept for organizations that are serious about keeping their customers happy. It is therefore important for them to develop well-thought-out SLOs and make certain that management is intellectually equipped to derive valuable business perspectives from them.
Hell hath no fury like a customer not being able to access an online service when they want to. They expect the online services to always be on, always be accessible, and always treat them like there’s no one else in the world who matters more. Thank heavens then for giving these online services the ability to use site reliability engineering (SRE) to keep their customers happy, engaged, and most importantly, feeling valued.
Info-Tech members moving to Agile are frequently unsure of the role of PMs and the PMO in an Agile environment. Any organization used to traditional (Waterfall) project management will need to make adjustments in support of Agile or risk losing the benefits.
GitHub has announced that, effective April 14, 2020, all of its core features will be free for everyone. This will include private development within organizations that have previously paid for some subscription plans.
Many Info-Tech members are wrestling with how to best manage their software development productivity while working from home, especially for teams using a Waterfall approach. Sprinkling some Agile practices into their normal routine could improve transparency and show continuous value delivery.
When deciding on how to license your products or components, you don’t start with debating open vs. closed source code. It starts with asking simple questions around your overall goals.
Not all metrics are good and using poor metrics can have a serious negative impact on your organization. Maximize the value of your software development lifecycle (SDLC) metrics by using this thoughtful and judicious approach to ranking and selecting them.
RPA projects fail more often than one would expect. The ease with which RPA tools allow workflows to get designed and implemented makes it easy to avoid building a strong foundation for the work being done.
We live in a metrics-fixated world where having more metrics is always thought to be better than having less, and Software Development Life Cycle (SDLC) metrics are no exception. But the truth is that any badly chosen or managed metric will do more harm than good to your organization. To avoid these pitfalls, take ownership for SDLC metrics away from managers and put it into the hands of those who can best manage it: your development teams.