Comprehensive Software Reviews to make better IT decisions
Choose the Right QA Tools to Validate and Verify Product Value and Stability
The ultimate goal for product development is to deliver valuable, usable, and functional features, enhancements, and changes to end users in a way that aligns to business stakeholder expectations and objectives. This goal requires a deep understanding of the quality attributes that define product success from both business and technical perspectives and correctly enforcing these attributes in a streamlined quality assurance (QA) practice.
QA is an integral part of your software development lifecycle (SDLC), as it ensures quality standards and controls are upheld throughout development. Your QA tools should be able to accommodate the various quality requirements from the various stages of your SDLC to ensure the product meets stakeholder expectations. Injecting the proper validation steps early in your SDLC can help minimize the high costs of fixing defects found later in development. QA automation capabilities offer organizations the ability to expand test coverage and better utilize the time allocated to testing.
Automation can only be achieved by standardizing and enforcing a repeatable development and testing process which lays out the artifacts that need to be created, roles who need to be involved, quality attributes that must be clarified, and the activities that need to be completed to deliver a valuable product. However, the appropriate mechanisms need to be in place to continuously improve this process to better fit and govern your evolving business and technical environments. See our blueprints for additional information for defining, implementing, and maturing your testing practice:
Much like any other development tools, you will need to evaluate how your QA tools fit into your overall tooling environment and strategy.
- Application Lifecycle Management (ALM) – ALM tools help manage and monitor the development of products as they progress through the SDLC, and orchestrate the completion of manual and automated work. A key benefit of ALM tools is that they provide traceability of user stories and requirements to code to test cases to release packages to support ticket, assuming they are defined and templated. See our Choose the Right ALM Solutions to Manage Product Delivery note for more information.
- Continuous Delivery – QA tools are integral parts of the larger product delivery pipeline which can include a number of other automation tools, such as automated build, continuous integration, and automated deployment. Therefore, QA tools must be specifically configured to accommodate the outputs of upstream development activities and create compatible inputs for downstream development activities. This step requires a close look at the attributes of development artifacts and how they trigger QA and other automated development activities.
- Open Source vs Proprietary Tools – Open source is a popular choice of tools among today’s organizations. However, there are a number of things you should consider in your research and selection:
- Tooling Landscape – Most of the open source tools focus on a particular set of features for specific technology stacks. Clearly identify what you are looking for at a minimum from functional and non-functional requirements perspectives.
- Cost vs Value Consideration – Open source may not generate the cost savings you and your stakeholders expect. The implementation may be cheap but on-going support may drive down your ROI.
- Governance Model – Each open source community has its own approach to govern the changes to its products, whether that be an open developer forum or a benevolent dictator approving changes.
- Open Source Community – Review the livelihood and participation of the community to address implementation and operational challenges.
- Consulting Availability – If you are not receiving the support you need from the community, consider the available third-party resources that can assist you. Unfortunately, this specialized skill set may come at a high cost.
- Integration Capabilities – Be cautious if vendors have invested interest in certain open source tools, as this may lead to biased treatment towards their partners. Ultimately, you don’t want to be left on the hook for support if the vendor shifts gears.
QA tools, much like any other optimization or implementation, should not be done in isolation as local improvements can drive down the performance of the entire delivery pipeline.
Players in the QA Landscape
QA tools leverage graphical user interfaces and automation capabilities to provide developers, testers, and operations with the ability to easily create, manage, execute, and report tests and results. Today’s QA tools have several common attributes:
- Record and Script Tests – Ability to generate tests by recording the actions of a user.
- Graphical User Interface – User-friendly interface into software testing product.
- Reporting – A reporting suite containing canned reports with the ability to export in multiple formats.
- Requirements Traceability – Errors, queries, and defects logged and tracked throughout the testing lifecycle; test cases/scripts mapped to high-level requirements; and the ability to trace progress on resolving issues connected to requirements.
- Reporting and Analytics – Ability for users to find the required data using context-based filtering, sorting, and reporting; reports offer real-time data based on report criteria; and users can create dashboards to track progress of issue management-related tasks and project activity.
- Risk Analysis – Ability to perform impact analysis based on test results on the business and technical risks of an application going live.
- Test Artifact Management – Create, template, manage, and version test artifacts (e.g. test plans, test cases, test scripts, test dashboards, defect logs, test data, test environments).
- Integration With Third-Party Tools – Easily integrate with other systems and tools through out-of-the-box plugins or customizable APIs.
Each QA tool vendor and open source community will have their own approach to testing and alignment to industry standards and best practices. They will also differentiate themselves based on the type of applications and platforms they support, integration partners, degree of QA lifecycle support, and automation capabilities. Some vendors only manage testing activities and do not provide the native capability to test. Therefore, your choice of tool and vendor will depend on the reason why you need to test and how you intend on using the test results.
- Comprehensive Test Management Suites – These suites provide a centralized, unified, and comprehensive platform for managing manual and automated tests, test processes, assets, plans, activities, and results against defined quality goals and requirements. A wide range of tests (e.g. regression tests, integration tests, performance tests, function tests, security tests) can be natively executed through these suites for applications on a number of platforms. Notable vendors in this category include: Microsoft, Micro Focus, Perforce, Parasoft, IBM, CA Technologies, and TestRail.
- Tactical and System-Specific Test Tools – These tools are designed to perform a specific set of tests for applications on certain technology platforms. Some tools are aligned to particular industry testing frameworks (such as NUnit), support certain development methodologies (e.g. ATDD, BDD), or are architected in a specific way (e.g. cloud, SOA). These tools are often used to complement feature gaps in comprehensive test management suites or are plugged into ALM solutions to extend their testing capabilities. Notable vendors in this category include: T-Plan, Akamai (acquired SOASTA), Selenium, TechExcel, SmartBear, Appium, Sauce Labs, Perfecto, AppDynamics (now part of Cisco), and Ranorex. Additional tools can be found in XebiaLabs.
In addition to the core features defined earlier, vendors and open source communities will incorporate a number of advanced features to streamline test creation and execution and support continuous delivery and automation:
- Traceability – Testing artifacts map to business and non-functional requirements and links to code, test cases, defects, support tickets, and other development artifacts.
- API Testing – Testing for desktop framework APIs (e.g. Java, .NET), web-based APIs, and cloud APIs.
- Mobile Testing – Performance testing (network-related, response time, memory usage testing), security testing (client-side and server-side security), functionality testing (form factor, gestures, internationalization/localization, interruptions), compatibility testing, automated testing, record and reuse test scripts across different applications, and testing across multiple mobile platforms (web, native, hybrid).
- Cloud Testing – Record and reuse test scripts across all applications to conduct testing in multiple cloud test labs that are running concurrently, and perform functional, security, and performance testing of a cloud-based application.
- Test Lab, Data, and Environment Management – Creation, execution, and removal of virtual and cloud test labs and environments. Ability to simulate the application’s intended environment with virtual or real data or infrastructure/network images.
- Business Process Testing – Trace business processes to test cases, and run business components in business process testing.
- Unit Testing – Ability to test individual code modules, and send notifications when testing for code modules fail/succeed to respective stakeholders.
- Regression Testing – Ability to test the impact of code changes, and send notifications when code changes fail/succeed to respective stakeholders.
- Security Testing – Evaluate a system’s susceptibility to security threats, including applications scanning, penetration tests, static analysis, threat modeling, and vulnerability scanning.
- Low Code Test Creation – Automatic generation of test cases and scripts using natural language, business process flows, or wireframe diagrams.
- Real User Monitoring – Record and track all user interactions of a deployed application.
- Application Performance Monitoring (APM) – Tracks an application’s use and performance by monitoring the various system stack components supporting the application’s execution.
- Predictive Defect Analysis and Forecasting – Trigger alerts of potential issues based on observations made in the SDLC or provide static analysis that indicates potential issues without the execution of the code. These potential issues can be based on historic work or vendor provided insights.
- Static Code Analysis – Review the code structure, quality, readability, coding style, and code coverage of your applications.
See our Implement a Proactive and Consistent Vendor Selection Process blueprint for more information about procurement best practices (including RFP templates).
- Review and architect your current ALM tools to accommodate QA tools. See our Choose the Right ALM Solutions to Manage Product Delivery note for more information about ALM tools.
- Amend development guidelines, SDLC practices, and roles and responsibilities to accommodate QA practices. See our Modernize Your SDLC, Build a Strong Foundation for Quality, and Optimize Your SQA Practice Using a Full Lifecycle Approach blueprints for guidance in the review of your SDLC and QA practices.
QA tools empower all development roles to create and manage a comprehensive list of tests and execute them in a continuous, and often automated, process. These tools can expand test coverage which can motivate development teams to proactively improve product quality and value delivery. However, automation does not eliminate the importance of quality accountability and the need for continuous QA process improvement. Evaluate the rationale behind the need of QA tools and identify opportunities where automation can bolster the value of your products. In some cases, a single tactical tool can generate a higher ROI than a comprehensive testing suite.
Want to Know More?
Platform product owners struggle to develop a product roadmap and prioritize backlog items when their system supports multiple external products across business lines.
Establishing and monitoring SDLC metrics can drive behavior change. But metrics are prone to creating unexpected outcomes and must be used with care. Use metrics judiciously to avoid gaming or ambivalent behavior, productivity loss, and unintended consequences.
Aha has deepened its integration with Jira via new support for linking not only issues/features but also their respective dependencies, thereby improving overall requirement visibility.
Atlassian Jira has taken an important step throughout 2019: bringing Jira Software and Bitbucket capabilities to Visual Studio Code.
Intland Software’s codeBeamer ALM provides out-of-the-box complaint templates for organizations developing software in the automotive, medical, aviation and defense, and life sciences & pharma industries.
CollabNet TeamForge ensures development governance, compliance, and standards are maintained whether an organization is using Agile, Waterfall, or hybrid software methodologies.
The root causes of throughput issues go beyond the work assigned to delivery teams or the tactics to complete them. Cultural, leadership, and management factors play critical roles in team productivity and motivation.
Perforce Helix ALM is a suite of integrated products that centralizes and manages requirements, testing, and issue artifacts.
Aha! has improved its integration with Azure DevOps to improve release and sprint visibility for both developers and stakeholders.