Organizations have been measuring output in some form and using it to fine-tune their efforts for ages, yet every generation seems to struggle to identify key metrics. In the software realm, since the first bit of software code was executed successfully in 1948, companies have striven to boost their software’s value and quality, often with mixed results.
Between 2014 and 2016, in particular, renewed interest in reaching the “pinnacle” of software excellence only seemed to cause confusion, as project managers and their teams argued over what constituted software value. For example, functionality enabled as part of defect resolution was often perceived as a new feature and not a necessary improvement.
Ironically, a very effective methodology for defect resolution, the goal-question-metric (GQM) approach, has existed for more than 40 years. Pioneered as a means of evaluating project defects at NASA’s Goddard Space Flight Center, GQM was first applied to software more than 20 years ago. But despite its proven value, many software engineers still haven’t embraced this method. They should.
The value of GQM
Barriers vary from one team to the next and can even be unique to a particular situation or company. Nevertheless, from a global perspective, they are often rooted in one or both of two challenges: Your metrics are not aligned with software quality goals and therefore do not inform the firm’s effort to pursue targeted improvements, or your individual teams have their own sets of metrics, with no cohesive vision or strategy to standardize and apply them as a group.
Dismissing the value of GQM is a losing strategy if you're pursuing quality goals.
Identifying and applying relevant software metrics enables both software teams and decision makers to quantitatively evaluate the effectiveness of development processes and the quality of resulting outcomes. Metrics can form the foundation for meaningful improvement, support cost-benefit analyses, and enable better decision making. A GQM approach helps teams ensure their metrics reflect software goals—and more importantly—business priorities.
A path to accurate software measurement
As with any engineering discipline, the development of software requires a measurement mechanism for feedback and evaluation. Measurement should help answer myriad questions associated with the software process, from “How much will a new project cost?” to “How many defects remained after deployment?” Measurement enables teams to take corrective action—and to evaluate the impact of those actions.
Measurement is so important—and consistent in its execution—that it is defined and governed by an ISO standard. Per the standard, software is measured to:
- Create the quality of the current product or process
- Anticipate future qualities of the product or process
- Enhance the quality of a product or process
- Regulate the state of the project in relation to budget and schedule
For measurement to be effective, you must apply it in a top-down fashion, because the range of observable attributes in software—time, number of defects, complexity, lines of code, failure severity, defect density, and more—is too great for a bottom-up approach. Yet, even with a top-down approach, teams benefit greatly from a formalized way to identify what they should measure. That’s what makes the GQM approach so valuable.
Define what you need to measure
GQM is based on the theory that all measurements should be goal-oriented. There should be some rationale and need for collecting measurements, which in this case is software improvement, rather than collecting for the sake of collecting. Each measurement collected must inform one of the goals. Questions are then derived to help refine and articulate the goal. They specify what you need to know that would indicate whether the goal is being achieved.
Furthermore, the GQM approach is based on the assumption that organizations must perform three steps before they can measure in a purposeful way:
- Specify the goals for the business and its projects
- Ask questions to trace those goals to the data that defines them
- Provide a framework for interpreting the data (developing metrics) with respect to the stated goals
Define your goals, questions, and metrics
One of the most valuable aspects of the GQM approach is its ability to identify for software teams exactly what constitutes each element.
A goal is defined for an object, relative to a particular environment, and is informed by varied models of quality and points of view. It answers the questions:
- What problems are you encountering?
- What do your users, customers, or support personnel complain about?
- What improvements does the business need from the IT or software development organization?
Goals should be expressed in terms of beneficial outcomes such as predictability, early ROI, faster time to market, improved quality, and lower cost.
For most companies, software predictability is more important than bells and whistles. Therefore, many questions are framed from the perspective of increasing predictability. Questions should also:
- Characterize the way the assessment/achievement of a specific goal will be orchestrated
- Provide definition for the object of measurement (e.g., product, process, resource) with respect to a selected quality/predictability issue
- Facilitate the effort as teams work to determine their outcomes based on the target perspective
Metrics are the data associated with every question, allowing it to be answered in a quantifiable manner. The data can be objective (defects in current release) or subjective (level of user satisfaction). Leading indicators are preferable, but trailing indicators are acceptable when necessary.
Again, predictability in software is paramount in most current operating environments, yet prevailing factors mean there is no one-size-fits-all solution because:
- Every scenario is different. The recommended set of metrics for a large enterprise adopting an agile methodology would not be the same as that for an organization working to adopt a DevOps mindset. A portfolio team focused on business value and time to market would have yet another set.
- Goals should change over time. A metrics program should drive improvements, and as the system improves, teams must reevaluate the system and identify new goals.
- The most valuable metrics matter the most. Amassing metrics or using an old set of metrics that a project manager or IT manager dug up is unlikely to provide value, and can even harm the effort. Rather, over time, software teams should define a small set of the most valuable “evergreen” metrics and retire the old ones that no longer answer pertinent questions.
Finally, your overarching ambition should always be to focus on current improvement goals. These also vary from one company to the next. They can address a variety of objectives, but they should always be quantifiable. A few examples include:
- Shorten average software development schedules by (%) compared to (year)
- Raise defect removal efficiency from (%) to (%)
- Lower software defects from (%) to (%)
Go with what's proven
The GQM method is highly useful in delivery of all software, whether mission-critical or not. It is a proven mechanism for eliminating defects, improving overall project planning and reducing project costs. At a minimum, companies that fail to apply it extend their software lifecycles without adding value, potentially reducing the value of their final products.
To get your team started with the GQM methodology, position it as an exciting opportunity rather than just another approach they need to master. If buy-in is required from upper management, show how this approach will result in better software, faster.
Once you’re ready to go, tease your team with the idea that a new approach is going to make their jobs easier and their code cleaner. Ask them to read this article, then hold a Q&A session to address concerns. Finally, plan a kick-off meeting to boost optimism and confidence.
As with any new approach, innate resistance will be your greatest challenge. Reassure your team this approach won’t require them to give up the tools they currently use. It won’t require them to write or test a slew of new code. In fact, it will make their lives easier. That’s an idea everyone can get behind.
Keep learning
Take a deep dive into the state of quality with TechBeacon's Guide. Plus: Download the free World Quality Report 2022-23.
Put performance engineering into practice with these top 10 performance engineering techniques that work.
Find to tools you need with TechBeacon's Buyer's Guide for Selecting Software Test Automation Tools.
Discover best practices for reducing software defects with TechBeacon's Guide.
- Take your testing career to the next level. TechBeacon's Careers Topic Center provides expert advice to prepare you for your next move.
