Measuring Code Coverage in an Embedded System

Many software developers measuring code coverage in an embedded system use a test system very different than their original design. The engineer knows it’s a poor methodology, but it can easily happen. Like the Boiling Frog Fable, unnoticed incremental temperature changes lead to the poor frog’s demise...  

Let’s explore the plight of the software developer.  

Code coverage often occurs late in the development process. At this point both the hardware and software have been optimized for cost and the system closely meets specified requirements. Starting code coverage analysis at this stage often means adding “printf” statements or some type of serial transmission code into every function of code so the microcontroller can say “Program Counter = X.” Once these code segments are added throughout the project, code resource requirements often exceed the limits of the cost-optimized controller. One alternative is to replace the microcontroller with a larger memory variant (hopefully one is available in the same package). Execution speed has likely bogged down with extra code, so maybe the MCU clock speed can be increased to boost system performance? 

The device also needs a communication channel to output information. Wires are needed to splice in a communication port and transceiver for a computer to capture the output stream. If no extra pins are available, a new board is needed to accommodate a higher pin count microcontroller. Alternatively, required functionality could be replaced with code coverage routines which requires multiple software versions to test the code in blocks.  

Now that code coverage functionality has been (painfully) added, the resulting datalog file is massive, unwieldy and difficult to correlate to the original source code. Any quality-minded developer would ask, “How does this test system represent the original design at all?” 

Let’s examine the benefits and limitations of code coverage, how MPLAB^® Code Coverage works – and how to use it to measure code coverage in an embedded system. 

Benefits and Limitations 

A proper testing methodology is paramount to developing a robust embedded system and correctly utilizing a code coverage tool is only way to ensure your tests are doing their job. Wise code developers understand “untested code is broken code.” Anyone who has lived through a product recall understands the value code coverage brings to improving product quality.  

Code developers who have minimum code coverage requirements are often surprised at how low their initial code coverage results are. It can also be very difficult to write test code that covers specific flows. Using a tool that requires large modifications to the original design further complicates this effort.  

Code coverage is a deceivingly simple equation: 

                                                          Amount of Code Executed
                                                              ________________________
                                                               Total Amount of Code
 

A savvy engineer writing embedded code in C or C++ knows their code compiles into assembly might ask, “Which code is calculated?” This is an important question to understand because the answer varies across tool providers and it can significantly impact this equation’s result. We’ll cover MPLAB Code Coverage implementation later in this blog.  

A common misconception is that code with 100% coverage is “better” than code with 95%. Code coverage percentage indicates how well the test suite exercises the application. It gives no indication of code inefficiency, missing functionality, performance or correctness. 

It’s also important to distinguish testing methodology from a code coverage tool. Searching the internet, you can find code coverage tools that claim statement coverage, function coverage, branch coverage, MC/DC, etc. These types of descriptions can be confusing because the vendor is using testing methodology terms to describe their code coverage tool. These are two distinct concepts. A testing methodology such as branch coverage or MC/DC describes how source code will be exercised. In contrast, a code coverage tool determines which machine instructions were executed. Because one line of a high-level source code can result in multiple machine instructions, the tool can indicate a line of source code as fully or partially executed or not executed at all.  

How MPLAB Code Coverage Works

For starters, MPLAB Code Coverage is designed for small embedded systems. It’s special because it leverages knowledge that only MPLAB XC Compilers have about your project. No tool knows your code better than your compiler. Since Microchip makes our own compilers, our code coverage tool can leverage this information.  

Our compilers know the minimum number of instrumentation points needed and where to place them to minimize impact to code size and performance. Our compilers also know how much data memory is available and uses it optimally to store runtime information. Our analysis shows that our MPLAB Code Coverage tool uses one half to one third the instrumentation points of competitive offerings. And because it uses only single bit-set instructions as markers, the comparison to competitors’ solutions based on “printf” instructions isn’t close. MPLAB Code Coverage impact is so small, typically <1%, that we could not find a project that was unable to perform analysis using its original hardware.  

MPLAB Code Coverage has handy tools for navigating results. Every line of source code and assembly instruction is marked as either fully or partially executed or not executed at all. To clarify the question, our tool analyzes coverage at the assembly instruction level. So coverage information is natively displayed in MPLAB X Integrated Development Environment (IDE) Program Memory Window in assembly but assimilated to show various levels of execution in the source code window as well. Quick jumps between source and assembly instruction code is supported for quick assessment of partially covered code.  

Code coverage is shown for individual files and functions as well as project level. Highlights can be easily toggled on and off for easier reading. Dashboards track code coverage progress and HTML reports can be generated at a button’s touch. Best of all, MPLAB Code Coverage is fully integrated into Microchip’s MPLAB X IDE which provides easy task switches between tools and displayed results in configurable output windows.  

Easily Measure Code Coverage in an Embedded System 

MPLAB Code Coverage is the simplest code coverage tool on the market, just enable it in MPLAB X IDE, then build, program, exercise and read your device.  

The tool cleverly uses data memory to store coverage information that is decoded and displayed within the IDE. This process is so unique, it’s protected by U.S. Patent Number 10402309. It supports all of Microchip’s PIC, SAM, AVR and dsPIC offerings, works on both Free and Pro versions of MPLAB XC Compilers and is available as a workstation license.  

As a cost-effective solution MPLAB Code Coverage can be purchased for $799 and can be downloaded from Microchip Purchasing and Client Services (microchipdirect) or purchased from your favorite Microchip distributor.