Statistically Comparing Two Alternatives - a PDH Online Course for Engineers

Statistically Comparing Two Alternatives

Course Overview

Due to measurement noise, it often can be difficult to determine whether a change made to a system had any real impact. Measurement noise also makes it difficult to compare the performance of two different systems. In this course, you will extend your understanding of confidence intervals to determine if there is a statistically significant difference in the performance of two alternatives. You will learn about both before-and-after comparisons and comparisons using unpaired measurements.

This course includes a multiple-choice quiz at the end, which is designed to enhance the understanding of the course materials.

Learning Objective

After completing this 2-hour course, you will be able to:

Determine if a statistically significant difference exists in the performance of two different systems;
Determine if a change made to a system produces a statistically significant difference in performance;
Understand when to compare the difference of the mean values; and
Understand when to compare the mean of the differences.

Reading Assignment

The reading assignment for this course is Chapter 5.1 of Measuring Computer Performance: A Practitioner's Guide, David J. Lilja, Cambridge University Press, 2000.

If you don't have this book, you can purchase Chapter 1 in PDF format online at eBooks.com for a modest cost. The price for this course listed on this website does not include the cost of purchasing the chapter through eBooks.com. However, the price has been reduced to compensate for the cost of purchasing the chapter required. If you plan to take all 6 courses (E132 to E137) based on this book, you may consider to purchase a hard copy of the book or the entire book in PDF format online through eBooks.com.

Key Terms

Before-and-after comparisons
Mean of the differences
Noncorresponding measurements
Unpaired observations
Difference of the means
Pooled standard deviation
Effective number of degrees of freedom

Study Notes

Errors (noise) in a set measurements introduce uncertainty into the results you can obtain from those measurements. If you are trying to compare the performance of two different computer systems, for instance, the noise in your measurements may obscure whether or not there is a real difference between the systems.

In this course, you will learn how confidence intervals can be used to determine if there is a statistically significant difference between two systems in spite of the noise in your measurements.

To make this type of comparison, you typically would take several measurements of the execution time of the same benchmark program running on two different systems. You then find a confidence interval for the difference of the means of each set of measurements. If the confidence interval of this difference of the means includes 0, you must conclude that there is not a statistically significant difference between the two systems, at the given confidence level. This approach is quite general and can be applied even when you do not have the same number of measurements on each system. There are a few small things you need to be careful of when applying this technique, such as how to compute the standard deviation of the difference of the means and how to determine the effective number of degrees of freedom.

If you know that there is some direct correspondence between individual measurements of the two different systems, you can apply the before-and-after comparison technique. This technique is most appropriate when you are trying to determine if a change you have made to a system has any statistically significant impact. For instance, you may be trying to determine if adding more memory to a system produces a statistically significant change in performance. This technique is very similar to the one described above, except that you compute a confidence interval for the mean of the differences of each pair of measurements. This technique is quite straight-forward after you understand the basics of confidence intervals. It does require that the same number of measurements be made on each before-and-after configuration of the system, though.

Finally, a similar type of confidence interval approach can be used to determine if there is a statistically significant difference between two proportions.

Quiz

Once you finish studying the above course content, you need to take a quiz to obtain the PDH credits.

DISCLAIMER: The materials contained in the online course are not intended as a representation or warranty on the part of PDH Center or any other person/organization named herein. The materials are for general information only. They are not a substitute for competent professional advice. Application of this information to a specific project should be reviewed by a registered architect and/or professional engineer/surveyor. Anyone making use of the information set forth herein does so at their own risk and assumes any and all resulting liability arising therefrom.