Outliers

“Outliers”

Outliers are often infuriating, but they can be very informative too.

Unexpected …

“Outliers” are observations that are unexpectedly different from their sisters. They are not necessarily wrong and are often the most interesting and informative observations in the sample.

Sometimes, however, they are wrong, a result of a data transcription error, or some other attributable cause. If it can be determined that such an observation does not belong, it may be removed. BUT convenience is NOT a sufficient justification. Many an analysis has been for naught because an annoying “outlier” was ignored, sometimes with very expensive consequences.

Extreme observations

In any event, valid or spurious, these extreme observations can have an unsettling influence on a mathematical model. (Sometimes they can have an unsettling influence on the analyst, too.) While there is no prescription that works in every circumstance, it is prudent first to determine if the observation is errant and can be removed. (If so, what steps have you taken to assure that what produced this one won’t cause another?)

If you still must contend with the unwelcomed surprise, what does it mean? What is it telling you (perhaps even screaming at you)? What about your world-view is inconsistent with this result? Should you reconsider your perspective? What possible explanations have you not yet considered?

(Here is where the temptation to cleverness can be your undoing. Mother Nature is unimpressed with eloquence and logical legerdemain. An explanation must be correct, not merely plausible, and consensus is a poor measure of veracity. You may be able to talk yourself into believing something, but Mother Nature will do whatever she will, even if you’ve convinced yourself otherwise.)

If you must keep it, and still can’t explain it, how much should you let an outlier influence your model? Should you use conventional practices and live with the skewed consequences? (Sometimes you should.) Or should you choose methods that are less sensitive to such extreme observations?

Unmasking an outlier is some of the most interesting detective work in engineering, requiring patience, insight and experience.