MIL-HDBK-1823A tells how to plan an NDE experiment, design and
fabricate reliability demonstration specimens, acquire the system
performance data, and provides statistical methods for analyzing the
data to produce POD(a) curves, 95% confidence bounds, noise
analysis, and noise vs. detection trade-off curves. It presents
worked-out examples using real Hit/Miss and ‚ data, and serves as a
user’s manual for the mh1823 POD software.
These methods are statistical best-practices and have universal
applicability – NDE of engines, airframes, ground vehicles – subject
to the following limitations:
The NDE systems must produce output that can be reduced to either a
quantitative signal, ‚, or a binary response, hit/miss. (Images
therefore will require some pre-processing to provide either ‚ or
Hit/Miss as input to these analysis methods.)
The specimens must have targets with measurable characteristics,
like size or chemical composition. This precludes amorphous targets
like corrosion unless a specific measure can be associated with it,
such that other corrosion having that same measure will produce the
same output from the NDE equipment.
This mh1823 POD software assumes that the input data are correct.
That is, if the size is X, then that is the true size. If the
response is Y, then that is the true response. Situations where
these conditions cannot be ensured (e.g. where target sizing is only
approximate) will necessarily provide only approximate results. (The
problem of accurate crack sizing is discussed in Handbook Appendix
I.1 Departures from Underlying Assumptions – Crack Sizing and POD
Analysis of Images.)
This mh1823 POD software assumes that a POD curve goes to zero on
the left, and to one on the right. Data for which min(POD) > 0
(perhaps due to signal contamination by excessive background noise),
or max(POD) < 1 (resulting from random misses not related to target
size) cannot be correctly represented by a model for which min(POD)
= 0 and max(POD) =1. (See MIL-HDBK-1823, Appendix I-4 "Asymptotic
If the input data do not meet MIL-HDBK-1823 requirements, the mh1823
POD software may still produce an answer, but it will be WRONG.
Mid 1980s: A working group from the USAF, UDRI, GEAE, P&W, and Garrett,
later Allied-Signal, now Honeywell, produced MIL-HDBK-1823, "Nondestructive Evaluation System
Reliability Assessment." While it would be some years before an official
publication was available, the draft became the de facto world standard for
establishing quantitatively the effectiveness of inspections by measuring POD
(Probability of Detection). Charles Annis, P.E. was the primary author and
overall editor of the first edition of MIL-HDBK-1823. That first Working Group
included Chuck Annis and Steve Cargill at P&W, Dennis Corbly and Jim Griffiths
at GE, Al Berens and Pete Hovey of UDRI, and Vicki Panhuise at Allied-Signal.
Sharon Vukelich was the primary Air Force contributor.
1993: NATO AGARD (North Atlantic Treaty Organization, Advisory Group
for Aerospace Research and Development) sponsored 2-day POD Short
Course based on MIL-HDBK-1823 that Chuck Annis and Sharon Vukelich
presented in Ankara, Turkey, Lisbon, Portugal, Patras, Greece, and
Late 1990s: USAF officially publishes MIL-HDBK-1823, 30 April, 1999,
and again in April 2004. By then the Handbook had been the de facto
world standard for nearly two decades.
Early 2000s: Model-Assisted POD - The MAPOD Working Group was formed
in Austin in 2003 with impetus from James Malas, Ph.D., Chief,
Nondestructive Evaluation Branch, Materials and Manufacturing
Directorate, Air Force Research Laboratories. Bruce Thompson
and Bill Meeker were key contributors.
2006: Under AFRL sponsorship, Chuck Annis began work to incorporate
two decades of NDE progress in an update of MIL-HDBK-1823, under
Agreement: 06-S508-010-16-C1, Prime contract: F33615-03-D-5204.
The work was largely funded by USAF AFRL/MLLP, Nondestructive Evaluation
Branch, Materials and Manufacturing Directorate, James C. Malas, Ph.D.,
Branch Chief. Special thanks to Charles Buynak, and to Jeremy Knopp who
was the technical monitor. Numerous colleagues provided counsel for the
update including Steve Cargill of Aerospace Structural Integrity, Floyd
Spencer at Sandia National Laboratories, Al Berens at UDRI, and Bill Meeker
and Bruce Thompson of Iowa State.
The work was completed in February, 2007, although it was not officially
released by the Air Force until April, 2009.
The deliverables of the 2006-2007 contract to update MIL-HDBK-1823
included the draft of the updated Handbook and algorithms necessary
to implement the methods described in the Handbook. Software was not a deliverable item.
However it was soon obvious that delivering descriptions of statistical
algorithms, with no convenient implementation of them, would be of limited
utility, so I wrote the mh1823 POD software suite to make the handbook's
methods more widely usable. The software is © Charles Annis, P.E. and is not
in the Public Domain, however it can be obtained, gratis, by sending a request to:
Charles.Annis@StatisticalEngineering.com, with "mh1823" in the title block.
The software is in use world-wide, including some university programs.
7 April, 2009: The 2007 update was released by the USAF. Download a
NOTE: The mh1823 POD algorithms use R, the most powerful statistical
and graphics engine available anywhere, for all data manipulation,
statistical analysis, and graphics. Because R is open-source (and
free), and because all of the algorithms and methods developed here
are based on modern, well documented statistical best practices
described in the open literature, there is nothing proprietary in
this mh1823 POD software. Since there are no restrictions on its
use, the mh1823 POD software can be used as a universal standard for
performing Probability of Detection (POD) analysis.
Click here to see the mh1823 POD software version history.