Both IPCC and it´s reviewer, InterAcademy Council, messed up on “Quantified measures of uncertainty”!

IPCC´s “Guidance note on expression of uncertainty” is a particularly important governing document for IPCC. The document is affecting all the working groups. There are two main themes in the guidance note on expression of uncertainty:

– Qualitatively expression of confidence
– Quantified measures of uncertainty

InterAcademy Council gave bad advice on both.

This post is about Quantified measures of uncertainty.

Recommendations from the InterAcademy Council on “communicating uncertainty”

On 10 March 2010, the Secretary-General of the United Nations Ban Ki-moon and IPCC Chair Pachauri requested InterAcademy Council (IAC)  to undertake a review of the processes and procedures of the InterGovernmental Panel on Climate Change.

The report was published in October 2010:
Climate Change Assessments, Review of the Processes & Procedures of the IPCC (2010)

This post is about the advices from IAC on the expression of uncertainty. From the foreword it is clear that IAC recognize the importance of proper handling of scientific uncertainty, and that the consequences is of great importance to humanity:

Recognizing that the issue of climate change is of great importance to humanity, is complex in regard to scientific uncertainty, and has potential long-term consequences, the IAC Board agreed to undertake this review.
– Foreword

The importance of uncertainty is further evident as recommendations on communication of uncertainty is one of the most significant key recommendations by IAC.

Key recommendations
The Committee’s main recommendations relate to IPCC’s governance and management, its review process, characterizing and communicating uncertainty, communications, and transparency in the assessment process.

Everything should be lined up for IAC to give United Nations proper advice on the expression of uncertainty then.

The advices from IAC on the expression of uncertainty are summarized in the section “Characterizing and communicating uncertainty”:

Recommendation: Quantitative probabilities (as in the likelihood scale) should be used to describe the probability of well-defined outcomes only when there is sufficient evidence. Authors should indicate the basis for assigning a probability to an outcome or event (e.g., based on measurement, expert judgment, and/or model runs).
– Page xv

IPCC guidance note on consistent treatment of uncertainty

These recommendations resulted in the revision of the IPCC document:
Guidance Note for Lead Authors of the IPCC Fifth Assessment Report on Consistent Treatment of Uncertainties … 6-7 July 2010

In annex B it is stated:

Annex B: Addressing the InterAcademy Council Recommendations
The 2010 independent review of the IPCC by the InterAcademy Council (IAC)3, released on August 30, 2010, included six recommendations related to the evaluation of evidence and treatment of uncertainty in IPCC reports. These recommendations are listed below, with brief summaries explaining how the AR5 guidance addresses their key elements.

And don´t ask me how a review released August 30, 2010 can be taken into account in a guidance document dated 6-7 July 2010. I have no idea how that came about.

These guidance notes are intended to address the recommendation by IAC:

These notes refine background material provided to support the Third and Fourth Assessment Reports 1,2,3; they represent the results of discussions at a Cross-Working Group Meeting on Consistent Treatment of Uncertainties convened in July 2010. They also address key elements of the recommendations made by the 2010 independent review of the IPCC by the InterAcademy Council.

The Guidance on consistent treatment of uncertainties put an important role on the review editors:

Review Editors play an important role in ensuring consistent use of this calibrated language within each Working Group report. Each Working Group will supplement these notes with more specific guidance on particular issues consistent with the common approach given here.

So everything should be just fine then?
IAC recommends IPCC to use a qualitative level-of-understanding scale and Quantitative probabilities when there is sufficient evidence – and IPCC follow that advice.

There are two main themes in the guidance note on expression of uncertainty:
– Qualitatively expression of confidence
– Quantified measures of uncertainty

The AR5 will rely on two metrics for communicating the degree of certainty in key findings:

• Confidence in the validity of a finding, based on the type, amount, quality, and consistency of evidence (e.g., mechanistic understanding, theory, data, models, expert judgment) and the degree of agreement. Confidence is expressed qualitatively.
• Quantified measures of uncertainty in a finding expressed probabilistically (based on statistical analysis of observations or model results, or expert judgment).

Guidance Note for Lead Authors of the IPCC Fifth Assessment Report on Consistent Treatment of Uncertainties

Well – everything could have been fine – if the advice were good, and if the advice were followed in a proper way – but everything isn´t fine. I will tell you why.

First, I think a good way of working is to check if there are any existing international standards relevant to the expression of uncertainty in estimates. Actually, there is:

The international standard for expression of uncertainty!

To understand the shortcomings of IPCC guidance note on expression of uncertainty, I will first set the standard by referring how uncertainty should have been reported in accordance with international standards.

There exists one internationally recognized guideline for expression of uncertainty:
Guide to the expression of uncertainty in measurement.
The guideline is freely available at the link above.

The name of the Guideline is: Guide to expression of uncertainty in measurement . However – measurement must be understood in a wider context as any estimate of any quantity.

The following is stated about the guide in the introduction:
“This Guide establishes general rules for evaluating and expressing uncertainty in measurement that are intended to be applicable to a broad spectrum of measurements…

This Guide was prepared by a joint working group consisting of experts nominated by the Bureau International des Poids et Measures (BIPM), the International Electrotechnical Commission (IEC), the International Organization for Standardization (ISO), and the International Organization of Legal Metrology (OIML)…

The Working Group was convened by the Bureau International des Poids et Mesures (BIPM) in response to a request of the International Committee for Weights and Measures (CIPM)…

The ClPM Recommendation is the only recommendation concerning the expression of uncertainty in measurement adopted by an intergovernmental organization…

The following seven organizations supported the development of this Guide, which is published in their name:
BIPM: Bureau International des Poids et Measures
IEC: International Electrotechnical Commission
IFCC: International Federation of Clinical Chemistry
ISO: International Organization for Standardization
IUPAC: International Union of Pure and Applied Chemistry
IUPAP: International Union of Pure and Applied Physics
OlML: International Organization of Legal Metrology ..”

On reporting of uncertainty

Guide to the expression of uncertainty in measurement covers all aspects of quantifying and reporting uncertainty. This post only address reporting of uncertainty – not how uncertainty is estimated or quantified in the first place.

Reporting of uncertainty is covered in section 7 in the Guide to expression of uncertainty in measurement. The main points are:

Ref.: 7.1.4  in Guide to the expression of uncertainty …:

Although in practice the amount of information necessary to document a measurement result depends on its intended use, the basic principle of what is required remains unchanged: when reporting the result of a measurement and its uncertainty, it is preferable to err on the side of providing too much information rather than too little. For example, one should

a) describe clearly the methods used to calculate the measurement result and its uncertainty from the experimental observations and input data;
b) list all uncertainty components and document fully how they were evaluated;
c) present the data analysis in such a way that each of its important steps can be readily followed and the calculation of the reported result can be independently repeated if necessary;
d) give all corrections and constants used in the analysis and their sources.

A test of the foregoing list is to ask oneself “Have I provided enough information in a sufficiently clear manner that my result can be updated in the future if new information or data become available?”

Ref.: 7.2.3 in Guide to the expression of uncertainty …:

Simply put, the result of an estimate should be reported by:
– giving a full description of how the measurand Y is defined
– stating the result of the measurement as Y = y ± U and give the units of y and U
– giving the approximate level of confidence associated with the interval y ± U
– state how the level of confidence was determined;”

Now we know how uncertainty in an estimate should be reported. However, an example might be appropriate.

Expression of uncertainty in accordance with the international standard

The following is an example of how uncertainty could have been expressed in accordance with the international standard: Guide to the expression of uncertainty in measurement.

Regarding confidence level – the following extract from Wikipedia is clarifying:
“In statistics, the so-called 68–95–99.7 rule is a shorthand used to remember the percentage of values that lie within a band around the mean in a normal distribution with a width of one, two and three standard deviations, respectively; more accurately, 68.27%, 95.45% and 99.73% of the values lie within one, two and three standard deviations of the mean, respectively.» (Wikipedia; 68–95–99.7_rule)

If, for example, the uncertainty of the variable “equilibrium climate sensitivity” is normal distributed (that is to say if the uncertainty follows the central limit theorem). A correct way of expressing equilibrium climate sensitivity could then have been:

Equilibrium climate sensitivity is 3 °C +/- 1,5 °C at 68 % confidence level.
Which is the same as to say that:
Equilibrium climate sensitivity is 3 °C +/- 3 °C at 95 % confidence level.
or:
Equilibrium climate sensitivity is 3 °C +/- 4,5 °C at 99,7 % confidence level.
or even:
Equilibrium climate sensitivity is 3 °C +/- 6 °C at 99,99 % confidence level.

A 95% confidence level is commonly used for expression of uncertainty. However, all the statement above are equivalent statements about uncertainty in the equilibrium climate sensitivity.The statements above may seem different, but they mean exactly the same!

In addition, the statements about uncertainty should be followed by the information required by the standard.

IPCC´s weird guideline on quantification of uncertainty!

It is then time to see what IPCC did, and how IPCC recommends to express uncertainty in their governing document: Guidance Note for Lead Authors of the IPCC Fifth Assessment Report on Consistent Treatment of Uncertainties

The following table is central to the quantification of uncertainty by IPCC.

What immediately comes to mind is that in terms of a normal distribution, 66-100% probability is an enormous range. To illustrate this, it would be like saying that the probability distribution can be anything between the blue and the yellow probability distribution in the figure below. Actually, it is worse – a 100 % probability would require a very narrow curve. In terms of a gaussian normal distribution, it is extremely stupid to say that the term “likely” is equivalent to a 66-100% likelihood of an outcome in a certain range.

The only thing which is certain by this way of quantifying uncertainty is that IPCC is totally incompetent within the expression of uncertainty.

The likelihood scale in Table 1. is followed by the following guideline to explain it´s use:

10) Likelihood, as defined in Table 1, provides calibrated language for describing quantified uncertainty. It can be used to express a probabilistic estimate of the occurrence of a single event or of an outcome (e.g., a climate parameter, observed trend, or projected change lying in a given range). Likelihood may be based on statistical or modeling analyses, elicitation of expert views, or other quantitative analyses. The categories defined in this table can be considered to have “fuzzy” boundaries. A statement that an outcome is “likely” means that the probability of this outcome can range from ≥66% (fuzzy boundaries implied) to 100% probability. This implies that all alternative outcomes are “unlikely” (0-33% probability). When there is sufficient information, it is preferable to specify the full probability distribution or a probability range (e.g., 90- 95%) without using the terms in Table 1. “About as likely as not” should not be used to express a lack of knowledge (see Paragraph 8 for that situation). Additionally, there is evidence that readers may adjust their interpretation of this likelihood language according to the magnitude of perceived potential consequences.11

The following sentence reveals that the terms in Table 1. Likelihood scale is supposed to be used when there is insufficient information:

When there is sufficient information, it is preferable to specify the full probability distribution or a probability range (e.g., 90- 95%) without using the terms in Table 1.

But anyhow, within expression of uncertainty, it is inappropriate to specify a probability range as indicated in this sentence. That would be like expressing the probability of a probability – and that is pretty weird.

To give an idea how extensively the weird terms invented by IPCC are used in the report, I searched for the term “likely” in the contribution from Working Group I; on the scientific basis; to the fifth assessment report by IPCC. The term likely was used on 426 out of 1552 pages in the report.

The terms invented by IPCC to express likelihood might seem impressive to the untrained eye. To the trained eye, the terms are nothing but weird.

However, It should be mentioned that in the contribution from Working Group I to the fifth assessment report by IPCC, the following note indicates a proper expression of uncertainty:

In the WGI contribution to the AR5, uncertainty is quantified using 90% uncertainty intervals unless otherwise stated. The 90% uncertainty interval, reported in square brackets, is expected to have a 90% likelihood of covering the value that is being estimated.

• Note on Page 5.

This way of expressing uncertainty could have been in accordance with the international guideline if the uncertainty estimate was also followed by the information required by the standard – which in general it isn´t.

Summary

The InterAcademy Council could have advised United Nations climate panel IPCC to quantify, express and report uncertainty in accordance with the only internationally accepted guideline for expression of uncertainty: Guide to the expression of uncertainty in measurement.   That would have saved them for a lot of words.

The following seven organizations supported the development of that Guide, which was published in their name:
BIPM: Bureau International des Poids et Measures
IEC: International Electrotechnical Commission
IFCC: International Federation of Clinical Chemistry
ISO: International Organization for Standardization
IUPAC: International Union of Pure and Applied Chemistry
IUPAP: International Union of Pure and Applied Physics
OlML: International Organization of Legal Metrology ..”

Unfortunately – The InterAcademy Council failed to notice, mention and recommend this guideline for IPCC.

Rather than making up their own stuff in a hasty way, United Nations climate panel IPCC could have referred to the international standard themselves. IPCC could have said that all uncertainty estimates in the writings of the panel should be quantified, documented and reported in accordance to:  Guide to the expression of uncertainty in measurement. (Freely available).

Unfortunately – United Nations climate panel IPCC also failed to notice and impose this guideline on the working groups.

IPCC´s way of expressing uncertainty is totally inept, weird and in disagreement with the only recognized standard for expression of uncertainty.

And – there are no excuses – a search for the term “uncertainty” would have led even the most ignorant to the Wikipedia article uncertainty – which states:

The most commonly used procedure for calculating measurement uncertainty is described in the “Guide to the Expression of Uncertainty in Measurement” (GUM) published by ISO.

The bottom line is that both InterAcademy Council and United Nations climate panel messed up on uncertainty.