| From | Ben Santer | Date | Mon, 02 Jun 2008 09:32:01 -0700 |
|---|---|---|---|
| To | Carl Mears | ||
| CC | Steven Sherwood, Peter Thorne, Leopold Haimberger, Karl E.Taylor, Tom Wigley, John Lanzante, Susan Solomon, Melissa Free, peter gleckler, Phil Jones, Thomas R Karl, Steve Klein, carl mears, Doug Nychka, Gavin Schmidt, Frank Wentz | ||
| Subject | Re: Our d3* test | ||
Dear Carl, This issue is now covered in the version of the manuscript that I sent out on Friday. The d2* and d3* statistics have been removed. The new d1* statistic DOES involve the standard error of the model average trend in the denominator (together with the adjusted standard error of the observed trend; see equation 12 in revised manuscript). The slight irony here is that the new d1* statistic essentially reduces to the old d1* statistic, since the adjusted standard error of the observed trend is substantially larger than the standard error of the model average trend... With best regards, Ben Carl Mears wrote: > Hi > > I think I agree (partly, anyway) with Steve S. > > I think that d3* partly double counts the uncertainty. > > Here is my thinking that leads me to this: > > Assume we have a "perfect model". A perfect model means in this context > 1. Correct sensitivities to all forcing terms > 2. Forcing terms are all correct > 3. Spatial temporal structure of internal variability is correct. > > In other words, the model output has exactly the correct "underlying" > trend, but > different realizations of internal variability and this variability has > the right > structure. > > We now run the model a bunch of times and compute the trend in each case. > The spread in the trends is completely due to internal variability. > > We compare this to the "perfect" real world trend, which also has > uncertainty due > to internal variability (but nothing else). > > To me either one of the following is fair: > > 1. We test whether the observed trend is inside the distribution of > model trends. The uncertainty in the > observed trend is already taken care of by the spread in modeled trends, > since the representation of > internal uncertainty is accurate. > > 2. We test whether the observed trend is equal to the mean model trend, > within uncertainty. Uncertainty here is > the uncertainty in the observed trend s{b{o}}, combined with the > uncertainty in the mean model trend (SE{b{m}}. > > If we use d3*, I think we are doing both these at once, and thus double > counting the internal variability > uncertainty. Option 2 is what Steve S is advocating, and is close to > d1*, since SE{b{m}} is so small. > Option 1 is d2*. > > Of course the problem is that our models are not perfect, and a > substantial portion of the spread in > model trends is probably due to differences in sensitivity and forcing, > and the representation > of internal variability can be wrong. I don't know how to separate the > model trend distribution into > a "random" and "deterministic" part. I think d1* and d2* above get at > the problem from 2 different angles, > while d3* double counts the internal variability part of the > uncertainty. So it is not surprising that we > get some funny results for synthetic data, which only have this kind of > uncertainty. > > Comments? > > -Carl > > > > > On May 29, 2008, at 5:36 AM, Steven Sherwood wrote: > >> >> On May 28, 2008, at 11:46 PM, Ben Santer wrote: >>> >>> Recall that our current version of d3* is defined as follows: >>> >>> d3* = ( b{o} - < >>> >>> where >>> >>> b{o} = Observed trend >>> < >>> s{ >>> s{b{o}} = Standard error of the observed trend (adjusted for >>> autocorrelation effects) >> >> Shouldn't the first term under sqrt be the standard deviation of the >> estimate of < >> than the standard deviation of >> equivalent to a z-score, relevant to the null hypothesis that models >> on average get the trend right. As written, I think the distribution >> of d3* will have less than unity variance under this hypothesis. >> >> SS >> >> >> ----- >> Steven Sherwood >> Steven.Sherwood@yale.edu >> Yale University ph: 203 >> 432-3167 >> P. O. Box 208109 fax: 203 >> 432-3134 >> New Haven, CT 06520-8109 >> http://www.geology.yale.edu/~sherwood >> >> >> >> >> >> > -- ---------------------------------------------------------------------------- Benjamin D. Santer Program for Climate Model Diagnosis and Intercomparison Lawrence Livermore National Laboratory P.O. Box 808, Mail Stop L-103 Livermore, CA 94550, U.S.A. Tel: (925) 422-2486 FAX: (925) 422-7675 email: santer1@llnl.gov ---------------------------------------------------------------------------- | |||