Rejection odds and rejection ratios: A proposal for statistical practice in testing hypotheses

Much of science is (rightly or wrongly) driven by hypothesis testing. Even in situations where the hypothesis testing paradigm is correct, the common practice of basing inferences solely on p-values has been under intense criticism for over 50 years. We propose, as an alternative, the use of the odds of a correct rejection of the null hypothesis to incorrect rejection. Both pre-experimental versions (involving the power and Type I error) and post-experimental versions (depending on the actual data) are considered. Implementations are provided that range from depending only on the p-value to consideration of full Bayesian analysis. A surprise is that all implementations -- even the full Baye…

Optimal Reporting of Predictions

Abstract Consider a problem in which you and a group of other experts must report your individual predictive distributions for an observable random variable X to some decision maker. Suppose that the report of each expert is assigned a prior weight by the decision maker and that these weights are then updated based on the observed value of X. In this situation you will try to maximize your updated, or posterior, weight by appropriately choosing the distribution that you report, rather than necessarily simply reporting your honest predictive distribution. We study optimal reporting strategies under various conditions regarding your knowledge and beliefs about X and the reports of the other e…

Bayesian Design of “Successful” Replications

Replication of experiments is commonin applied research. However, systematic studies of the goals and motivations of a “replication” are rare. As a consequence, there does not seem to be a precise notion of what a “success” when replicating means. This article discusses some of the possible goals for replication; this leads to different (but precise) notions of “success” when replicating. Bayesian hierarchical models allow for a flexible and explicit incorporation of the assumed relationship among the experiments. Bayesian predictive distributions are a natural tool to compute the probability of the replication being successful, and hence to design the replication so that the probability of…

The Effective Sample Size

Model selection procedures often depend explicitly on the sample size n of the experiment. One example is the Bayesian information criterion (BIC) criterion and another is the use of Zellner–Siow priors in Bayesian model selection. Sample size is well-defined if one has i.i.d real observations, but is not well-defined for vector observations or in non-i.i.d. settings; extensions of critera such as BIC to such settings thus requires a definition of effective sample size that applies also in such cases. A definition of effective sample size that applies to fairly general linear models is proposed and illustrated in a variety of situations. The definition is also used to propose a suitable ‘sc…

What Bayesians Expect of Each Other

Abstract Our goal is to study general properties of one Bayesian's subjective beliefs about the behavior of another Bayesian's subjective beliefs. We consider two Bayesians, A and B, who have different subjective distributions for a parameter θ, and study Bayesian A's expectation of Bayesian B's posterior distribution for θ given some data Y. We show that when θ can take only two values, Bayesian A always expects Bayesian B's posterior distribution to lie between the prior distributions of A and B. Conditions are given under which a similar result holds for an arbitrary real-valued parameter θ. For a vector parameter θ we present useful expressions for the mean vector and covariance matrix …

Bayesian measures of surprise for outlier detection

From a Bayesian point of view, testing whether an observation is an outlier is usually reduced to a testing problem concerning a parameter of a contaminating distribution. This requires elicitation of both (i) the contaminating distribution that generates the outlier and (ii) prior distributions on its parameters. However, very little information is typically available about how the possible outlier could have been generated. Thus easy, preliminary checks in which these assessments can often be avoided may prove useful. Several such measures of surprise are derived for outlier detection in normal models. Results are applied to several examples. Default Bayes factors, where the contaminating…

Morris H. Degroot

Prior-based Bayesian information criterion

We present a new approach to model selection and Bayes factor determination, based on Laplace expansions (as in BIC), which we call Prior-based Bayes Information Criterion (PBIC). In this approach, the Laplace expansion is only done with the likelihood function, and then a suitable prior distribution is chosen to allow exact computation of the (approximate) marginal likelihood arising from the Laplace approximation and the prior. The result is a closed-form expression similar to BIC, but now involves a term arising from the prior distribution (which BIC ignores) and also incorporates the idea that different parameters can have different effective sample sizes (whereas BIC only allows one ov…

Applications and Limitations of Robust Bayesian Bounds and Type II MLE

Three applications of robust Bayesian analysis and three examples of its limitations are given. The applications that are reviewed are the development of an automatic Ockham’s Razor, outlier detection, and analysis of weighted distributions. Limitations of robust Bayesian bounds are highlighted through examples that include analysis of a paranormal experiment and a hierarchical model. This last example shows a disturbing difference between actual hierarchical Bayesian analysis and robust Bayesian bounds, a difference which also arises if, instead, a Type II MLE or empirical Bayes analysis is performed.

Inferencia bayesiana sobre el coeficiente de correlacion de una poblacion normal bivariante

En el presente articulo se utiliza el metodo de maximizacion de la informacion desconocida para deducir las distribuciones inicial y final de referencia cuando se trata de hacer inferencias sobre el coeficiente de correlacion de una poblacion normal bivariante

Bayesian Checking of the Second Levels of Hierarchical Models

Hierarchical models are increasingly used in many applications. Along with this increased use comes a desire to investigate whether the model is compatible with the observed data. Bayesian methods are well suited to eliminate the many (nuisance) parameters in these complicated models; in this paper we investigate Bayesian methods for model checking. Since we contemplate model checking as a preliminary, exploratory analysis, we concentrate on objective Bayesian methods in which careful specification of an informative prior distribution is avoided. Numerous examples are given and different proposals are investigated and critically compared.

Criteria for Bayesian model choice with application to variable selection

In objective Bayesian model selection, no single criterion has emerged as dominant in defining objective prior distributions. Indeed, many criteria have been separately proposed and utilized to propose differing prior choices. We first formalize the most general and compelling of the various criteria that have been suggested, together with a new criterion. We then illustrate the potential of these criteria in determining objective model selection priors by considering their application to the problem of variable selection in normal linear models. This results in a new model selection objective prior with a number of compelling properties.

A Bayesian analysis of a queueing system with unlimited service

Abstract A queueing system occurs when “customers” arrive at some facility requiring a certain type of “service” provided by the “servers”. Both the arrival pattern and the service requirements are usually taken to be random. If all the servers are busy when customers arrive, they usually wait in line to get served. Queues possess a number of mathematical challenges and have been mainly approached from a probability point of view, and statistical analysis are very scarce. In this paper we present a Bayesian analysis of a Markovian queue in which customers are immediately served upon arrival, and hence no waiting lines form. Emergency and self-service facilities provide many examples. Techni…

Bayesian prediction inM/M/1 queues

Simple queues with Poisson input and exponential service times are considered to illustrate how well-suited Bayesian methods are used to handle the common inferential aims that appear when dealing with queue problems. The emphasis will mainly be placed on prediction; in particular, we study the predictive distribution of usual measures of effectiveness in anM/M/1 queue system, such as the number of customers in the queue and in the system, the waiting time in the queue and in the system, the length of an idle period and the length of a busy period.

Incorporating Uncertainties into Traffic Simulators

A Bayesian analysis of the thermal challenge problem

Abstract A major question for the application of computer models is Does the computer model adequately represent reality? Viewing the computer models as a potentially biased representation of reality, Bayarri et al. [M. Bayarri, J. Berger, R. Paulo, J. Sacks, J. Cafeo, J. Cavendish, C. Lin, J. Tu, A framework for validation of computer models, Technometrics 49 (2) (2007) 138–154] develop the simulator assessment and validation engine ( SAVE ) method as a general framework for answering this question. In this paper, we apply the SAVE method to the challenge problem which involves a thermal computer model designed for certain devices. We develop a statement of confidence that the devices mode…

Rejoinder: Bayesian Checking of the Second Levels of Hierarchical Models

Rejoinder: Bayesian Checking of the Second Levels of Hierarchical Models [arXiv:0802.0743]

A Bayesian Sequential Look at u-Control Charts

We extend the usual implementation of u-control charts (uCCs) in two ways. First, we overcome the restrictive (and often inadequate) assumptions of the Poisson model; next, we eliminate the need for the questionable base period by using a sequential procedure. We use empirical Bayes(EB) and Bayes methods and compare them with the traditional frequentist implementation. EB methods are somewhat easy to implement, and they deal nicely with extra-Poisson variability (and, at the same time, informally check the adequacy of the Poisson assumption). However, they still need the base period. The sequential, full Bayes approach, on the other hand, also avoids this drawback of traditional u-charts. T…

Extending conventional priors for testing general hypotheses in linear models

We consider that observations come from a general normal linear model and that it is desirable to test a simplifying null hypothesis about the parameters. We approach this problem from an objective Bayesian, model-selection perspective. Crucial ingredients for this approach are 'proper objective priors' to be used for deriving the Bayes factors. Jeffreys-Zellner-Siow priors have good properties for testing null hypotheses defined by specific values of the parameters in full-rank linear models. We extend these priors to deal with general hypotheses in general linear models, not necessarily of full rank. The resulting priors, which we call 'conventional priors', are expressed as a generalizat…

Degroot, Morris H.

Una oportunidad para Bayes

PROBABILISTIC QUANTIFICATION OF HAZARDS: A METHODOLOGY USING SMALL ENSEMBLES OF PHYSICS-BASED SIMULATIONS AND STATISTICAL SURROGATES

This paper presents a novel approach to assessing the hazard threat to a locale due to a large volcanic avalanche. The methodology combines: (i) mathematical modeling of volcanic mass flows; (ii) field data of avalanche frequency, volume, and runout; (iii) large-scale numerical simulations of flow events; (iv) use of statistical methods to minimize computational costs, and to capture unlikely events; (v) calculation of the probability of a catastrophic flow event over the next T years at a location of interest; and (vi) innovative computational methodology to implement these methods. This unified presentation collects elements that have been separately developed, and incorporates new contri…

Bayesian Hierarchical Models for Random Routes in Finite Populations

In many practical situations involving sampling from finite populations, it is not possible (or it is prohibitely expensive) to access, or to even produce, a listing of all of the units in the population. In these situations, inferences can not be based on random samples from the population. Random routes are widely used procedures to collect data in absence of well defined sampling frames, and they usually have either been improperly analyzed as random samples, or entirely ignored as useless. We present here a Bayesian analysis of random routes that incorporates the information provided but carefully takes into account the non- randomness in the selection of the units.

Generalization of Jeffreys Divergence-Based Priors for Bayesian Hypothesis Testing

Summary We introduce objective proper prior distributions for hypothesis testing and model selection based on measures of divergence between the competing models; we call them divergence-based (DB) priors. DB priors have simple forms and desirable properties like information (finite sample) consistency and are often similar to other existing proposals like intrinsic priors. Moreover, in normal linear model scenarios, they reproduce the Jeffreys–Zellner–Siow priors exactly. Most importantly, in challenging scenarios such as irregular models and mixture models, DB priors are well defined and very reasonable, whereas alternative proposals are not. We derive approximations to the DB priors as w…

Using Statistical and Computer Models to Quantify Volcanic Hazards

Risk assessment of rare natural hazards, such as large volcanic block and ash or pyroclastic flows, is addressed. Assessment is approached through a combination of computer modeling, statistical modeling, and extreme-event probability computation. A computer model of the natural hazard is used to provide the needed extrapolation to unseen parts of the hazard space. Statistical modeling of the available data is needed to determine the initializing distribution for exercising the computer model. In dealing with rare events, direct simulations involving the computer model are prohibitively expensive. The solution instead requires a combination of adaptive design of computer model approximation…

Bayesian analysis and design for comparison of effect-sizes

Comparison of effect-sizes, or more generally, of non-centrality parameters of non-central t distributions, is a common problem, especially in meta-analysis. The usual simplifying assumptions of either identical or non-related effect-sizes are often too restrictive to be appropriate. In this paper, the effect-sizes are modeled as random effects with t distributions. Bayesian hierarchical models are used both to design and analyze experiments. The main goal is to compare effect-sizes. Sample sizes are chosen so as to make accurate inferences about the difference of effect-sizes and also to convincingly solve the testing of equality of effect-sizes if such is the goal.

PValues for Composite Null Models

Abstract The problem of investigating compatibility of an assumed model with the data is investigated in the situation when the assumed model has unknown parameters. The most frequently used measures of compatibility are p values, based on statistics T for which large values are deemed to indicate incompatibility of the data and the model. When the null model has unknown parameters, p values are not uniquely defined. The proposals for computing a p value in such a situation include the plug-in and similar p values on the frequentist side, and the predictive and posterior predictive p values on the Bayesian side. We propose two alternatives, the conditional predictive p value and the partial…

A Bayesian comparison of cluster, strata, and random samples

When sampling from finite populations, simple random sampling (SRS) is rarely used in practice, due to either high cost or information to be gained from more efficient designs. Bayesian hierarchical models are a natural framework to model the non-randomness in the sample. This paper concentrates on the effects that the design has on inference about characteristics of the finite population, and makes a critical comparison among some common designs.