AUTHORS

DETAILS

Distribution, posting, or copying of this PDF is strictly prohibited without written permission of the National Academies Press.  
(

Request Permission)

 Unless otherwise indicated, all materials in this PDF are copyrighted by the National Academy of Sciences.

Copyright © National Academy of Sciences. All rights reserved.

THE NATIONAL ACADEMIES PRESS

Visit the National Academies Press at 

NAP.edu

 and login or register to get:

–  10% off the price of print titles

–  Special offers and discounts

BUY THIS BOOK

FIND RELATED TITLES

This PDF is available at

 

SHARE

http://www.nap.edu/21915

132 pages | 7 x 10 | PAPERBACK
ISBN 978-0-309-39202-0 | DOI: 10.17226/21915

Statistical Challenges in Assessing and Fostering the 
Reproducibility of Scientific Results:  Summary of a Workshop 

Michelle Schwalbe, Rapporteur; Committee on Applied and 
Theoretical Statistics; Board on Mathematical Sciences and Their 
Applications; Division on Engineering and Physical Sciences; 
National Academies of Sciences, Engineering, and Medicine 

Copyright © National Academy of Sciences. All rights reserved.

Statistical Challenges in Assessing and Fostering the Reproducibility of Scientific Results:  Summary of a Workshop

Michelle Schwalbe, Rapporteur

Committee on Applied and Theoretical Statistics 

Board on Mathematical Sciences and Their Applications

Division on Engineering and Physical Sciences

Statistical Challenges in Assessing 

and Fostering the Reproducibility 

of Scientifi c Results

Summary of a Workshop

Copyright © National Academy of Sciences. All rights reserved.

Statistical Challenges in Assessing and Fostering the Reproducibility of Scientific Results:  Summary of a Workshop

THE NATIONAL ACADEMIES PRESS  500 Fifth Street, NW  Washington, DC 20001

This workshop was supported by Grant No. DMS-1351163 between the National Academies of Sci-
ences and the National Science Foundation. Any opinions, findings, or conclusions expressed in this 
publication do not necessarily reflect the views of any organization or agency that provided support 
for the project.

International Standard Book Number-13:  978-0-309-39202-0
International Standard Book Number-10:  0-309-39202-0
Digital Object Identifier:  10.17226/21915

Cover: The photo “Peas in a Pod” by Andrew_Writer is used with Creative Commons Attribution 2.0 
Generic  license  (https://www.flickr.com/photos/dragontomato/3698617520/in/gallery-106928713@
N04-72157638971870524/).

This report is available in limited quantities from: 

Board on Mathematical Sciences and Their Applications
National Academies of Sciences, Engineering, and Medicine 
500 Fifth Street NW
Washington, DC 20001
bmsa@nas.edu
http://www.nas.edu/bmsa

Additional  copies  of  this  workshop  summary  are  available  for  sale  from  the  National  Academies 
Press, 500 Fifth Street, NW, Keck 360, Washington, DC 20001; (800) 624-6242 or (202) 334-3313; 
http://www.nap.edu/.

Copyright 2016 by the National Academy of Sciences. All rights reserved.

Printed in the United States of America

Suggested  citation:  National  Academies  of  Sciences,  Engineering,  and  Medicine.  2016.  Statistical 
Challenges in Assessing and Fostering the Reproducibility of Scientific Results: Summary of a Workshop. 
Washington, DC: The National Academies Press. doi: 10.17226/21915.

Copyright © National Academy of Sciences. All rights reserved.

Statistical Challenges in Assessing and Fostering the Reproducibility of Scientific Results:  Summary of a Workshop

The National Academy of Sciences was established in 1863 by an Act of Congress, signed 
by President Lincoln, as a private, nongovernmental institution to advise the nation on 
issues related to science and  technology. Members are elected by their peers for outstand-
ing contributions to research. Dr. Ralph J. Cicerone is president.

The  National  Academy  of  Engineering  was  established  in  1964  under  the  charter  of 
the National Academy of Sciences to bring the practices of engineering to advising the 
 nation. Members are elected by their peers for extraordinary contributions to engineer-
ing. Dr. C. D. Mote, Jr., is president.

The National Academy of Medicine (formerly the Institute of Medicine) was estab lished 
in 1970 under the charter of the National Academy of  Sciences to advise the nation on 
medical and health issues. Members are elected by their peers for distinguished contribu-
tions to medicine and health. Dr. Victor J. Dzau is president.

The three Academies work together as the National Academies of Sciences,  Engineering, 
and Medicine
 to provide independent, objective analysis and  advice to the nation and 
conduct other activities to solve complex problems and inform public policy decisions. 
The Academies also encourage education and research, recognize outstanding contribu-
tions to knowledge, and increase public understanding in  matters of science, engineering, 
and medicine. 

Learn  more  about  the  National  Academies  of  Sciences,  Engineering,  and  Medicine  at 
www.national-academies.org

Copyright © National Academy of Sciences. All rights reserved.

Statistical Challenges in Assessing and Fostering the Reproducibility of Scientific Results:  Summary of a Workshop

Copyright © National Academy of Sciences. All rights reserved.

Statistical Challenges in Assessing and Fostering the Reproducibility of Scientific Results:  Summary of a Workshop

v

PLANNING COMMITTEE ON  

STATISTICAL CHALLENGES IN ASSESSING AND FOSTERING 

THE REPRODUCIBILITY OF SCIENTIFIC RESULTS

CONSTANTINE GATSONIS, Brown University, Co-Chair
GIOVANNI PARMIGIANI, Dana-Farber Cancer Institute, Co-Chair
STEPHEN FIENBERG, Carnegie Mellon University
STEVEN N. GOODMAN, Stanford University School of Medicine 
JOHN H. HOLMES, University of Pennsylvania Perelman School of Medicine 
ALAN F. KARR, RTI International
JELENA KOVA EVI , Carnegie Mellon University
XIHONG LIN, Harvard University
ROGER PENG, Johns Hopkins Bloomberg School of Public Health
VICTORIA STODDEN, University of Illinois, Urbana-Champaign

Staff

MICHELLE K. SCHWALBE, Senior Program Officer
RODNEY N. HOWARD, Administrative Assistant
LINDA CASOLA, Senior Program Assistant and Staff Editor

Copyright © National Academy of Sciences. All rights reserved.

Statistical Challenges in Assessing and Fostering the Reproducibility of Scientific Results:  Summary of a Workshop

vi

COMMITTEE ON APPLIED AND THEORETICAL STATISTICS

CONSTANTINE GATSONIS, Brown University, Chair
DEEPAK AGARWAL, LinkedIn
KATHERINE BENNETT ENSOR, Rice University 
MONTSERRAT (MONTSE) FUENTES, North Carolina State University
ALFRED O. HERO III, University of Michigan
DAVID M. HIGDON, Virginia Bioinformatics Institute
ROBERT E. KASS, Carnegie Mellon University
JOHN LAFFERTY, University of Chicago
XIHONG LIN, Harvard University
JOSÉ M. F. MOURA, Carnegie Mellon University
SHARON-LISE T. NORMAND, Harvard University
GIOVANNI PARMIGIANI, Dana-Farber Cancer Institute
ADRIAN RAFTERY, University of Washington
LANCE WALLER, Emory University
EUGENE WONG, University of California, Berkeley

Staff

MICHELLE K. SCHWALBE, Director
RODNEY N. HOWARD, Administrative Assistant
LINDA CASOLA, Senior Program Assistant

Copyright © National Academy of Sciences. All rights reserved.

Statistical Challenges in Assessing and Fostering the Reproducibility of Scientific Results:  Summary of a Workshop

vii

BOARD ON MATHEMATICAL SCIENCES AND THEIR APPLICATIONS

DONALD SAARI, University of California, Irvine, Chair
DOUGLAS N. ARNOLD, University of Minnesota
JOHN B. BELL, E. O. Lawrence Berkeley National Laboratory
VICKI M. BIER, University of Wisconsin, Madison 
JOHN R. BIRGE, University of Chicago 
L. ANTHONY COX, JR., Cox Associates, Inc.
MARK L. GREEN, University of California, Los Angeles
BRYNA KRA, Northwestern University
JOSEPH A. LANGSAM, Morgan Stanley (retired)
ANDREW W. LO, Massachusetts Institute of Technology
DAVID MAIER, Portland State University
WILLIAM A. MASSEY, Princeton University
JUAN C. MEZA, University of California, Merced
CLAUDIA NEUHAUSER, University of Minnesota
FRED S. ROBERTS, Rutgers University
GUILLERMO R. SAPIRO, Duke University 
CARL P. SIMON, University of Michigan
KATEPALLI SREENIVASAN, New York University
ELIZABETH A. THOMPSON, University of Washington 

Staff

SCOTT T. WEIDMAN, Director
NEAL GLASSMAN, Senior Program Officer
MICHELLE K. SCHWALBE, Senior Program Officer
RODNEY N. HOWARD, Administrative Assistant
BETH DOLAN, Financial Associate

Copyright © National Academy of Sciences. All rights reserved.

Statistical Challenges in Assessing and Fostering the Reproducibility of Scientific Results:  Summary of a Workshop

Copyright © National Academy of Sciences. All rights reserved.

Statistical Challenges in Assessing and Fostering the Reproducibility of Scientific Results:  Summary of a Workshop

ix

Acknowledgment of Reviewers

This workshop summary has been reviewed in draft form by individuals chosen 

for their diverse perspectives and technical expertise, in accordance with procedures 
approved  by  the  Report  Review  Committee.  The  purpose  of  this  independent 
 review is to provide candid and critical comments that will assist the institution in 
making its published workshop summary as sound as possible and to ensure that 
the workshop summary meets institutional standards for objectivity, evidence, and 
responsiveness to the project’s charge. The review comments and draft manuscript 
remain confidential to protect the integrity of the study process. We wish to thank 
the following individuals for their review of this workshop summary:

C. Glenn Begley, TetraLogic Pharmaceuticals, 
Joel Greenhouse, Carnegie Mellon University, 
Joelle Lomax, Science Exchange, and
Randy Schekman, University of California, Berkeley.

Although  the  reviewers  listed  above  have  provided  many  constructive  com-

ments  and  suggestions,  they  were  not  asked  to  endorse  the  views  presented  at 
the workshop, nor did they see the final draft of the workshop summary before 
its release. The review of this workshop summary was overseen by Lawrence D. 
Brown, University of Pennsylvania, who was responsible for making certain that 
an independent examination of this workshop summary was carried out in accor-
dance with institutional procedures and that all review comments were carefully 
considered. Responsibility for the final content of this summary rests entirely with 
the author and the institution.