OMB Federal Register Notice - Vol. 67, No. 36

[Federal Register: February 22, 2002 (Volume 67, Number 36)]
[Notices]
[Page 8451-8460]
From the Federal Register Online via GPO Access [wais.access.gpo.gov]
[DOCID:fr22fe02-117]

[[Page 8451]]

-----------------------------------------------------------------------

Part IX

Office of Management and Budget

-----------------------------------------------------------------------

Guidelines for Ensuring and Maximizing the Quality, Objectivity,
Utility, and Integrity of Information Disseminated by Federal Agencies;
Notice; Republication

[[Page 8452]]

-----------------------------------------------------------------------

OFFICE OF MANAGEMENT AND BUDGET


Guidelines for Ensuring and Maximizing the Quality, Objectivity,
Utility, and Integrity of Information Disseminated by Federal Agencies;
Republication

    Editorial Note: Due to numerous errors, this document is being
reprinted in its entirety. It was originally printed in the Federal
Register on Thursday, January 3, 2002 at 67 FR 369-378 and was
corrected on Tuesday, February 5, 2002 at 67 FR 5365.

AGENCY: Office of Management and Budget, Executive Office of the
President.

ACTION: Final guidelines.

-----------------------------------------------------------------------

SUMMARY: These final guidelines implement section 515 of the Treasury
and General Government Appropriations Act for Fiscal Year 2001 (Public
Law 106-554; H.R. 5658). Section 515 directs the Office of Management
and Budget (OMB) to issue government-wide guidelines that ``provide
policy and procedural guidance to Federal agencies for ensuring and
maximizing the quality, objectivity, utility, and integrity of
information (including statistical information) disseminated by Federal
agencies.'' By October 1, 2002, agencies must issue their own
implementing guidelines that include ``administrative mechanisms
allowing affected persons to seek and obtain correction of information
maintained and disseminated by the agency'' that does not comply with
the OMB guidelines. These final guidelines also reflect the changes OMB
made to the guidelines issued September 28, 2001, as a result of
receiving additional comment on the ``capable of being substantially
reproduced'' standard (paragraphs V.3.B, V.9, and V.10), which OMB
previously issued on September 28, 2001, on an interim final basis.

DATES: Effective Date: January 3, 2002.

FOR FURTHER INFORMATION CONTACT: Brooke J. Dickson, Office of
Information and Regulatory Affairs, Office of Management and Budget,
Washington, DC 20503. Telephone (202) 395-3785 or by e-mail to
informationquality@omb.eop.gov.

SUPPLEMENTARY INFORMATION: In section 515(a) of the Treasury and
General Government Appropriations Act for Fiscal Year 2001 (Public Law
106-554; H.R. 5658), Congress directed the Office of Management and
Budget (OMB) to issue, by September 30, 2001, government-wide
guidelines that ``provide policy and procedural guidance to Federal
agencies for ensuring and maximizing the quality, objectivity, utility,
and integrity of information (including statistical information)
disseminated by Federal agencies * * *'' Section 515(b) goes on to
state that the OMB guidelines shall:
    ``(1) apply to the sharing by Federal agencies of, and access to,
information disseminated by Federal agencies; and
    ``(2) require that each Federal agency to which the guidelines
apply--
    ``(A) issue guidelines ensuring and maximizing the quality,
objectivity, utility, and integrity of information (including
statistical information) disseminated by the agency, by not later than
1 year after the date of issuance of the guidelines under subsection
(a);
    ``(B) establish administrative mechanisms allowing affected persons
to seek and obtain correction of information maintained and
disseminated by the agency that does not comply with the guidelines
issued under subsection (a); and
    ``(C) report periodically to the Director--
    ``(i) the number and nature of complaints received by the agency
regarding the accuracy of information disseminated by the agency and;
    ``(ii) how such complaints were handled by the agency.''
    Proposed guidelines were published in the Federal Register on June
28, 2001 (66 FR 34489). Final guidelines were published in the Federal
Register on September 28, 2001 (66 FR 49718). The Supplementary
Information to the final guidelines published in September 2001
provides background, the underlying principles OMB followed in issuing
the final guidelines, and statements of intent concerning detailed
provisions in the final guidelines.
    In the final guidelilnes published in September 2001, OMB also
requested additional comment on the ``capable of being substantially
reproduced'' standard and the related definition of ``influential
scientific or statistical information'' (paragraphs V.3.B, V.9, and
V.10), which were issued on an interim final basis. The final
guidelines published today discuss the public comments OMB received,
the OMB response, and amendments to the final guidelines published in
September 2001.
    In developing agency-specific guidelines, agencies should refer
both to the Supplementary Information to the final guidelines published
in the Federal Register on September 28, 2001 (66 FR 49718), and also
to the Supplementary Information published today. We stress that the
three ``Underlying Principles'' that OMB followed in drafting the
guidelines that we published on September 28, 2001 (66 FR 49719), are
also applicable to the amended guidelines that we publish today.
    In accordance with section 515, OMB has designed the guidelines to
help agencies ensure and maximize the quality, utility, objectivity and
integrity of the information that they disseminate (meaning to share
with, or give access to, the public). It is crucial that information
Federal agencies disseminate meets these guidelines. In this respect,
the fact that the Internet enables agencies to communicate information
quickly and easily to a wide audience not only offers great benefits to
society, but also increases the potential harm that can result from the
dissemination of information that does not meet basic information
quality guidelines. Recognizing the wide variety of information Federal
agencies disseminate and the wide variety of dissemination practices
that agencies have, OMB developed the guidelines with several
principles in mind.
    First, OMB designed the guidelines to apply to a wide variety of
government information dissemination activities that may range in
importance and scope. OMB also designed the guidelines to be generic
enough to fit all media, be they printed, electronic, or in other form.
OMB sought to avoid the problems that would be inherent in developing
detailed, prescriptive, ``one-size-fits-all'' government-wide
guidelines that would artificially require different types of
dissemination activities to be treated in the same manner. Through this
flexibility, each agency will be able to incorporate the requirements
of these OMB guidelines into the agency's own information resource
management and administrative practices.
    Second, OMB designed the guidelines so that agencies will meet
basic information quality standards. Given the administrative
mechanisms required by section 515 as well as the standards set forth
in the Paperwork Reduction Act, it is clear that agencies should not
disseminate substantive information that does not meet a basic level of
quality. We recognize that some government information may need to meet
higher or more specific information quality standards than those that
would apply to other types of government information. The more
important the information, the higher the quality standards to which it
should be held, for example, in those situations involving
``influential scientific, financial, or statistical information'' (a
phrase defined in these guidelines). The guidelines recognize, however,
that

[[Page 8453]]

information quality comes at a cost. Accordingly, the agencies should
weigh the costs (for example, including costs attributable to agency
processing effort, respondent burden, maintenance of needed privacy,
and assurances of suitable confidentiality) and the benefits of higher
information quality in the development of information, and the level of
quality to which the information disseminated will be held.
    Third, OMB designed the guidelines so that agencies can apply them
in a common-sense and workable manner. It is important that these
guidelines do not impose unnecessary administrative burdens that would
inhibit agencies from continuing to take advantage of the Internet and
other technologies to disseminate information that can be of great
benefit and value to the public. In this regard, OMB encourages
agencies to incorporate the standards and procedures required by these
guidelines into their existing information resources management and
administrative practices rather than create new and potentially
duplicative or contradictory processes. The primary example of this is
that the guidelines recognize that, in accordance with OMB Circular A-
130, agencies already have in place well-established information
quality standards and administrative mechanisms that allow persons to
seek and obtain correction of information that is maintained and
disseminated by the agency. Under the OMB guidelines, agencies need
only ensure that their own guidelines are consistent with these OMB
guidelines, and then ensure that their administrative mechanisms
satisfy the standards and procedural requirements in the new agency
guidelines. Similarly, agencies may rely on their implementation of the
Federal Government's computer security laws (formerly, the Computer
Security Act, and now the computer security provisions of the Paperwork
Reduction Act) to establish appropriate security safeguards for
ensuring the ``integrity'' of the information that the agencies
disseminate.
    In addition, in response to concerns expressed by some of the
agencies, we want to emphasize that OMB recognizes that Federal
agencies provide a wide variety of data and information. Accordingly,
OMB understands that the guidelines discussed below cannot be
implemented in the same way by each agency. In some cases, for example,
the data disseminated by an agency are not collected by that agency;
rather, the information the agency must provide in a timely manner is
compiled from a variety of sources that are constantly updated and
revised and may be confidential. In such cases, while agencies'
implementation of the guidelines may differ, the essence of the
guidelines will apply. That is, these agencies must make their methods
transparent by providing documentation, ensure quality by reviewing the
underlying methods used in developing the data and consulting (as
appropriate) with experts and users, and keep users informed about
corrections and revisions.

Summary of OMB Guidelines

    These guidelines apply to Federal agencies subject to the Paperwork
Reduction Act (44 U.S.C. chapter 35). Agencies are directed to develop
information resources management procedures for reviewing and
substantiating (by documentation or other means selected by the agency)
the quality (including the objectivity, utility, and integrity) of
information before it is disseminated. In addition, agencies are to
establish administrative mechanisms allowing affected persons to seek
and obtain, where appropriate, correction of information disseminated
by the agency that does not comply with the OMB or agency guidelines.
Consistent with the underlying principles described above, these
guidelines stress the importance of having agencies apply these
standards and develop their administrative mechanisms so they can be
implemented in a common sense and workable manner. Moreover, agencies
must apply these standards flexibly, and in a manner appropriate to the
nature and timeliness of the information to be disseminated, and
incorporate them into existing agency information resources management
and administrative practices.
    Section 515 denotes four substantive terms regarding information
disseminated by Federal agencies: quality, utility, objectivity, and
integrity. It is not always clear how each substantive term relates--or
how the four terms in aggregate relate--to the widely divergent types
of information that agencies disseminate. The guidelines provide
definitions that attempt to establish a clear meaning so that both the
agency and the public can readily judge whether a particular type of
information to be disseminated does or does not meet these attributes.
    In the guidelines, OMB defines ``quality'' as the encompassing
term, of which ``utility,'' ``objectivity,'' and ``integrity'' are the
constituents. ``Utility'' refers to the usefulness of the information
to the intended users. ``Objectivity'' focuses on whether the
disseminated information is being presented in an accurate, clear,
complete, and unbiased manner, and as a matter of substance, is
accurate, reliable, and unbiased. ``Integrity'' refers to security--the
protection of information from unauthorized access or revision, to
ensure that the information is not compromised through corruption or
falsification. OMB modeled the definitions of ``information,''
``government information,'' ``information dissemination product,'' and
``dissemination'' on the longstanding definitions of those terms in OMB
Circular A-130, but tailored them to fit into the context of these
guidelines.
    In addition, Section 515 imposes two reporting requirements on the
agencies. The first report, to be promulgated no later than October 1,
2002, must provide the agency's information quality guidelines that
describe administrative mechanisms allowing affected persons to seek
and obtain, where appropriate, correction of disseminated information
that does not comply with the OMB and agency guidelines. The second
report is an annual fiscal year report to OMB (to be first submitted on
January 1, 2004) providing information (both quantitative and
qualitative, where appropriate) on the number, nature, and resolution
of complaints received by the agency regarding its perceived or
confirmed failure to comply with these OMB and agency guidelines.

Public Comments and OMB Response

    Applicability of Guidelines. Some comments raised concerns about
the applicability of these guidelines, particularly in the context of
scientific research conducted by Federally employed scientists or
Federal grantees who publish and communicate their research findings in
the same manner as their academic colleagues. OMB believes that
information generated and disseminated in these contexts is not covered
by these guidelines unless the agency represents the information as, or
uses the information in support of, an official position of the agency.
    As a general matter, these guidelines apply to ``information'' that
is ``disseminated'' by agencies subject to the Paperwork Reduction Act
(44 U.S.C. 3502(1)). See paragraphs II, V.5 and V.8. The definitions of
``information'' and ``dissemination'' establish the scope of the
applicability of these guidelines. ``Information'' means ``any
communication or representation of knowledge such as facts or data * *
*'' This definition of information in paragraph V.5 does ``not include
opinions, where the agency's presentation makes it clear that what is

[[Page 8454]]

being offered is someone's opinion rather than fact or the agency's
views.''
    ``Dissemination'' is defined to mean ``agency initiated or
sponsored distribution of information to the public.'' As used in
paragraph V.8, ``agency INITIATED * * * distribution of information to
the public'' refers to information that the agency disseminates, e.g.,
a risk assessment prepared by the agency to inform the agency's
formulation of possible regulatory or other action. In addition, if an
agency, as an institution, disseminates information prepared by an
outside party in a manner that reasonably suggests that the agency
agrees with the information, this appearance of having the information
represent agency views makes agency dissemination of the information
subject to these guidelines. By contrast, an agency does not
``initiate'' the dissemination of information when a Federally employed
scientist or Federal grantee or contractor publishes and communicates
his or her research findings in the same manner as his or her academic
colleagues, even if the Federal agency retains ownership or other
intellectual property rights because the Federal government paid for
the research. To avoid confusion regarding whether the agency agrees
with the information (and is therefore disseminating it through the
employee or grantee), the researcher should include an appropriate
disclaimer in the publication or speech to the effect that the ``views
are mine, and do not necessarily reflect the view'' of the agency.
    Similarly, as used in paragraph V.8., ``agency * * * SPONSORED
distribution of information to the public'' refers to situations where
an agency has directed a third-party to disseminate information, or
where the agency has the authority to review and approve the
information before release. Therefore, for example, if an agency
through a procurement contract or a grant provides for a person to
conduct research, and then the agency directs the person to disseminate
the results (or the agency reviews and approves the results before they
may be disseminated), then the agency has ``sponsored'' the
dissemination of this information. By contrast, if the agency simply
provides funding to support research, and it the researcher (not the
agency) who decides whether to disseminate the results and--if the
results are to be released--who determines the content and presentation
of the dissemination, then the agency has not ``sponsored'' the
dissemination even though it has funded the research and even if the
Federal agency retains ownership or other intellectual property rights
because the Federal government paid for the research. To avoid
confusion regarding whether the agency is sponsoring the dissemination,
the researcher should include an appropriate disclaimer in the
publication or speech to the effect that the ``views are mine, and do
not necessarily reflect the view'' of the agency. On the other hand,
subsequent agency dissemination of such information requires that the
information adhere to the agency's information quality guidelines. In
sum, these guidelines govern an agency's dissemination of information,
but generally do not govern a third-party's dissemination of
information (the exception being where the agency is essentially using
the third-party to disseminate information on the agency's behalf).
Agencies, particularly those that fund scientific research, are
encouraged to clarify the applicability of these guidelines to the
various types of information they and their employees and grantees
disseminate.
    Paragraph V.8 also states that the definition of ``dissemination''
does not include ``* * * distribution limited to correspondence with
individuals or persons, press releases, archival records, public
filings, subpoenas or adjudicative processes.'' The exemption from the
definition of ``dissemination'' for ``adjudicative processes'' is
intended to exclude, from the scope of these guidelines, the findings
and determinations that an agency makes in the course of adjudications
involving specific parties. There are well-established procedural
safeguards and rights to address the quality of adjudicatory decisions
and to provide persons with an opportunity to contest decisions. These
guidelines do not impose any additional requirements on agencies during
adjudicative proceedings and do not provide parties to such
adjudicative proceedings any additional rights of challenge or appeal.
    The Presumption Favoring Peer-Reviewed Information.As a general
matter, in the scientific and research context, we regard technical
information that has been subjected to formal, independent, external
peer review as presumptively objective. As the guidelines state in
paragraph V.3.b.i: ``If data and analytic results have been subjected
to formal, independent, external peer review, the information may
generally be presumed to be of acceptable objectivity.'' An example of
a formal, independent, external peer review is the review process used
by scientific journals.
    Most comments approved of the prominent role that peer review plays
in the OMB guidelines. Some comments contended that peer review was not
accepted as a universal standard that incorporates an established,
practiced, and sufficient level of objectivity. Other comments stated
that the guidelines would be better clarified by making peer review one
of several factors that an agency should consider in assessing the
objectivity (and quality in general) of original research. In addition,
several comments noted that peer review does not establish whether
analytic results are capable of being substantially reproduced. In
light of the comments, the final guidelines in new paragraph V.3.b.i
qualify the presumption in favor of peer-reviewed information as
follows: ``However, this presumption is rebuttable based on a
persuasive showing by the petitioner in a particular instance.''
    We believe that transparency is important for peer review, and
these guidelines set minimum standards for the transparency of agency-
sponsored peer review. As we state in new paragraph V.3.b.i: ``If data
and analytic results have been subjected to formal, independent,
external peer review, the information may generally be presumed to be
of acceptable objectivity. However, this presumption is rebuttable
based on a persuasive showing by the petitioner in a particular
instance. If agency-sponsored peer review is employed to help satisfy
the objectivity standard, the review process employed shall meet the
general criteria for competent and credible peer review recommended by
OMB-OIRA to the President's Management Council (9/20/01) (http://
www.whitehouse.gov/omb/inforeg/oira_review-process.html), namely, `that
(a) peer reviewers be selected primarily on the basis of necessary
technical expertise, (b) peer reviewers be expected to disclose to
agencies prior technical/policy positions they may have taken on the
issues at hand, (c) peer reviewers be expected to disclose to agencies
their sources of personal and institutional funding (private or public
sector), and (d) peer reviews be conducted in an open and rigorous
manner.' ''
    The importance of these general criteria for competent and credible
peer review has been supported by a number of expert bodies. For
example, ``the work of fully competent peer-review panels can be
undermined by allegations of conflict of interest and bias. Therefore,
the best interests of the Board are served by effective policies and
procedures regarding potential conflicts of interest, impartiality, and
panel balance.'' (EPA's Science Advisory

[[Page 8455]]

Board Panels: Improved Policies and Procedures Needed to Ensure
Independence and Balance, GAO-01-536, General Accounting Office,
Washington, DC, June 2001, page 19.) As another example, ``risk
analyses should be peer-reviewed and accessible--both physically and
intellectually--so that decision-makers at all levels will be able to
respond critically to risk characterizations. The intensity of the peer
reviews should be commensurate with the significance of the risk or its
management implications.'' (Setting Priorities, Getting Results: A New
Direction for EPA, Summary Report, National Academy of Public
Administration, Washington, DC, April 1995, page 23.)
    These criteria for peer reviewers are generally consistent with the
practices now followed by the National Research Council of the National
Academy of Sciences. In considering these criteria for peer reviewers,
we note that there are many types of peer reviews and that agency
guidelines concerning the use of peer review should tailor the rigor of
peer review to the importance of the information involved. More
generally, agencies should define their peer-review standards in
appropriate ways, given the nature and importance of the information
they disseminate.
    Is Journal Peer Review Always Sufficient? Some comments argued that
journal peer review should be adequate to demonstrate quality, even for
influential information that can be expected to have major effects on
public policy. OMB believes that this position overstates the
effectiveness of journal peer review as a quality-control mechanism.
    Although journal peer review is clearly valuable, there are cases
where flawed science has been published in respected journals. For
example, the NIH Office of Research Integrity recently reported the
following case regarding environmental health research:

    ``Based on the report of an investigation conducted by [XX]
University, dated July 16, 1999, and additional analysis conducted
by ORI in its oversight review, the US Public Health Service found
that Dr. [X] engaged in scientific misconduct. Dr. [X] committed
scientific misconduct by intentionally falsifying the research
results published in the journal SCIENCE and by providing falsified
and fabricated materials to investigating officials at [XX]
University in response to a request for original data to support the
research results and conclusions report in the SCIENCE paper. In
addition, PHS finds that there is no original data or other
corroborating evidence to support the research results and
conclusions reported in the SCIENCE paper as a whole.'' (66 FR
52137, October 12, 2001).

    Although such cases of falsification are presumably rare, there is
a significant scholarly literature documenting quality problems with
articles published in peer-reviewed research. ``In a [peer-reviewed]
meta-analysis that surprised many--and some doubt--researchers found
little evidence that peer review actually improves the quality of
research papers.'' (See, e.g., Science, Vol. 293, page 2187 (September
21, 2001.)) In part for this reason, many agencies have already adopted
peer review and science advisory practices that go beyond journal peer
review. See, e.g., Sheila Jasanoff, The Fifth Branch: Science Advisers
as Policy Makers, Cambridge, MA, Harvard University Press, 1990; Mark
R. Powell, Science at EPA: Information in the Regulatory Process.
Resources for the Future, Washington, DC., 1999, pages 138-139; 151-
153; Implementation of the Environmental Protection Agency's Peer
Review Program: An SAB Evaluation of Three Reviews, EPA-SAB-RSAC-01-
009, A Review of the Research Strategies Advisory Committee (RSAC) of
the EPA Science Advisory Board (SAB), Washington, DC., September 26,
2001. For information likely to have an important public policy or
private sector impact, OMB believes that additional quality checks
beyond peer review are appropriate.
    Definition of ``Influential''. OMB guidelines apply stricter
quality standards to the dissemination of information that is
considered ``influential.'' Comments noted that the breadth of the
definition of ``influential'' in interim final paragraph V.9 requires
much speculation on the part of agencies.
    We believe that this criticism has merit and have therefore
narrowed the definition. In this narrower definition, ``influential'',
when used in the phrase ``influential scientific, financial, or
statistical information'', is amended to mean that ``the agency can
reasonably determine that dissemination of the information will have or
does have a clear and substantial impact on important public policies
or important private sector decisions.'' The intent of the new phrase
``clear and substantial'' is to reduce the need for speculation on the
part of agencies. We added the present tense--``or does have''--to this
narrower definition because on occasion, an information dissemination
may occur simultaneously with a particular policy change. In response
to a public comment, we added an explicit reference to ``financial''
information as consistent with our original intent.
    Given the differences in the many Federal agencies covered by these
guidelines, and the differences in the nature of the information they
disseminate, we also believe it will be helpful if agencies elaborate
on this definition of ``influential'' in the context of their missions
and duties, with due consideration of the nature of the information
they disseminate. As we state in amended paragraph V.9, ``Each agency
is authorized to define `influential' in ways appropriate for it given
the nature and multiplicity of issues for which the agency is
responsible.''
    Reproducibility. As we state in new paragraph V.3.b.ii: ``If an
agency is responsible for disseminating influential scientific,
financial, or statistical information, agency guidelines shall include
a high degree of transparency about data and methods to facilitate the
reproducibility of such information by qualified third parties.'' OMB
believes that a reproducibility standard is practical and appropriate
for information that is considered ``influential'', as defined in
paragraph V.9--that ``will have or does have a clear and substantial
impact on important public policies or important private sector
decisions.'' The reproducibility standard applicable to influential
scientific, financial, or statistical information is intended to ensure
that information disseminated by agencies is sufficiently transparent
in terms of data and methods of analysis that it would be feasible for
a replication to be conducted. The fact that the use of original and
supporting data and analytic results have been deemed ``defensible'' by
peer-review procedures does not necessarily imply that the results are
transparent and replicable.
    Reproducibility of Original and Supporting Data. Several of the
comments objected to the exclusion of original and supporting data from
the reproducibility requirements. Comments instead suggested that OMB
should apply the reproducibility standard to original data, and that
OMB should provide flexibility to the agencies in determining what
constitutes ``original and supporting'' data. OMB agrees and asks that
agencies consider, in developing their own guidelines, which categories
of original and supporting data should be subject to the
reproducibility standard and which should not. To help in resolving
this issue, we also ask agencies to consult directly with relevant
scientific and technical communities on the feasibility of having the
selected categories of original and supporting data subject to the
reproducibility standard. Agencies are encouraged to address ethical,
feasibility, and confidentiality issues

[[Page 8456]]

with care. As we state in new paragraph V.3.b.ii.A, ``Agencies may
identify, in consultation with the relevant scientific and technical
communities, those particular types of data that can practicably be
subjected to a reproducibility requirement, given ethical, feasibility,
or confidentiality constraints.'' Further, as we state in our expanded
definition of ``reproducibility'' in paragraph V.10, ``If agencies
apply the reproducibility test to specific types of original or
supporting data, the associated guidelines shall provide relevant
definitions of reproducibility (e.g., standards for replication of
laboratory data).'' OMB urges caution in the treatment of original and
supporting data because it may often be impractical or even
impermissible or unethical to apply the reproducibility standard to
such data. For example, it may not be ethical to repeat a ``negative''
(ineffective) clinical (therapeutic) experiment and it may not be
feasible to replicate the radiation exposures studied after the
Chernobyl accident. When agencies submit their draft agency guidelines
for OMB review, agencies should include a description of the extent to
which the reproducibility standard is applicable and reflect
consultations with relevant scientific and technical communities that
were used in developing guidelines related to applicability of the
reproducibility standard to original and supporting data.
    It is also important to emphasize that the reproducibility standard
does not apply to all original and supporting data disseminated by
agencies. As we state in new paragraph V.3.b.ii.A, ``With regard to
original and supporting data related [to influential scientific,
financial, or statistical information], agency guidelines shall not
require that all disseminated data be subjected to a reproducibility
requirement.'' In addition, we encourage agencies to address how
greater transparency can be achieved regarding original and supporting
data. As we also state in new paragraph V.3.b.ii.A, ``It is understood
that reproducibility of data is an indication of transparency about
research design and methods and thus a replication exercise (i.e., a
new experiment, test, or sample) shall not be required prior to each
dissemination.'' Agency guidelines need to achieve a high degree of
transparency about data even when reproducibility is not required.
    Reproducibility of Analytic Results. Many public comments were
critical of the reproducibility standard and expressed concern that
agencies would be required to reproduce each analytical result before
it is disseminated. While several comments commended OMB for
establishing an appropriate balance in the ``capable of being
substantially reproduced'' standard, others considered this standard to
be inherently subjective. There were also comments that suggested the
standard would cause more burden for agencies.
    It is not OMB's intent that each agency must reproduce each
analytic result before it is disseminated. The purpose of the
reproducibility standard is to cultivate a consistent agency commitment
to transparency about how analytic results are generated: the specific
data used, the various assumptions employed, the specific analytic
methods applied, and the statistical procedures employed. If sufficient
transparency is achieved on each of these matters, then an analytic
result should meet the ``capable of being substantially reproduced''
standard.
    While there is much variation in types of analytic results, OMB
believes that reproducibility is a practical standard to apply to most
types of analytic results. As we state in new paragraph V.3.b.ii.B,
``With regard to analytic results related [to influential scientific,
financial, or statistical information], agency guidelines shall
generally require sufficient transparency about data and methods that
an independent reanalysis could be undertaken by a qualified member of
the public. These transparency standards apply to agency analysis of
data from a single study as well as to analyses that combine
information from multiple studies.'' We elaborate upon this principle
in our expanded definition of ``reproducibility'' in paragraph V.10:
``With respect to analytic results, `capable of being substantially
reproduced' means that independent analysis of the original or
supporting data using identical methods would generate similar analytic
results, subject to an acceptable degree of imprecision or error.''
    Even in a situation where the original and supporting data are
protected by confidentiality concerns, or the analytic computer models
or other research methods may be kept confidential to protect
intellectual property, it may still be feasible to have the analytic
results subject to the reproducibility standard. For example, a
qualified party, operating under the same confidentiality protections
as the original analysts, may be asked to use the same data, computer
model or statistical methods to replicate the analytic results reported
in the original study. See, e.g., ``Reanalysis of the Harvard Six
Cities Study and the American Cancer Society Study of Particulate Air
Pollution and Mortality,'' A Special Report of the Health Effects
Institute's Particle Epidemiology Reanalysis Project, Cambridge, MA,
2000.
    The primary benefit of public transparency is not necessarily that
errors in analytic results will be detected, although error correction
is clearly valuable. The more important benefit of transparency is that
the public will be able to assess how much an agency's analytic result
hinges on the specific analytic choices made by the agency.
Concreteness about analytic choices allows, for example, the
implications of alternative technical choices to be readily assessed.
This type of sensitivity analysis is widely regarded as an essential
feature of high-quality analysis, yet sensitivity analysis cannot be
undertaken by outside parties unless a high degree of transparency is
achieved. The OMB guidelines do not compel such sensitivity analysis as
a necessary dimension of quality, but the transparency achieved by
reproducibility will allow the public to undertake sensitivity studies
of interest.
    We acknowledge that confidentiality concerns will sometimes
preclude public access as an approach to reproducibility. In response
to public comment, we have clarified that such concerns do include
interests in ``intellectual property.'' To ensure that the OMB
guidelines have sufficient flexibility with regard to analytic
transparency, OMB has, in new paragraph V.3.b.ii.B.i, provided agencies
an alternative approach for classes or types of analytic results that
cannot practically be subject to the reproducibility standard. ``[In
those situations involving influential scientific, financial, or
statistical information * * * ] making the data and methods publicly
available will assist in determining whether analytic results are
reproducible. However, the objectivity standard does not override other
compelling interests such as privacy, trade secrets, intellectual
property, and other confidentiality protections. '' Specifically, in
cases where reproducibility will not occur due to other compelling
interests, we expect agencies (1) to perform robustness checks
appropriate to the importance of the information involved, e.g.,
determining whether a specific statistic is sensitive to the choice of
analytic method, and, accompanying the information disseminated, to
document their efforts to assure the needed robustness in information
quality, and (2) address in their guidelines the

[[Page 8457]]

degree to which they anticipate the opportunity for reproducibility to
be limited by the confidentiality of underlying data. As we state in
new paragraph V.3.b.ii.B.ii, ``In situations where public access to
data and methods will not occur due to other compelling interests,
agencies shall apply especially rigorous robustness checks to analytic
results and document what checks were undertaken. Agency guidelines
shall, however, in all cases, require a disclosure of the specific data
sources that have been used and the specific quantitative methods and
assumptions that have been employed.''
    Given the differences in the many Federal agencies covered by these
guidelines, and the differences in robustness checks and the level of
detail for documentation thereof that might be appropriate for
different agencies, we also believe it will be helpful if agencies
elaborate on these matters in the context of their missions and duties,
with due consideration of the nature of the information they
disseminate. As we state in new paragraph V.3.b.ii.B.ii, ``Each agency
is authorized to define the type of robustness checks, and the level of
detail for documentation thereof, in ways appropriate for it given the
nature and multiplicity of issues for which the agency is
responsible.''
    We leave the determination of the appropriate degree of rigor to
the discretion of agencies and the relevant scientific and technical
communities that work with the agencies. We do, however, establish a
general standard for the appropriate degree of rigor in our expanded
definition of ``reproducibility'' in paragraph V.10: ``
`Reproducibility' means that the information is capable of being
substantially reproduced, subject to an acceptable degree of
imprecision. For information judged to have more (less) important
impacts, the degree of imprecision that is tolerated is reduced
(increased).'' OMB will review each agency's treatment of this issue
when reviewing the agency guidelines as a whole.
    Comments also expressed concerns regarding interim final paragraph
V.3.B.iii, ``making the data and models publicly available will assist
in determining whether analytic results are capable of being
substantially reproduced,'' and whether it could be interpreted to
constitute public dissemination of these materials, rendering moot the
reproducibility test. (For the equivalent provision, see new paragraph
V.3.b.ii.B.i.) The OMB guidelines do not require agencies to reproduce
each disseminated analytic result by independent reanalysis. Thus,
public dissemination of data and models per se does not mean that the
analytic result has been reproduced. It means only that the result
should be CAPABLE of being reproduced. The transparency associated with
this capability of reproduction is what the OMB guidelines are designed
to achieve.
    We also want to build on a general observation that we made in our
final guidelines published in September 2001. In those guidelines we
stated: ``... in those situations involving influential scientific[,
financial,] or statistical information, the substantial reproducibility
standard is added as a quality standard above and beyond some peer
review quality standards'' (66 FR 49722 (September 28, 2001)). A
hypothetical example may serve to illustrate this point. Assume that
two Federal agencies initiated or sponsored the dissemination of five
scientific studies after October 1, 2002 (see paragraph III.4) that
were, before dissemination, subjected to formal, independent, external
peer review, i.e., that met the presumptive standard for
``objectivity'' under paragraph V.3.b.i. Further assume, at the time of
dissemination, that neither agency reasonably expected that the
dissemination of any of these studies would have ``a clear and
substantial impact'' on important public policies, i.e., that these
studies were not considered ``influential'' under paragraph V.9, and
thus not subject to the reproducibility standards in paragraphs
V.3.b.ii.A or B. Then assume, two years later, in 2005, that one of the
agencies decides to issue an important and far-reaching regulation
based clearly and substantially on the agency's evaluation of the
analytic results set forth in these five studies and that such agency
reliance on these five studies as published in the agency's notice of
proposed rulemaking would constitute dissemination of these five
studies. These guidelines would require the rulemaking agency, prior to
publishing the notice of proposed rulemaking, to evaluate these five
studies to determine if the analytic results stated therein would meet
the ``capable of being substantially reproduced'' standards in
paragraph V.3.b.ii.B and, if necessary, related standards governing
original and supporting data in paragraph V.3.b.ii.A. If the agency
were to decide that any of the five studies would not meet the
reproducibility standard, the agency may still rely on them but only if
they satisfy the transparency standard and--as applicable--the
disclosure of robustness checks required by these guidelines.
Otherwise, the agency should not disseminate any of the studies that
did not meet the applicable standards in the guidelines at the time it
publishes the notice of proposed rulemaking.
    Some comments suggested that OMB consider replacing the
reproducibility standard with a standard concerning ``confirmation'' of
results for influential scientific and statistical information.
Although we encourage agencies to consider ``confirmation'' as a
relevant standard--at least in some cases--for assessing the
objectivity of original and supporting data, we believe that
``confirmation'' is too stringent a standard to apply to analytic
results. Often the regulatory impact analysis prepared by an agency for
a major rule, for example, will be the only formal analysis of an
important subject. It would be unlikely that the results of the
regulatory impact analysis had already been confirmed by other
analyses. The ``capable of being substantially reproduced'' standard is
less stringent than a ``confirmation'' standard because it simply
requires that an agency's analysis be sufficiently transparent that
another qualified party could replicate it through reanalysis.
    Health, Safety, and Environmental Information. We note, in the
scientific context, that in 1996 the Congress, for health decisions
under the Safe Drinking Water Act, adopted a basic standard of quality
for the use of science in agency decisionmaking. Under 42 U.S.C. 300g-
1(b)(3)(A), an agency is directed, ``to the degree that an Agency
action is based on science,'' to use ``(i) the best available, peer-
reviewed science and supporting studies conducted in accordance with
sound and objective scientific practices; and (ii) data collected by
accepted methods or best available methods (if the reliability of the
method and the nature of the decision justifies use of the data).''
    We further note that in the 1996 amendments to the Safe Drinking
Water Act, Congress adopted a basic quality standard for the
dissemination of public information about risks of adverse health
effects. Under 42 U.S.C. 300g-1(b)(3)(B), the agency is directed, ``to
ensure that the presentation of information [risk] effects is
comprehensive, informative, and understandable.'' The agency is further
directed, ``in a document made available to the public in support of a
regulation [to] specify, to the extent practicable--(i) each population
addressed by any estimate [of applicable risk effects]; (ii) the
expected risk or central estimate of

[[Page 8458]]

risk for the specific populations [affected]; (iii) each appropriate
upper-bound or lower-bound estimate of risk; (iv) each significant
uncertainty identified in the process of the assessment of [risk]
effects and the studies that would assist in resolving the uncertainty;
and (v) peer-reviewed studies known to the [agency] that support, are
directly relevant to, or fail to support any estimate of [risk] effects
and the methodology used to reconcile inconsistencies in the scientific
data.''
    As suggested in several comments, we have included these
congressional standards directly in new paragraph V.3.b.ii.C, and made
them applicable to the information disseminated by all the agencies
subject to these guidelines: ``With regard to analysis of risks to
human health, safety and the environment maintained or disseminated by
the agencies, agencies shall either adopt or adapt the quality
principles applied by Congress to risk information used and
disseminated pursuant to the Safe Drinking Water Act Amendments of 1996
(42 U.S.C. 300g-1(b)(3)(A) & (B)).'' The word ``adapt'' is intended to
provide agencies flexibility in applying these principles to various
types of risk assessment.
    Comments also argued that the continued flow of vital information
from agencies responsible for disseminating health and medical
information to medical providers, patients, and the public may be
disrupted due to these peer review and reproducibility standards. OMB
responded by adding to new paragraph V.3.b.ii.C: ``Agencies responsible
for dissemination of vital health and medical information shall
interpret the reproducibility and peer-review standards in a manner
appropriate to assuring the timely flow of vital information from
agencies to medical providers, patients, health agencies, and the
public. Information quality standards may be waived temporarily by
agencies under urgent situations (e.g., imminent threats to public
health or homeland security) in accordance with the latitude specified
in agency-specific guidelines.''
    Administrative Correction Mechanisms. In addition to commenting on
the substantive standards in these guidelines, many of the comments
noted that the OMB guidelines on the administrative correction of
information do not specify a time period in which the agency
investigation and response must be made. OMB has added the following
new paragraph III.3.i to direct agencies to specify appropriate time
periods in which the investigation and response need to be made.
``Agencies shall specify appropriate time periods for agency decisions
on whether and how to correct the information, and agencies shall
notify the affected persons of the corrections made.''
    Several comments stated that the OMB guidelines needed to direct
agencies to consider incorporating an administrative appeal process
into their administrative mechanisms for the correction of information.
OMB agreed, and added the following new paragraph III.3.ii: ``If the
person who requested the correction does not agree with the agency's
decision (including the corrective action, if any), the person may file
for reconsideration within the agency. The agency shall establish an
administrative appeal process to review the agency's initial decision,
and specify appropriate time limits in which to resolve such requests
for reconsideration.'' Recognizing that many agencies already have a
process in place to respond to public concerns, it is not necessarily
OMB's intent to require these agencies to establish a new or different
process. Rather, our intent is to ensure that agency guidelines specify
an objective administrative appeal process that, upon furthercomplaint
by the affected person, reviews an agency's decision to disagree with
the correction request. An objective process will ensure that the
office that originally disseminates the information does not have
responsibility for both the initial response and resolution of a
disagreement. In addition, the agency guidelines should specify that if
the agency believes other agencies may have an interest in the
resolution of any administrative appeal, the agency should consult with
those other agencies about their possible interest.
    Overall, OMB does not envision administrative mechanisms that would
burden agencies with frivolous claims. Instead, the correction process
should serve to address the genuine and valid needs of the agency and
its constituents without disrupting agency processes. Agencies, in
making their determination of whether or not to correct information,
may reject claims made in bad faith or without justification, and are
required to undertake only the degree of correction that they conclude
is appropriate for the nature and timeliness of the information
involved, and explain such practices in their annual fiscal year
reports to OMB.
    OMB's issuance of these final guidelines is the beginning of an
evolutionary process that will include draft agency guidelines, public
comment, final agency guidelines, development of experience with OMB
and agency guidelines, and continued refinement of both OMB and agency
guidelines. Just as OMB requested public comment before issuing these
final guidelines, OMB will refine these guidelines as experience
develops and further public comment is obtained.

    Dated: December 21, 2001.
John D. Graham,
Administrator, Office of Information and Regulatory Affairs.

Guidelines for Ensuring and Maximizing the Quality, Objectivity,
Utility, and Integrity of Information Disseminated by Federal
Agencies

I. OMB Responsibilities

    Section 515 of the Treasury and General Government Appropriations
Act for FY2001 (Public Law 106-554) directs the Office of Management
and Budget to issue government-wide guidelines that provide policy and
procedural guidance to Federal agencies for ensuring and maximizing the
quality, objectivity, utility, and integrity of information, including
statistical information, disseminated by Federal agencies.

II. Agency Responsibilities

    Section 515 directs agencies subject to the Paperwork Reduction Act
(44 U.S.C. 3502(1)) to--
    1. Issue their own information quality guidelines ensuring and
maximizing the quality, objectivity, utility, and integrity of
information, including statistical information, disseminated by the
agency no later than one year after the date of issuance of the OMB
guidelines;
    2. Establish administrative mechanisms allowing affected persons to
seek and obtain correction of information maintained and disseminated
by the agency that does not comply with these OMB guidelines; and
    3. Report to the Director of OMB the number and nature of
complaints received by the agency regarding agency compliance with
these OMB guidelines concerning the quality, objectivity, utility, and
integrity of information and how such complaints were resolved.

III. Guidelines for Ensuring and Maximizing the Quality, Objectivity,
Utility, and Integrity of Information Disseminated by Federal Agencies

    1. Overall, agencies shall adopt a basic standard of quality
(including objectivity, utility, and integrity) as a performance goal
and should take appropriate steps to incorporate information quality
criteria into agency information dissemination practices. Quality is to
be ensured and established at levels appropriate to the nature and
timeliness of the information to be disseminated. Agencies shall adopt

[[Page 8459]]

specific standards of quality that are appropriate for the various
categories of information they disseminate.
    2. As a matter of good and effective agency information resources
management, agencies shall develop a process for reviewing the quality
(including the objectivity, utility, and integrity) of information
before it is disseminated. Agencies shall treat information quality as
integral to every step of an agency's development of information,
including creation, collection, maintenance, and dissemination. This
process shall enable the agency to substantiate the quality of the
information it has disseminated through documentation or other means
appropriate to the information.
    3. To facilitate public review, agencies shall establish
administrative mechanisms allowing affected persons to seek and obtain,
where appropriate, timely correction of information maintained and
disseminated by the agency that does not comply with OMB or agency
guidelines. These administrative mechanisms shall be flexible,
appropriate to the nature and timeliness of the disseminated
information, and incorporated into agency information resources
management and administrative practices.
    i. Agencies shall specify appropriate time periods for agency
decisions on whether and how to correct the information, and agencies
shall notify the affected persons of the corrections made.
    ii. If the person who requested the correction does not agree with
the agency's decision (including the corrective action, if any), the
person may file for reconsideration within the agency. The agency shall
establish an administrative appeal process to review the agency's
initial decision, and specify appropriate time limits in which to
resolve such requests for reconsideration.
    4. The agency's pre-dissemination review, under paragraph III.2,
shall apply to information that the agency first disseminates on or
after October 1, 2002. The agency's administrative mechanisms, under
paragraph III.3., shall apply to information that the agency
disseminates on or after October 1, 2002, regardless of when the agency
first disseminated the information.

IV. Agency Reporting Requirements

    1. Agencies must designate the Chief Information Officer or another
official to be responsible for agency compliance with these guidelines.
    2. The agency shall respond to complaints in a manner appropriate
to the nature and extent of the complaint. Examples of appropriate
responses include personal contacts via letter or telephone, form
letters, press releases or mass mailings that correct a widely
disseminated error or address a frequently raised complaint.
    3. Each agency must prepare a draft report, no later than April 1,
2002, providing the agency's information quality guidelines and
explaining how such guidelines will ensure and maximize the quality,
objectivity, utility, and integrity of information, including
statistical information, disseminated by the agency. This report must
also detail the administrative mechanisms developed by that agency to
allow affected persons to seek and obtain appropriate correction of
information maintained and disseminated by the agency that does not
comply with the OMB or the agency guidelines.
    4. The agency must publish a notice of availability of this draft
report in the Federal Register, and post this report on the agency's
website, to provide an opportunity for public comment.
    5. Upon consideration of public comment and after appropriate
revision, the agency must submit this draft report to OMB for review
regarding consistency with these OMB guidelines no later than July 1,
2002. Upon completion of that OMB review and completion of this report,
agencies must publish notice of the availability of this report in its
final form in the Federal Register, and post this report on the
agency's web site no later than October 1, 2002.
    6. On an annual fiscal-year basis, each agency must submit a report
to the Director of OMB providing information (both quantitative and
qualitative, where appropriate) on the number and nature of complaints
received by the agency regarding agency compliance with these OMB
guidelines and how such complaints were resolved. Agencies must submit
these reports no later than January 1 of each following year, with the
first report due January 1, 2004.

V. Definitions

    1. ``Quality'' is an encompassing term comprising utility,
objectivity, and integrity. Therefore, the guidelines sometimes refer
to these four statutory terms, collectively, as ``quality.''
    2. ``Utility'' refers to the usefulness of the information to its
intended users, including the public. In assessing the usefulness of
information that the agency disseminates to the public, the agency
needs to consider the uses of the information not only from the
perspective of the agency but also from the perspective of the public.
As a result, when transparency of information is relevant for assessing
the information's usefulness from the public's perspective, the agency
must take care to ensure that transparency has been addressed in its
review of the information.
    3. ``Objectivity'' involves two distinct elements, presentation and
substance.
    a. ``Objectivity'' includes whether disseminated information is
being presented in an accurate, clear, complete, and unbiased manner.
This involves whether the information is presented within a proper
context. Sometimes, in disseminating certain types of information to
the public, other information must also be disseminated in order to
ensure an accurate, clear, complete, and unbiased presentation. Also,
the agency needs to identify the sources of the disseminated
information (to the extent possible, consistent with confidentiality
protections) and, in a scientific, financial, or statistical context,
the supporting data and models, so that the public can assess for
itself whether there may be some reason to question the objectivity of
the sources. Where appropriate, data should have full, accurate,
transparent documentation, and error sources affecting data quality
should be identified and disclosed to users.
    b. In addition, ``objectivity'' involves a focus on ensuring
accurate, reliable, and unbiased information. In a scientific,
financial, or statistical context, the original and supporting data
shall be generated, and the analytic results shall be developed, using
sound statistical and research methods.
    i. If data and analytic results have been subjected to formal,
independent, external peer review, the information may generally be
presumed to be of acceptable objectivity. However, this presumption is
rebuttable based on a persuasive showing by the petitioner in a
particular instance. If agency-sponsored peer review is employed to
help satisfy the objectivity standard, the review process employed
shall meet the general criteria for competent and credible peer review
recommended by OMB-OIRA to the President's Management Council (9/20/01)
(http://www.whitehouse.gov/omb/inforeg/oira_review-process.html),
namely, ``that (a) peer reviewers be selected primarily on the basis of
necessary technical expertise, (b) peer reviewers be expected to
disclose to agencies prior technical/policy positions they may have
taken on the issues at hand, (c) peer reviewers be expected to disclose
to agencies their sources of personal and

[[Page 8460]]

institutional funding (private or public sector), and (d) peer reviews
be conducted in an open and rigorous manner.''
    ii. If an agency is responsible for disseminating influential
scientific, financial, or statistical information, agency guidelines
shall include a high degree of transparency about data and methods to
facilitate the reproducibility of such information by qualified third
parties.
    A. With regard to original and supporting data related thereto,
agency guidelines shall not require that all disseminated data be
subjected to a reproducibility requirement. Agencies may identify, in
consultation with the relevant scientific and technical communities,
those particular types of data that can practicable be subjected to a
reproducibility requirement, given ethical, feasibility, or
confidentiality constraints. It is understood that reproducibility of
data is an indication of transparency about research design and methods
and thus a replication exercise (i.e., a new experiment, test, or
sample) shall not be required prior to each dissemination.
    B. With regard to analytic results related thereto, agency
guidelines shall generally require sufficient transparency about data
and methods that an independent reanalysis could be undertaken by a
qualified member of the public. These transparency standards apply to
agency analysis of data from a single study as well as to analyses that
combine information from multiple studies.
    i. Making the data and methods publicly available will assist in
determining whether analytic results are reproducible. However, the
objectivity standard does not override other compelling interests such
as privacy, trade secrets, intellectual property, and other
confidentiality protections.
    ii. In situations where public access to data and methods will not
occur due to other compelling interests, agencies shall apply
especially rigorous robustness checks to analytic results and document
what checks were undertaken. Agency guidelines shall, however, in all
cases, require a disclosure of the specific data sources that have been
used and the specific quantitative methods and assumptions that have
been employed. Each agency is authorized to define the type of
robustness checks, and the level of detail for documentation thereof,
in ways appropriate for it given the nature and multiplicity of issues
for which the agency is responsible.
    C. With regard to analysis of risks to human health, safety and the
environment maintained or disseminated by the agencies, agencies shall
either adopt or adapt the quality principles applied by Congress to
risk information used and disseminated pursuant to the Safe Drinking
Water Act Amendments of 1996 (42 U.S.C. 300g-1(b)(3)(A) & (B)).
Agencies responsible for dissemination of vital health and medical
information shall interpret the reproducibility and peer-review
standards in a manner appropriate to assuring the timely flow of vital
information from agencies to medical providers, patients, health
agencies, and the public. Information quality standards may be waived
temporarily by agencies under urgent situations (e.g., imminent threats
to public health or homeland security) in accordance with the latitude
specified in agency-specific guidelines.
    4. ``Integrity'' refers to the security of information--protection
of the information from unauthorized access or revision, to ensure that
the information is not compromised through corruption or falsification.
    5. ``Information'' means any communication or representation of
knowledge such as facts or data, in any medium or form, including
textual, numerical, graphic, cartographic, narrative, or audiovisual
forms. This definition includes information that an agency disseminates
from a web page, but does not include the provision of hyperlinks to
information that others disseminate. This definition does not include
opinions, where the agency's presentation makes it clear that what is
being offered is someone's opinion rather than fact or the agency's
views.
    6. ``Government information'' means information created, collected,
processed, disseminated, or disposed of by or for the Federal
Government.
    7. ``Information dissemination product'' means any books, paper,
map, machine-readable material, audiovisual production, or other
documentary material, regardless of physical form or characteristic, an
agency disseminates to the public. This definition includes any
electronic document, CD-ROM, or web page.
    8. ``Dissemination'' means agency initiated or sponsored
distribution of information to the public (see 5 CFR 1320.3(d)
(definition of ``Conduct or Sponsor'')). Dissemination does not include
distribution limited to government employees or agency contractors or
grantees; intra- or inter-agency use or sharing of government
information; and responses to requests for agency records under the
Freedom of Information Act, the Privacy Act, the Federal Advisory
Committee Act or other similar law. This definition also does not
include distribution limited to correspondence with individuals or
persons, press releases, archival records, public filings, subpoenas or
adjudicative processes.
    9. ``Influential'', when used in the phrase ``influential
scientific, financial, or statistical information'', means that the
agency can reasonably determine that dissemination of the information
will have or does have a clear and substantial impact on important
public policies or important private sector decisions. Each agency is
authorized to define ``influential'' in ways appropriate for it given
the nature and multiplicity of issues for which the agency is
responsible.
    10. ``Reproducibility'' means that the information is capable of
being substantially reproduced, subject to an acceptable degree of
imprecision. For information judged to have more (less) important
impacts, the degree of imprecision that is tolerated is reduced
(increased). If agencies apply the reproducibility test to specific
types of original or supporting data, the associated guidelines shall
provide relevant definitions of reproducibility (e.g., standards for
replication of laboratory data). With respect to analytic results,
``capable of being substantially reproduced'' means that independent
analysis of the original or supporting data using identical methods
would generate similar analytic results, subject to an acceptable
degree of imprecision or error.

[FR Doc. 02-59 Filed 1-2-02; 1:36 pm]
billing code 3110-01-M

    Editorial Note: Due to numerous errors, this document is being
reprinted in its entirety. It was originally printed in the Federal
Register on Thursday, January 3, 2002 at 67 FR 369-378 and was
corrected on Tuesday, February 5, 2002 at 67 FR 5365.

[FR Doc. R2-59 Filed 2-21-02; 8:45 am]
BILLING CODE 1505-01-D