26 August 1997 Source: http://www.hr.doe.gov/IG/IG-0407.txt ------------------------------------------------------------------- June 17, 1997 MEMORANDUM FOR THE SECRETARY FROM: John C. Layton Inspector General SUBJECT: INFORMATION: Report on "Audit of the Department of Energy's Scientific and Technical Information Process" BACKGROUND: The Department of Energy has historically devoted significant resources to fund research and development activities. During FY 1995, approximately $5.7 billion was obligated for research and development to management and operating contractors, and another $1.8 billion was obligated through direct procurements, cooperative agreements, and other financial instruments. Scientific and technical information, in most instances, is the only demonstrable result of this research. The objective of this audit was to determine whether the Department's field and contractor sites had developed and maintained systems to manage and track their scientific and technical information and whether these systems ensured that the Department's Office of Scientific and Technical Information received the work products resulting from Department of Energy funded initiatives. DISCUSSION: The audit found that (1) the Department and its contractors had not implemented systems to effectively identify, collect, and disseminate scientific and technical information on a life-cycle basis as required and (2) the Office of Scientific and Technical Information was not receiving all scientific and technical information generated by the contractors. The Department and its management and operating contractors did not identify at the outset of research and development projects the deliverables that would be conveyed to stakeholders. Without a process to identify anticipated deliverables, the Department could not adequately track or monitor the overall success of its research and development program or ensure broad-based dissemination of scientific and technical information. For direct procurements, the Department created the Technical Information Monitoring System (TIMS) to provide life-cycle management of scientific and technical information generated under direct procurements. However, Departmental personnel did not fully maintain or utilize this system. Procurement awards were closed even though the system indicated that deliverables had not been received. In addition, almost one-third of the active awards had delinquent scientific and technical information products. We recommended for management and operating contractor activities that the Deputy Secretary direct Cognizant Secretarial Officers to develop and implement a system to track scientific and technical information on a life-cycle basis. For direct procurements, we recommended that the Deputy Secretary reiterate to field activities the importance of managing scientific and technical information in accordance with established policies and procedures. These recommendations are consistent with the intent of the Government Performance and Results Act in that the Department and its contractors (1) are accountable for scientific and technical information and (2) need to establish scientific and technical information performance measures as well as a life- cycle tracking system. In response to the audit recommendations, the Deputy Secretary endorsed a plan proposed by the Director, Office of Energy Research. This plan calls for the Office of Scientific and Technical Information to initiate and coordinate a strategic planning process that will engage the Departmental community in establishing goals and objectives for scientific and technical information. The plan also calls for the development of performance measures and the establishment of a quality assurance process. Attachment cc: Deputy Secretary Under Secretary U.S. DEPARTMENT OF ENERGY OFFICE OF INSPECTOR GENERAL AUDIT OF THE DEPARTMENT OF ENERGY'S SCIENTIFIC AND TECHNICAL INFORMATION PROCESS The Office of Inspector General wants to make the distribution of its reports as customer friendly and cost effective as possible. Therefore, this report will be available electronically through the Internet five to seven days after publication at the following alternative addresses: Department of Energy Headquarters Gopher gopher.hr.doe.gov Department of Energy Headquarters Anonymous FTP vm1.hqadmin.doe.gov Department of Energy Human Resources Administration Home Page http://www.hr.doe.gov/ig Your comments would be appreciated and can be provided on the Customer Response Form attached to the report. This report can be obtained from the U.S. Department of Energy Office of Scientific and Technical Information P.O. Box 62 Oak Ridge, Tennessee 37831 Report No.:DOE/IG-0407 Capital Regional Audit Office Date of Issue: June 17, 1997 Germantown, MD 20874 AUDIT OF THE DEPARTMENT OF ENERGY'S SCIENTIFIC AND TECHNICAL INFORMATION PROCESS TABLE OF CONTENTS Page SUMMARY . . . . . . . . . . . . ... . . . . . . . . . . . . 1 PART I - APPROACH AND OVERVIEW . . . . . . . . . . . . . . 2 Introduction . . . . . . . . . . . . . . . . . . 2 Scope and Methodology . . . . . . . . . . . . . 2 Background . . . . . . . . . . . . . . . . . . . 3 Observations and Conclusions. . . . . . . . . . . 4 PART II - FINDINGS AND RECOMMENDATIONS . . . . . . . . . . 6 1. Management and Operating Contractor Technical Information . . . . . . . . . . . . . . 6 2. Direct Procurement Technical Information .. . . .. . .. .. . . . . . 14 PART III - MANAGEMENT AND AUDITOR COMMENTS . . . . . . 19 PART IV- APPENDIX . . . . . . . . . . . . . . . . . . . 22 Deputy Secretary's Comments . . . . . . . 22 U.S. DEPARTMENT OF ENERGY OFFICE OF INSPECTOR GENERAL OFFICE OF AUDIT SERVICES AUDIT OF THE DEPARTMENT OF ENERGY'S SCIENTIFIC AND TECHNICAL INFORMATION PROCESS Audit Report Number: DOE/IG-0407 SUMMARY The Department of Energy obligated approximately $7.5 billion in FY 1995 to management and operating contractors, grantees, and other financial instrument recipients for research and development efforts. A key product of research and development work is scientific and technical information. Departmental policies and procedures require the programs that fund the creation of scientific and technical information to follow life-cycle management practices. The audit was performed to determine whether the Department's field and contractor sites had developed and maintained systems to manage and track their scientific and technical information and whether these systems ensured that the Department's Office of Scientific and Technical Information received the work products resulting from Department of Energy funded initiatives. The audit disclosed that the scientific and technical information generated by management and operating contractors was not managed on a life-cycle basis. Implementation and execution problems also existed in the collection and dissemination of products generated under direct procurements. These conditions existed because expected deliverables for management and operating contractors were not identified at the beginning of a project. For direct procurements, Departmental managers were not aware of or bypassed established procedures. In both instances, effective and comprehensive quality assurance processes had not been implemented. As a consequence, the Department was not in a position to know whether it received value for its significant investment in research and development or whether all scientific and technical information resulting from these efforts received the widest possible dissemination in the scientific community. We recommended that the Deputy Secretary direct Cognizant Secretarial Officers to implement a system to track scientific and technical information on a life-cycle basis. For direct procurements, we recommended that the Deputy Secretary reiterate to field activities the importance of managing scientific and technical information in accordance with established policies and procedures. (Signed) _________________________ Office of Inspector General PART I APPROACH AND OVERVIEW INTRODUCTION The Department of Energy has historically devoted significant resources to fund research and development activities. The Department holds ownership and unlimited rights to all scientific and technical information produced by contractors. This information, in most instances, is the only demonstrable result of Department of Energy funded research. The Department, under its authorizing legislation, is charged with the responsibility of collecting and disseminating scientific and technical information to promote scientific progress and public understanding. This audit examined the Department's system for managing scientific and technical information. One objective of the audit was to determine whether the Department's field and contractor sites had developed and maintained systems to track and manage their scientific and technical information. A second objective was to determine whether these systems ensured that the Department's Office of Scientific and Technical Information received the scientific and technical information products resulting from Department of Energy funded work. SCOPE AND METHODOLOGY The audit was performed at three management and operating contractor sites (Sandia National Laboratories, the Yucca Mountain Site Characterization Project, and Brookhaven National Laboratory) and at three Departmental procurement activities (Federal Energy Technology Center [FETC] - Morgantown Site, Headquarters' Procurement Operations Office, and Oakland Operations Office). Fieldwork was conducted from January through October 1996. The sites selected for review were chosen based on the FY 1995 funds obligated to research and development activities and the type of work performed. Additional audit work was performed at the Yucca Mountain Site Characterization Office and the Office of Scientific and Technical Information (OSTI) at Headquarters and Oak Ridge. To accomplish the audit objectives, scientific and technical information policies and procedures were reviewed to determine how management and operating contractors managed scientific and technical information. Contractor systems were examined to determine if these systems provided for life-cycle administration of technical information products. Data maintained in contractor systems for reports and conferences were also compared to OSTI listings. For scientific and technical information generated as a result of direct procurements, field office policies and procedures were reviewed. Detailed tests were performed at Headquarters, Oakland, and the FETC - Morgantown Site using random samples of FY 1995 active and inactive procurement awards. In this context, contract files were examined and contractual reporting requirements were compared to information in OSTI's Technical Information Monitoring System (TIMS). In addition, information in the TIMS was reviewed to determine whether OSTI had received required deliverables. As a part of the audit, discussions were held with officials responsible for managing scientific and technical information. Headquarters, field office, and contractor officials provided applicable guidance and directives and supplied information on specific awards and management systems. Computer-generated data in OSTI's Report Holdings File database and the TIMS were also reviewed to determine whether these systems contained accurate information. The results of this analysis are discussed in Part II of this report. The audit was made in accordance with generally accepted Government auditing standards for performance audits, which included tests of internal controls and compliance with laws and regulations to the extent necessary to satisfy the objectives of the audit. Internal controls were assessed with respect to the management of the Department's scientific and technical information. Because the review of internal controls was limited, it would not necessarily have disclosed all internal control and compliance deficiencies that may have existed. The Director, Office of Energy Research, and officials from the Offices of Procurement and Assistance Management and Scientific and Technical Information waived the exit conference. BACKGROUND The Department of Energy is the fourth largest provider of research and development resources within the Federal Government. During FY 1995, approximately $5.7 billion was obligated for research and development to management and operating contractors who operate Government-owned scientific laboratories for the Department of Energy. Another $1.8 billion was obligated through grants, cooperative agreements, and prime contracts. Thus, the research and development mission represented about one-third of the Department's overall funding. It is imperative that the Department's laboratory contractors be accountable for research and development expenditures and that the taxpayers receive fair value for their investment in these programs. To achieve this, the Department must be able to demonstrate the results of its research and development expenditures. Scientific and technical information is a key product of research and development work. Scientific and technical information is defined as information in any format or medium that is derived from scientific and technical studies; work; or investigations that relate to research, development, demonstration, and other specialized areas and includes unclassified, classified, declassified, and sensitive information. According to DOE Order 1430.1D, "Scientific and Technical Information Management," Departmental programs that fund the creation of scientific and technical information must follow life-cycle management practices. Life-cycle management is a process that is defined as planning for, budgeting, producing, processing, disseminating, and storing in a cost-effective manner scientific and technical information to ensure widespread use by U.S. industry and the general public. The Department identifies, collects, and disseminates scientific and technical information through its Technical Information Management Program. OSTI is responsible for administering the program and for providing centralized planning, development, and utilization of scientific and technical information. Prior reviews of the Department's scientific and technical information program have identified difficulties in collecting and ensuring the delivery of scientific and technical information to the Department. A review of the Department's records management practices by the National Archives Record Administration in 1988 noted "...the tendency for scientists to consider the results of their activities to be their personal property..." rather than handle them as Federal records. An Office of Inspector General audit conducted in 1995 showed that the Superconducting Super Collider Program had not provided the Department with approximately 21 percent of the scientific and technical information generated in connection with this project. The legislative history of the Department of Energy underscores the importance of managing and disseminating scientific and technical information. The Atomic Energy Act of 1954, as amended, created a program for the dissemination of unclassified information to encourage scientific and industrial progress. The Department of Energy Organization Act of 1977 and other laws including the American Technology Pre-eminence Act of 1991 and the Energy Policy Act of 1992 require the Department to make its information available and disseminate the information resulting from its programs. The American Technology Pre-eminence Act further requires the Department to transfer information that results from federally funded research and development activities to the National Technical Information Service of the Department of Commerce in a timely manner for dissemination to the private sector, academia, state and local governments, and Federal agencies. Also, Title 44, U.S.C., states that Government publications, which are publicly releasable, shall be made available to the depository libraries through the Superintendent of Documents, Government Printing Office, for public information. Since inception, the Department has recognized its central role in collecting and disseminating scientific and technical information resulting from its extensive research and development activities. The Department of Energy's 1995 performance agreement with the President, for example, states that the Department will be "...a major partner in world class science and technology through its national laboratories, research centers, university research, and its educational and information dissemination programs." [emphasis added] OBSERVATIONS AND CONCLUSIONS Despite the Department's commitment to developing and maintaining a "world class" research and development program, the Department's process for managing scientific and technical information associated with its $7.5 billion research and development program was not fully effective. Scientific and technical information generated by management and operating contractors was not managed on a life-cycle basis. Implementation and execution problems also existed in the collection and dissemination of products produced under direct procurements (i.e., prime contracts, grants, and other financial assistance awards). Although management and operating contractors received the majority of the Department's research and development dollars annually (in FY 1995 about $5.7 billion obligated for management and operating contractors versus about $1.8 billion obligated for direct procurements), they had not implemented systems to provide life-cycle management of scientific and technical information. Despite Departmental requirements, the contractors at the locations included in the audit did not identify at the outset of Department funded research and development projects what deliverables would be conveyed to stakeholders. Without a process to identify anticipated outputs or deliverables, the Department could not adequately track or monitor the overall success of its research and development program. Further, the Department was unable to ensure broad-based dissemination of scientific and technical information products. This weakness was particularly noteworthy in FY 1996 because the Department was not in a position to examine and fully explain a 30 percent reduction in the number of scientific and technical information products transmitted to OSTI. In contrast, the Department did utilize a life-cycle system to manage scientific and technical information generated under direct procurements. The Department's Technical Information Monitoring System was intended to identify and anticipate scientific and technical information deliverables at the time of the award, track progress against reporting requirements, and record receipt of completed products. However, Departmental personnel did not fully maintain and utilize this system. The audit disclosed that procurement awards were closed and contractors and grantees received payment, despite the fact that the TIMS showed that final deliverables had not been received. The TIMS also indicated that almost one-third of the Department's $23.4 billion in active direct procurement awards had delinquent scientific and technical information deliverables. Nearly 40 percent of these expected deliverables were delinquent by 5 years or longer. This is described more fully on page 16 of this report. These conditions existed in part because of the low priority placed on the identification, collection, and dissemination of scientific and technical information. To correct the weaknesses noted, the Department needs to ensure that its contractors establish life-cycle technical information systems. Performance measures for scientific and technical information management also need to be incorporated into the Department's contracts to improve management of and accountability for information developed under those contracts. Further, quality assurance procedures need to be strengthened to ensure the effective collection and dissemination of scientific and technical information. The Deputy Secretary, the Director, Office of Energy Research, and the Office of Procurement and Assistance Management concurred with the report's recommendations and provided comments on the initial draft report. These comments have been incorporated into this report where appropriate, and the Deputy Secretary's comments are included verbatim in Part IV. Part II of this report provides details of the findings and Part III includes detailed management and auditor comments. PART II FINDINGS AND RECOMMENDATIONS 1. Management and Operating Contractor Technical Information FINDING The Department and its management and operating contractors are required to establish life-cycle systems to identify, collect, and disseminate scientific and technical products generated under Department of Energy funded research and development activities and provide these products to the Office of Scientific and Technical Information (OSTI). In FY 1995, about $5.7 billion was obligated for the research and development activities of the Department's management and operating contractors. Although the Department had initiated development of a new research and development tracking system, effective mechanisms to identify, collect, and disseminate scientific and technical information had not been implemented. The Department neither utilized a life-cycle management process nor ensured that all information generated by the Department's management and operating contractors was provided to OSTI. These conditions existed because expected deliverables were not identified at the beginning of a project and an effective quality review process had not been instituted. As a consequence, the Department was not in a position to know whether it received value for its significant investment in research and development or whether information emanating from these efforts received the widest possible dissemination. RECOMMENDATIONS We recommend that the Deputy Secretary direct Cognizant Secretarial Officers to implement the existing system or create a new research and development tracking system that: Identifies anticipated scientific and technical information deliverables when management and operating contractor task assignments or work authorizations are issued. Establishes processes to track scientific and technical information deliverables on a life-cycle basis that are integrated into Departmental systems. We recommend that the Deputy Assistant Secretary for Procurement and Assistance Management ensure that performance measures for scientific and technical information are incorporated into performance-based contracts for research and development activities. We recommend that the Director, Office of Energy Research: Work with cognizant field office representatives to initiate "for cause" reviews, similar to those conducted under the Department's Business Management Oversight Program, to ensure that the Department's goal regarding the identification, collection, and dissemination of scientific and technical information is met. Perform future assessments of contractor scientific and technical information management under the Department's newly developed Technical Program Oversight Process. MANAGEMENT REACTION The Deputy Secretary endorsed a plan proposed by the Director, Office of Energy Research, to engage the Cognizant Secretarial Officers and those in the field and laboratories to work cooperatively to address the audit recommendations. The Office of Procurement and Assistance Management agreed to take responsibility for adding language to its "Department of Energy Acquisition Guide" regarding the inclusion of performance measures for scientific and technical information in contracts. DETAILS OF FINDING SCIENTIFIC AND TECHNICAL INFORMATION REQUIREMENTS Department of Energy policy (DOE Order 1430.1D "Scientific and Technical Information Management") requires that scientific and technical information be managed on a life-cycle basis. Departmental program offices, the cognizant field office, and the Headquarters program offices must follow life-cycle information management practices to ensure that the information is planned for, budgeted, produced, processed, disseminated, and stored in a cost- effective way to ensure its maximum utilization by all customer segments including U.S. industry and the general public. The Department also requires that program elements and contractors provide the results of completed scientific and technical efforts to OSTI--the coordination point for customer access to all Department of Energy scientific and technical information. Under this system, the contractors that operate the Department's laboratories play a key role in this process. They are to: o Identify outputs in the form of anticipated deliverables when research and development tasks are authorized. o Track anticipated and actual deliverables. o Collect and transmit completed deliverables to OSTI. The Department's Guide for the Management of Scientific and Technical Information states that "program/project officials should specify information outputs that can be measured." At the initiation of a project or task, anticipated products could include technical reports, journal articles, professional publications, and presentations and conference proceedings. These deliverables give Departmental managers a basis for evaluating contractor performance in the research and development field. They also are an important vehicle used by the Department and its laboratories to make their technical findings available to the widest appropriate audience in the scientific community. Ultimately, the deliverables are critical in determining if the scientific work effort was of value in relation to the significant taxpayer investment in the Department of Energy's research and development program. MANAGEMENT SYSTEMS Our review at Sandia, Brookhaven, the Yucca Mountain Site Characterization Project, and corresponding discussions with cognizant Departmental offices disclosed that effective systems had not been established to manage the scientific and technical information resulting from the research accomplished at these facilities. Improvements were needed in identifying, collecting, and transmitting completed scientific and technical products to OSTI. The contractors did not have technical information management systems that: o documented anticipated project deliverables, o identified ongoing scientific projects, or o captured the periodic status or progress of the projects. The results of this audit raised concerns about the ability of the Department to identify and track anticipated versus actual technical deliverables. A parallel situation was addressed in a 1995 Office of Inspector General audit report "Audit of Program Administration by the Office of Energy Research" (DOE/IG-0376). The report concluded that the process used at that time within the Energy Research laboratory complex did not provide the Department with a method for determining if research and development schedules were met, resources were properly used, deliverables were provided as specified, or whether research was performed in accordance with the Department's mission. The report noted that Energy Research officials were concerned that the additional requirements to identify milestones, criteria, or deliverables for management and operating contractors "...could reduce desirable flexibility and stifle the creativity of the researchers." Anticipated Project Deliverables The Department's process for administering scientific and technical information at its management and operating contractors did not establish deliverables as part of the initial task assignment as outlined in DOE Order 5700.C "Work Authorization System." Neither the Department nor its laboratory contractors had the benefit of an initial baseline against which research and development progress could be measured. This eliminated the ability of Sandia, Brookhaven, Yucca Mountain, and the Department to identify and monitor the progress of its scientific and technical initiatives. Sandia, as an alternative, attempted to identify scientific and technical information deliverables through a document collection system. The Laboratory assigned numbers to potential information products when the principal investigators or researcher notified Sandia's scientific and technical information office that delivery of a technical document was imminent. However, even under this process, which did not include baseline data, there was no assurance that Sandia's scientific and technical office received all promised deliverables. The audit disclosed that 99 of 843 anticipated reports and conference papers assigned a tracking number in FY 1995 had not been received as of May 1996. Further, Sandia's system did not include followup procedures, which would have provided an explanation as to the status of the documents that researchers had indicated would be produced. Thus, it was possible that the documents were published, submission of the deliverables was delayed, or the anticipated deliverables were canceled without the knowledge of the Department or Sandia's office of scientific and technical information. Because of the inherent limitations in its process, Sandia could not confirm the existence or nonexistence of its anticipated work products. Instead, officials suggested that individual scientists would have to be contacted to determine the status of each project. Sandia subsequently adopted a new system similar to that employed by Brookhaven. Under this new approach, identification numbers were assigned to research products after the documents were submitted. Once again, the revised system did not require the establishment of anticipated deliverables at the time of task or project approval. Thus, there was no baseline against which actual performance could be evaluated. [emphasis added] Collection and Transmittal of Completed Technical Information Products The collection and transmittal of completed scientific and technical information products to OSTI is a critical component of the life-cycle management process. The audit disclosed weaknesses at Sandia, Brookhaven, and at Yucca Mountain in the transmittal of completed products to OSTI. Both Sandia and Brookhaven maintained database systems that were described previously in this report. We found that these systems needed improvement because their data on completed scientific and technical information products did not fully agree with data in OSTI's database. And, we could not find evidence that management had attempted to identify or reconcile the inconsistencies. At Yucca Mountain the collection and dissemination of information products was decentralized. The Department did not have a mechanism in place to ensure that technical deliverables prepared by the Project's management and operating contractor were transmitted to OSTI. Departmental and Contractor Database Systems Sandia and Brookhaven National Laboratories both maintained databases of known scientific and technical documents. Inconsistencies were noted between the data maintained in the contractors' databases and the data in OSTI's system. The OSTI system had no record of 3 percent of Sandia's documents and 7 percent of Brookhaven's documents. Deliverables not listed in the OSTI database included studies on "...Diagnosis of Cervical Pre-Cancer" and "...Ischemic Acute Renal Failure." It was difficult to assess the substance of these studies and the impact of their not being part of the OSTI database. Although the inconsistencies, in percentage terms, were small it was clear that the Department's goal of providing the widest possible dissemination of the results of its research and development activities was not well served. Other researchers may not have had the benefit of the outcome of this scientific inquiry. The effective dissemination of this information also assists the Department in ensuring that it does not fund duplicative research. Yucca Mountain Site Characterization Project At Yucca Mountain, the management and operating contractor did not maintain a system to record the collection of scientific and technical information. The contractor relied on the Department's project office to review and approve documents for publication and advise authors on which documents should be submitted to OSTI. No followup procedures were implemented to ensure that the documents were actually transmitted to OSTI. In FY 1995, the Department approved 176 Yucca Mountain Site Characterization Project scientific and technical information documents, prepared by management and operating contractors, for publication. Only 47 percent of these documents were ever transmitted to OSTI. The Acting Director, Office of Civilian Radioactive Waste Management, commented on the above audit observations. He emphasized that the Yucca Mountain Site Characterization Project conducts site-specific and engineering studies and that the Project has in place management systems to plan, produce, track, disseminate, and store scientific and technical information. The Acting Director, however, did agree to enhance the Project's Planning and Control System and ensure that documents identified for transmittal to OSTI are forwarded in a timely manner. LACK OF EMPHASIS ON TECHNICAL INFORMATION MANAGEMENT The Department's process for administering scientific and technical information at its M&O contractors appears to have been limited by: (1) a reluctance to establish deliverables as a part of the initial task assignment and (2) an absence of full scope reviews of contractor technical information management systems. The Department has an opportunity to correct these problems by adding performance measures in future contracts relating to contractor scientific and technical information deliverables. Process Implementation The Department's ability to administer scientific and technical information has been hindered by sporadic implementation of its work authorization processes. As in DOE Order 5700.7C "Work Authorization System," a process is established for identifying management and operating contractor scientific and technical information products. The Order requires that an approved work authorization and/or field work proposal identify the schedule, results, and product expected from the work; that Departmental officials ensure that a description of research in progress be provided to OSTI; and that a unique identifying number be assigned to the project throughout its life. Although this Order is still in effect and OSTI has actively encouraged its implementation, processes to administer technical information generated by management and operating contractors, according to OSTI, had not been successful. For example, an OSTI official stated that several program organizations within the Department did not use the fieldwork proposal system or provide unique identifying numbers for their projects. In another instance, a Brookhaven official stated that they did not believe it was necessary to identify deliverables since there were few reasons for researchers to withhold the results of their research efforts. In 1993, the Department was requested by the White House, Office of Science and Technology Policy, to participate in gathering information on research and development activities supported by the Federal Government. The Department was to electronically transmit information on the allocation of its budget for non-R&D and R&D activities, the substantive areas and stages of the research, and project descriptions by laboratory. The Department was not able to provide complete information, and the information that was provided was viewed as being unresponsive to the request. The importance of identifying and tracking scientific and technical deliverables was apparent in this exercise. To correct this situation, the Deputy Secretary subsequently directed the development of the R&D Tracking System. Beginning with a call for data in 1996, each laboratory was directed to identify the deliverable expected for each approved R&D project and track each project with a unique identifying number. Successful implementation of this system will be dependent upon the full support of Departmental program managers and management and operating contractors. OSTI Reviews In 1995 examinations of contractor scientific and technical information programs were incorporated into the business management system reviews under the Department's Business Management Oversight Pilot Project. The Business Management Pilot was oriented towards a business systems review. Prior to the implementation of the Business Management Pilot, OSTI conducted about 20 to 30 onsite Departmental and management and operating contractor appraisals each year. These reviews addressed three main objectives: an evaluation of life-cycle management of scientific and technical information resources, scientific and technical information reporting, and compatibility with the Department's scientific and technical information program. In contrast, the approach utilized in the Business Management Oversight Program was substantially less comprehensive, relying primarily on a process-based evaluation. Further, despite the significance of its research and development activities, in FY 1996 the Department conducted only three Business Management reviews addressing scientific and technical information management. After its initial participation in the Business Management Pilot in September 1995, OSTI proposed that the Department improve management of its scientific and technical information program by transferring responsibility for reviews of the program to the Department's Technical Program Oversight Process. This annual technical review process, to be implemented during calendar year 1996, was intended to improve oversight of technical programs at Departmental laboratories. OSTI had not received a response to its proposal as of the end of this audit. Regardless of the process chosen for the review of scientific and technical information, it is essential for the Department to ensure that its goals regarding the identification, collection, and dissemination of scientific and technical information are met. Scientific and Technical Information as Performance Measures As the Department transitions to performance-based contracting, managers will also need to develop performance measures for scientific and technical information. These performance measures or benchmarks assist in tracking Departmental and contractor accomplishments to ensure, among several goals, that the taxpayers receive full value for the investment made in the Department's research and development program. The Department's contract reform team emphasized this point when it concluded that: The Department must clearly state what we expect from our contractors and develop meaningful ways to measure whether they are meeting our expectations. Departmental officials have begun to develop "best-in- class" measures for inclusion in new and renegotiated contracts. However, performance measures developed by OSTI in 1995 for management of scientific and technical information were not included in recently awarded performance-based contracts. INVESTMENT IN SCIENTIFIC AND TECHNICAL INFORMATION The Department invested over $7 billion in FY 1995 in research and development activities. Nonetheless, the Department was not in a position to know whether it received value, in the form of scientific and technical information, for its investment. The Department had neither a systematic process to collect technical information products from its management and operating contractors nor a mechanism to compare actual scientific and technical information accomplishments against expectations. Scientific and Technical Information Products The significance of not having a system to manage scientific and technical information was illustrated by the recent drop in the number of technical information products being delivered to OSTI. Between FY 1995 and FY 1996, there was a 30 percent reduction in the number of information products transmitted to OSTI. In FY 1996, OSTI received 14,254 documents, compared to 20,452 documents it received in FY 1995. Yet, during this period, the Department's research and development budget remained relatively constant. We questioned officials to determine the cause for the significant reduction in documents transmitted to OSTI. Although they had not performed a definitive trend analysis, they speculated that this could have occurred due to a changing budget, an increase in the number of journal articles, or fewer conferences being held. However, we concluded that since appropriate benchmarks had not been established for scientific and technical information products, there was no way for Departmental officials to determine if they received the appropriate number of products or whether the sharp decline in the receipt of such products was consistent with the Departmental and congressional direction or expectations. 2. Direct Procurement Technical Information FINDING Departmental program managers and contracting personnel are responsible for obtaining and promptly transmitting to the Office of Scientific and Technical Information technical deliverables generated under contracts, grants, and other financial assistance instruments. The Department established a system to track and process this type of information for direct procurements, but weaknesses existed in its implementation. Procurement instruments were closed even though the OSTI database indicated that deliverables had not been provided, and many other deliverables were not being provided on a timely basis. The Department's tracking system at the time of the audit showed that almost one-third of the Department's $23.4 billion in active direct procurements had delinquent scientific and technical deliverables. These conditions existed because managers were not aware of or bypassed controls, and a comprehensive quality assurance process had not been established. The Department, as a consequence, could not be assured that all scientific and technical information products were being provided to its customers. RECOMMENDATIONS We recommend that the Deputy Secretary reemphasize to field activities the importance of processing scientific and technical information in accordance with Departmental policies and procedures, and require full implementation by all Departmental elements. We recommend that the Director, Office of Scientific and Technical Information, working in conjunction with the Deputy Assistant Secretary for Procurement and Assistance Management, institute a comprehensive quality assurance process that ensures that all field activities are fulfilling their responsibilities for collecting, processing, and disseminating scientific and technical information. MANAGEMENT REACTION The Deputy Secretary agreed to reemphasize scientific and technical information as a key deliverable of research and development. The Director, Office of Energy Research, indicated that the Offices of Scientific and Technical Information and Procurement and Assistance Management had initiated discussions to develop a quality assurance process. DETAILS OF FINDING TECHNICAL INFORMATION REPORTING REQUIREMENTS Departmental managers, in accordance with Department of Energy Acquisition Regulation (DEAR) 935.010 "Scientific and Technical Reports" and DOE Order 1332.2 "Uniform Reporting System for Federal Assistance," are required to specify scientific and technical information deliverables under contracts, grants, and other financial assistance instruments and ensure that these deliverables are promptly provided to OSTI. DOE Order 1430.1D "Scientific and Technical Information Management" emphasizes the importance of obtaining this information and states that procurement instruments are not considered closed until the Head of the cognizant Departmental element has ensured that the deliverables have been received. SCIENTIFIC AND TECHNICAL DELIVERABLES In contrast with the procedures used in administering management and operating contracts, the Department developed a life-cycle system to administer scientific and technical information generated under prime contracts, grants, and other financial assistance instruments. Key components of the system included identification of the deliverables; reviews of completed products for patent, classification, and sensitivity issues; tracking the status of information products utilizing automated databases; and receipt and storage of required scientific and technical deliverables by OSTI. Although a comprehensive system had been established, weaknesses existed in its implementation. Awards were closed without deliverables, and many scientific and technical deliverables were not provided to OSTI on a timely basis for dissemination. Deliverables Not Received or Provided in a Timely Manner Between FY 1991 and FY 1995, Departmental records indicated that 89 awards were closed even though required scientific and technical information products had not been provided to OSTI. The value of the closed awards was $138 million. In total, 389 separate deliverables required under these procurement actions were not provided to OSTI. In addition, many deliverables due under prime contracts, grants, and cooperative agreements were delinquent according to Departmental records. At the end of FY 1995, the Department had 7,030 active and inactive multi- year awards requiring scientific and technical information deliverables. The total value of these awards was $23.4 billion. Of this amount, procurement actions valued at $7.3 billion had delinquent scientific and technical products. To determine the extent to which scientific and technical information had not been provided to the Department's central coordination point, an aging schedule was prepared by OSTI of overdue annual and final reports. The following table shows the results of this analysis for the Headquarters, the FETC - Morgantown Site, and the Oakland procurement offices. These are three of the most active procurement offices in the Department that handle scientific and technical contracts, grants, and cooperative agreements. AGING SCHEDULE OF DELINQUENT ANNUAL AND FINAL REPORTS HQ Proc. FETC Oakland Total 0-1 yrs. 78 53 191 322 1-2 yrs. 47 10 97 154 2-3 yrs. 37 11 76 124 3-4 yrs. 30 4 45 79 5+ yrs. 97 12 315 424 TOTALS 289 90 724 1,103 In total, 1,103 scientific and technical information reports were delinquent for contracts, grants, and cooperative agreements issued by these three offices. Of this total, 424 annual and final reports had been delinquent for 5 years or more. We believe that the extensive delays in receipt of this data by OSTI minimize its usefulness to the Department and, potentially, in the scientific community. SYSTEM IMPLEMENTATION PROBLEMS OSTI did not receive many of these annual and final reports because Departmental managers incorrectly processed information, bypassed system controls, or did not understand their role in processing scientific and technical information. Further, the Department did not have an effective quality assurance process. Controls Bypassed or Not Understood Various problems were noted in the collection and transmittal of information to OSTI. In some instances, the delinquent reports had been received by a Departmental element but had not been forwarded to OSTI. For example, FETC - Morgantown Site received several quarterly and final reports during FY 1995 but had not provided them to the Department's central repository for scientific and technical information. A Morgantown official explained that many of these reports needed to go through a patent review process and had to be released by the contracting officer's representative before they went to OSTI. This process often delayed the transmittal of reports by anywhere from a few months to over a year. In other cases, data contained in the Department's Procurement Acquisition Data System (PADS) were not accurate. For example, an examination of 25 procurement award files at Headquarters indicated that the scientific and technical deliverables listed for 17 of these awards were incorrect in the PADS. PADS is the primary source of information for the Department's scientific and technical database (Technical Information Monitoring System--TIMS), and OSTI is dependent upon PADS to monitor the generation and collection of scientific and technical information. Data errors in the PADS system, therefore, limited OSTI's ability to perform its assigned task. Linkage between local database systems and the PADS and OSTI systems was also weak. Officials at FETC-Morgantown Site waived reporting requirements and changed information in their local system but did not update the Departmental systems. In another situation, officials at the Oakland Operations Office, from FY 1991 to FY 1994, closed eight awards valued at $7.8 million prior to obtaining required clearance from the Office of Scientific and Technical Information. Program managers also indicated that they had been excluded from the process or were not aware of their assigned responsibilities. For example, a contract file for a $240,000 award at Headquarters indicated that quarterly technical progress reports and a final technical report were due at the end of the award's performance period. However, none of the required reports had been sent to OSTI. The responsible procurement official stated that it was the "responsibility of the program officials to obtain scientific and technical information documents." The program official responded that she received the reports, but she had never seen the reporting checklist. The official was of the opinion that it was the procurement people who [were] responsible for obtaining scientific and technical information. According to DOE Order 1430.1D, both program and procurement officials are responsible for ensuring that this information is provided to OSTI. Quality Assurance Program The absence of an effective quality assurance program also contributed to problems in the receipt of technical information products. For example, a Departmental technical information officer in Oakland was unable to correct the problem of delinquent scientific and technical information reporting. At the Oakland Operations Office, a monthly report was prepared by the scientific and technical information officer and forwarded to procurement and program officials to notify them of delinquent technical information deliverables. However, this process did not result in improved receipt and transmittal of scientific and technical information to OSTI. OSTI also performed limited examinations of scientific and technical information statistics during its reviews at field activities. However, these reviews looked only at the number of information products obtained, not at why information was not being collected from contractors and forwarded to OSTI for dissemination. IMPACT OF DELINQUENT SCIENTIFIC AND TECHNICAL INFORMATION The Department is involved in major scientific efforts to clean up the environment, pursue basic research, develop alternative sources of energy, and research health and environmental issues. The value of the Department's research and development effort rests not only in the quality of the work, but also in the timely dissemination of scientific and technical information to the public and scientific community. The success of these efforts and the value of the research is not maximized when its availability and dissemination is limited. And, the Department's ability to demonstrate positive results from its expenditures is hindered. Information that is not obtained from a contractor or grantee, or information that resides in a local procurement or programmatic office, does not enhance or contribute to the expansion of scientific knowledge or allow the Department to fulfill its research and development mission. PART III MANAGEMENT AND AUDITOR COMMENTS The Deputy Secretary of Energy, the Director, Office of Energy Research, and the Office of Procurement and Assistance Management agreed with the report's recommendations. A summary of management's comments and the auditor's response follows. Finding 1. Management and Operating Contractor Technical Information Recommendation 1. The Deputy Secretary direct Cognizant Secretarial Officers to implement the existing system or create a new R&D tracking system that: (1) identifies anticipated scientific and technical information deliverables when management and operating contractor assignments or work authorizations are issued and (2) establishes processes to track scientific and technical information deliverables on a life-cycle basis that are integrated into Departmental systems. Management Comments. Concur. The Deputy Secretary agreed that the Department should emphasize scientific and technical information as a key deliverable of research and development and endorsed the plan from the Office of Energy Research that directed Cognizant Secretarial Officers to fully implement a tracking system to ensure life-cycle management of R&D projects including the resulting scientific and technical information. Auditor Comments. Management comments are responsive to the recommendation. Recommendation 2. The Deputy Assistant Secretary for Procurement and Assistance Management ensure that performance measures for scientific and technical information are incorporated into performance-based contracts for research and development activities. Management Comments. Concur. The Office of Procurement and Assistance Management agreed to fulfill its contracting responsibility by developing appropriate language concerning inclusion of performance measures for scientific and technical information to be included in its Acquisition Guide. This is to convey management's expectation that contracts contain performance measures; however, development and approval of performance measures is the responsibility of the cognizant program office under the requirements of the Government Performance and Results Act. Auditor Comments. While management concurred and proposed several positive initiatives, its comments were not fully responsive to the recommendation. The Office of Inspector General recognizes that it is the responsibility of cognizant program and field offices to establish scientific and technical information performance measures. However, it is Procurement's responsibility to ensure, through its contract approval process, that established scientific and technical information performance measures are contained in all applicable contracts. Procurement's response did not indicate what steps or actions it would take to ensure that appropriate measures are included in performance-based contracts. Recommendation 3. The Director, Office of Energy Research: (1) work with cognizant field office representatives to initiate "for cause" reviews, similar to those conducted under the Department's Business Management Oversight Program, to ensure that the Department's goal regarding the identification, collection, and dissemination of scientific and technical information is met and (2) perform future assessments of contractor scientific and technical information management under the Department's newly developed Technical Program Oversight Process. Management Comments. Concur. The Office of Energy Research indicated that the recommendations for scientific and technical information complemented other management changes currently underway at the Department, such as the Government Performance and Results Act of 1993. Part of that reform is to establish performance-based management with a focus on outcomes rather than processes. The Office of Energy Research plans to integrate steps for improvement into the strategic planning process, with annual performance plans and results reported beginning with the Fiscal Year 1999 budget cycle, consistent with the Department's overall planning efforts. The Office of Energy Research also agreed to establish performance expectations that will be incorporated in future technical program reviews with a goal to incorporate scientific and technical information in Fiscal Year 1998 program reviews. Auditor Comments. Management's comments are generally responsive to the recommendation. However, the Office of Energy Research needs to develop a detailed action plan on how it will work with cognizant field office representatives to initiate for cause reviews or perform future assessments of scientific and technical information management by the Department's contractors. Finding 2. Direct Procurement Technical Information Recommendation 1. The Deputy Secretary reemphasize to field activities the importance of processing scientific and technical information in accordance with Departmental policies and procedures, and require full implementation by all Departmental elements. Management Comments. Concur. The Deputy Secretary agreed that the Department should emphasize scientific and technical information as a key deliverable of research and development. As a means of taking corrective action, the Deputy Secretary endorsed the plan proposed by the Director, Office of Energy Research, to have the Office of Scientific and Technical Information initiate and coordinate a strategic planning process to engage the Department's R&D community in establishing goals, objectives, and measures for scientific and technical information. In addition, to ensure implementation, performance expectations will be established and incorporated in technical program reviews with a goal to incorporate scientific and technical information in Fiscal Year 1998. Auditor Comments. Management comments are responsive to the recommendation. Recommendation 2. The Director, Office of Scientific and Technical Information, working in conjunction with the Deputy Assistant Secretary for Procurement and Assistance Management, institute a comprehensive quality assurance process that ensures that all field activities are fulfilling their responsibilities for collecting, processing, and disseminating scientific and technical information. Management Comments. The Director, Office of Energy Research, concurred with the recommendation and indicated that the Office of Scientific and Technical Information had initiated communications with the Deputy Assistant Secretary for Procurement and Assistance Management to institute a quality assurance process. A memorandum including a summary of OSTI's experience with the field activities, citing specific quality assurance issues would be provided to Procurement and Assistance Management to develop a process to address the needs related to scientific and technical information. The Office of Procurement and Assistance Management agreed to assist the Director of OSTI in effectively communicating to the field offices to ensure that they understand the requirement to provide deliverables to OSTI. They will also assist OSTI in making their system more accurate by securing timely deliverables through such communications. Auditor Comments. The comments of the Director, Office of Energy Research, were responsive to the recommendation. However, the Office of Procurement and Assistance Management's response did not detail the actions it would take towards instituting a comprehensive quality assurance process that ensures that all field activities are fulfilling their responsibilities for collecting, processing, and disseminating scientific and technical information. APPENDIX PART IV DEPUTY SECRETARY'S COMMENTS Department of Energy Washington, DC 20585 OFFICE OF THE DEPUTY SECRETARY MEMORANDUM FOR: John C. Layton Inspector General FROM: Charles B. Curtis Deputy Secretary SUBJECT: Comments on the Initial Draft Report on "Audit of the Department of Energy's Scientific and Technical Information Process" DATE: April 30, 1997 I appreciate being given the opportunity to offer my views on the subject report. It raises several important issues related to the scientific and technical information resulting from the Department's research and development activities. I agree that the Department should emphasize scientific and technical information as a key deliverable of research and development. The value of the Department's research and development effort relies on adequate information being made available to the scientific community and the public, thereby helping Americans to better understand what we do. Further, as a means of taking corrective action, I endorse the plan proposed by the Director of the Office of Energy Research to have the Office of Scientific and Technical Information conduct strategic planning sessions to address these important issues. By working cooperatively through the Cognizant Secretarial Officers and those engaged in the field and laboratories, a stronger, more effective scientific and technical information process should result. IG Report No. DOE/IG-0407 CUSTOMER RESPONSE FORM The Office of Inspector General has a continuing interest in improving the usefulness of its products. We wish to make our reports as responsive as possible to our customers' requirements, and therefore ask that you consider sharing your thoughts with us. On the back of this form, you may suggest improvements to enhance the effectiveness of future reports. Please include answers to the following questions if they are applicable to you: 1. What additional background information about the selection, scheduling, scope, or procedures of the audit or inspection would have been helpful to the reader in understanding this report? 2. What additional information related to findings and recommendations could have been included in this report to assist management in implementing corrective actions? 3. What format, stylistic, or organizational changes might have made this report's overall message more clear to the reader? 4. What additional actions could the Office of Inspector General have taken on the issues discussed in this report which would have been helpful? Please include your name and telephone number so that we may contact you should we have any questions about your comments. Name ____________________________ Date_____________________ Telephone _______________________ Organization_____________ When you have completed this form, you may telefax it to the Office of Inspector General at (202) 586-0948, or you may mail it to: Office of Inspector General (IG-1) U.S. Department of Energy Washington, D.C. 20585 ATTN: Customer Relations If you wish to discuss this report or your comments with a staff member of the Office of Inspector General, please contact Wilma Slaughter at (202) 586-1924.