12 June 1997 Source: http://www.access.gpo.gov/su_docs/aces/aaces002.html ----------------------------------------------------------------------- Attack Warning: Status of the Cheyenne Mountain Upgrade Program (Letter Report, 09/01/94, GAO/AIMD-94-175). Colorado's Cheyenne Mountain complex is designed to track and warn of enemy air attacks. The Air Force program to modernize aging computer systems at the facility is eight years behind schedule and $792 million over budget. Air Force officials now project that the completion date will slip by another three years and that development costs will increase by at least $104 million. Both projections, GAO believes, may be understated. Moreover, major subsystem incompatibilities could prevent the upgraded system from becoming fully operational. --------------------------- Indexing Terms ----------------------------- REPORTNUM: AIMD-94-175 TITLE: Attack Warning: Status of the Cheyenne Mountain Upgrade Program DATE: 09/01/94 SUBJECT: Air defense systems Warning systems Systems evaluation Systems development life cycle Systems conversions Air Force procurement Cost overruns Systems compatibility Research and development costs Computerized information systems IDENTIFIER: NORAD Cheyenne Mountain Complex Computer System NORAD Cheyenne Mountain Upgrade Program Colorado Springs (CO) NORAD Tactical Warning and Attack Assessment System NORAD Communications System Segment Replacement Program NORAD Command Center Processing and Display System Replacement Program NORAD Survivable Communications Integration System ************************************************************************** * This file contains an ASCII representation of the text of a GAO * * report. Delineations within the text indicating chapter titles, * * headings, and bullets are preserved. Major divisions and subdivisions * * of the text, such as Chapters, Sections, and Appendixes, are * * identified by double and single lines. The numbers on the right end * * of these lines indicate the position of each of the subsections in the * * document outline. These numbers do NOT correspond with the page * * numbers of the printed product. * * * * No attempt has been made to display graphic images, although figure * * captions are reproduced. Tables are included, but may not resemble * * those in the printed version. * * * * A printed copy of this report may be obtained from the GAO Document * * Distribution Facility by calling (202) 512-6000, by faxing your * * request to (301) 258-4066, or by writing to P.O. Box 6015, * * Gaithersburg, MD 20884-6015. We are unable to accept electronic orders * * for printed documents at this time. * ************************************************************************** Cover ================================================================ COVER Report to the Chairman, Subcommittee on Defense, Committee on Appropriations, House of Representatives September 1994 ATTACK WARNING - STATUS OF THE CHEYENNE MOUNTAIN UPGRADE PROGRAM GAO/AIMD-94-175 Cheyenne Mountain Upgrade Program Abbreviations =============================================================== ABBREV ADP - x AIMD - Accounting and Information Management Division APCC - Alternate Processing and Correlation Center CCPDS-R - Command Center Processing and Display System Replacement CMU - Cheyenne Mountain Upgrade CSSR - Communications System Segment Replacement GAO - General Accounting Office IOC - initial operational capability ITW/AA - Integrated Tactical Warning and Attack Assessment NORAD - North American Aerospace Defense Command SCIS - Survivable Communications Integration System SPADOC - Space Defense Operations Center Letter =============================================================== LETTER B-257932 September 1, 1994 The Honorable John P. Murtha Chairman, Subcommittee on Defense Committee on Appropriations House of Representatives Dear Mr. Chairman: On March 22, 1994, you requested that we review the status of computer upgrades at the Air Force's Cheyenne Mountain Complex at Colorado Springs, Colorado. These upgrades, known collectively as the Cheyenne Mountain Upgrade (CMU) program, are intended to modernize the systems which are the nucleus of the worldwide Integrated Tactical Warning and Attack Assessment (ITW/AA) system. Designed to identify and track potential enemy objects, these systems provide critical surveillance, air defense warning, and attack assessment information to United States and Canadian leaders. The CMU program has experienced a series of development problems since it began in 1981. This report, the eleventh that GAO has issued on the program since 1988, discusses (1) the current status of the cost and schedule for the CMU program, (2) the status of CMU subsystems development and performance, (3) the status of the integration of the systems, and (4) management issues affecting future CMU development. Our work was performed between April 1994 and July 1994 in accordance with generally accepted government auditing standards. Appendix I provides a detailed description of our objectives, scope, and methodology for the report. RESULTS IN BRIEF ------------------------------------------------------------ Letter :1 After a series of delayed completion schedules and increased development cost estimates, the CMU program is 8 years behind schedule and $792 million over budget. The Air Force recently determined that the program will meet neither its currently scheduled completion date of December 1995 nor target development costs of $1.76 billion. Air Force officials now project that the completion date will slip by an additional 3 years and that development costs will increase by at least $104 million. Both projections, GAO believes, may be understated. Initial versions of several CMU subsystems that the Air Force has declared operational are unreliable and do not meet users' requirements. As a result, those subsystems must be operated in parallel with the systems they are meant to replace. Concurrently operating the old Cheyenne Mountain systems costs the Air Force an additional $22 million annually. Moreover, significant incompatibilities among the CMU component subsystems have been identified that could prevent the overall CMU system from becoming fully functional. Further incompatibilities have been identified between CMU and the rest of the ITW/AA system that could hinder integration of the worldwide ITW/AA system. In 1992, GAO reported that CMU would continue to face serious development and integration problems until the Air Force performs the analysis needed to define an overall CMU architecture.\1 Because the Air Force has not yet fully addressed this issue, integration problems continue to surface and will persist. The Air Force recognizes these problems and has already made some improvements to the CMU management structure to provide a more integrated approach. It is currently exploring further changes. Nevertheless, managing the complex schedule of developing, testing, deploying, and finally integrating the large number of independently developed CMU systems will pose formidable challenges. Continued close oversight of the program is warranted to ensure that the Air Force faces up to these challenges. -------------------- \1 Attack Warning: Lack of System Architecture Contributes to Major Development Problems (GAO/IMTEC-92-52, June 11, 1992). BACKGROUND ------------------------------------------------------------ Letter :2 The North American Aerospace Defense Command (NORAD) is responsible for warning United States and Canadian leaders of any missile, air, or space attack. This mission is supported by the ITW/AA system, which is designed to identify and track potential enemy objects. The ITW/AA system consists of a worldwide network of ballistic missile, atmospheric, and space warning systems; intelligence centers; associated communications links; and command and control centers. The computer and telecommunications hardware and software at the Cheyenne Mountain Complex, the command center for NORAD, form the nucleus of the ITW/AA system. Information from Cheyenne Mountain is provided to national command authorities primarily through the Air Force's Space Command. In 1981, the Air Force began a modernization effort consisting of five separate acquisitions to replace aging and obsolete computer systems at the Cheyenne Mountain Complex. The five original subsystems under development for the Cheyenne Mountain Complex are (1) the Communications System Segment Replacement (CSSR) to process and control most of the internal and external automated communications at the Cheyenne Mountain Complex, (2) the Survivable Communications Integration System (SCIS) to provide multiple survivable communications capabilities between missile warning sensors, command centers, and other users, (3) the Command Center Processing and Display System Replacement (CCPDS-R) to process and display ballistic missile warning data received from sensors located throughout the world, (4) Granite Sentry to process and display data for use by all air defense, command post, battle staff, and weather support activities, and (5) the Space Defense Operations Center 4 (SPADOC 4) to process space defense and space surveillance data. A sixth subsystem, the Alternate Processing and Correlation Center (APCC) at Offutt Air Force Base in Nebraska, was added in 1989. The APCC will provide backup missile warning and air defense information should the systems at the Cheyenne Mountain Complex fail. From 1988 to 1992, we issued 10 reports addressing a variety of aspects of the CMU program, including assessments of individual CMU subsystems as well as overall reviews of the CMU program. In general, the reports discussed development problems being experienced on subsystem projects and ways to preclude more such problems arising in the future. In 1992, we reported that the Air Force was continuing to develop CMU subsystems as five separate systems and that it lacked an overall CMU architecture, thus increasing the risks that the CMU would not meet systems requirements, operate as an integrated unit, or be capable of evolving to meet the needs of new missions in the future. We also noted that because of cost and schedule overruns, the Air Force had deferred some CMU requirements, leaving the system with less capability than originally planned. A list of our previous reports on the CMU program appears at the end of this report. THE CMU PROGRAM CONTINUES TO EXPERIENCE COST INCREASES AND SCHEDULE DELAYS ------------------------------------------------------------ Letter :3 The five original acquisition programs for the Cheyenne Mountain Complex were initially scheduled for completion in 1987 at a cost of $968 million. In 1989, we reported that the program was 7 years behind schedule and already $342 million over budget. We attributed these problems to a cumbersome and diffuse management structure.\2 The Air Force responded by combining the five original subsystems and the APCC backup subsystem into a single CMU program, and committing to completion by December 1995 at a cost of $1.58 billion. However, we reported in 1991 that this cost estimate did not include funding for all requirements, and we estimated that costs for completing a fully functional, mission-ready system would surpass $1.9 billion.\3 The Air Force recently determined that it will not meet either its December 1995 schedule milestone or its last approved $1.76 billion cost estimate. The Air Force states that it has been unable to meet its cost and schedule milestones because it underestimated the complexities of delivering and integrating systems to support three parallel warning missions\4 and the difficulty and cost of subsystem, mission, and integration testing. Air Force officials now project that CMU will be complete in December 1998 but that costs can only be estimated to exceed $1.864 billion, since they concede that additional unidentified integration problems are likely to occur. As discussed below, we believe even these cost estimates are understated. -------------------- \2 Attack Warning: Better Management Required to Resolve NORAD Integration Deficiencies (GAO/IMTEC-89-26, July 7, 1989). \3 Attack Warning: Costs to Modernize NORAD's Computer System Significantly Understated (GAO/IMTEC-91-23, April 10, 1991). \4 The three missions are to warn against ballistic missile, space, and atmospheric (bomber) attacks. MAJOR CMU SUBSYSTEMS STILL NEED SIGNIFICANT DEVELOPMENT WORK ------------------------------------------------------------ Letter :4 Although the Air Force committed CMU development to completion by December 1995, significant work remains to be done on some of its subsystems. Air Force operation and certification testing has revealed that these subsystems do not meet user requirements. Additionally, the costs for this work are being understated because the Air Force has commingled the costs of some development activities with costs related to other systems. THE AIR FORCE HAS PLACED SOME SUBSYSTEMS IN USE PREMATURELY ---------------------------------------------------------- Letter :4.1 The CMU subsystems are in various stages of development, test, and operations. SCIS is the only subsystem still completely in development; initial operational test and evaluation is scheduled to begin at 20 sensor sites in September 1994. CCPDS-R has been installed at Cheyenne Mountain, is being tested and monitored, but is not yet operational. Portions of CSSR and SPADOC 4 are operational; other parts are still being tested. Phase III of Granite Sentry is operational; the subsequent phase is under development. Construction of the APCC, the backup correlation center, is complete. The APCC missile warning functions are being tested and evaluated. The Air Force recently declared three of these subsystems (CSSR, CCPDS-R, and SPADOC 4) to have reached initial operational capability (IOC). However, tests conducted by the Air Force Operational Test and Evaluation Center found that none of the three subsystems met users' requirements. For example, the CSSR subsystem, which must relay critical attack and warning messages to other CMU subsystems, took too long to do this in many cases and sometimes failed to relay messages altogether. Additionally, further independent testing by U.S. Space Command found that only one of the three subsystems (SPADOC 4) met Air Force standards for accuracy, reliability, and timeliness. Passing these tests is essential before the new CMU subsystems can be made operational and the old computer systems removed. Until then, continuing to operate the old systems in conjunction with the new, costs the Air Force $22 million annually. DEVELOPMENT COSTS ARE UNDERSTATED ---------------------------------------------------------- Letter :4.2 Because the Air Force is paying for some CMU subsystem development work with funds designated for operations and maintenance, the cost of developing the CMU is being understated. While a system is officially in its development phase, any necessary fixes or changes are to be paid for using money set aside specifically for development. However, once a system is placed in regular use, any fixes, changes, or deferred enhancements are to be paid for using operations and maintenance funds. At the Cheyenne Mountain Complex, operations and maintenance funds are designated for maintaining the old ITW/AA systems as well as the new CMU subsystems that are being used with them. As a result, the costs for the substantial amount of development work that remains to be done on these CMU subsystems are being commingled with ITW/AA costs and are being paid from operations and maintenance funds. For example, in April 1994, the Air Force identified 34 changes essential for the independent operation of the SPADOC 4 subsystem that would be paid for with operations and maintenance funds. The Air Force was not able to provide an estimate of the associated development costs that are being paid for with operations and maintenance funds because they do not specifically track how much of their operations and maintenance funds are being used for CMU development activities. INTEGRATION PROBLEMS REMAIN UNRESOLVED ------------------------------------------------------------ Letter :5 During testing of the complete missile warning function, the Air Force identified significant incompatibilities among CMU's component subsystems, which could prevent the complete CMU system from functioning properly. Further incompatibilities have been identified between CMU and other, already deployed ITW/AA systems that could hinder integration of the worldwide ITW/AA system. While parts of the system might perform their functions successfully, the parts might fail to work together properly, causing warning data to be lost or inaccurately transmitted to decisionmakers. We reported on CMU integration problems as early as 1988 and, in a later report, recommended that the Air Force develop an overall system architecture. Such an architecture would have specified how subsystems interact and could have been used by developers as a common reference point to ensure interoperability among subsystems. Without it, we projected that the program would continue to face integration and development problems that were continuing to surface at that time.\5 As of April 1994, the Air Force unit responsible for ITW/AA integration had identified (1) eight unresolved compatibility problems with major impact to cost, schedule, and performance, (2) four compatibility problems that have been resolved, but for which the final cost and schedule impacts have not yet been determined, and (3) 144 potential integration problems that will require more analysis. The Air Force acknowledges that these 144 items may result in additional significant problems being identified. One example of a serious incompatibility among CMU subsystems involves the transmittal of data between SPADOC 4 and CSSR. The space surveillance data generated by SPADOC 4 must go through the CSSR communications system for distribution to ITW/AA system users. SPADOC 4 needs to transmit data at 30 frames per second; however, CSSR can process only 3 frames per second. During tests, this incompatibility has resulted in data either being lost or taking an unacceptably long time to be transmitted. The Air Force anticipates that the solution to this problem will require an upgrade to the CSSR computers and disks that will not be fully implemented until 1996.\6 Also, the Air Force has identified compatibility problems between CMU subsystems and the already deployed systems and sensor sites that constitute the remainder of the ITW/AA system. For example, each sensor site independently sends a message to the Cheyenne Mountain Complex when it identifies a potential threat. When more than one sensor identifies a threat, more than one message is sent to CMU. This serves to increase confidence in the veracity of the threat. However, under certain circumstances the SPADOC 4 and CCPDS-R subsystems may interpret multiple incoming messages as being extraneous duplicates and discard all but one, thus losing important information. Other integration problems exist as well, and, in general, the Air Force recognizes that it has underestimated the difficulty of integrating CMU with existing ITW/AA systems. -------------------- \5 Attack Warning: NORAD's Communications System Segment Replacement Program Should Be Reassessed (GAO/IMTEC-89-1, November 30, 1988) and Attack Warning: Lack of System Architecture Contributes to Major Development Problems (GAO/IMTEC-92-52, June 11, 1992). \6 This problem was first reported by us in Attack Warning: NORAD's Communications System Segment Replacement Program Should Be Reassessed (GAO/IMTEC-89-1, November 30, 1988). At that time we noted that several of the CMU subsystems were being sized to process different amounts of data, and we concluded that these inconsistent design parameters could adversely affect communications among CMU subsystems. AIR FORCE HAS MADE IMPROVEMENTS IN PROGRAM MANAGEMENT ------------------------------------------------------------ Letter :6 Two significant management improvements have occurred in the CMU program since we concluded our previous reviews. In 1989, we reported that the acquisition effort for ITW/AA systems was characterized by a cumbersome structure, divided responsibility, poor management continuity, and deferred problem resolution.\7 These problems contributed to the delivery of subsystems that did not meet specifications and could not be effectively integrated without additional, costly changes. Four years later, in 1993, the Air Force began taking steps to address these problems. The first improvement is that the Air Force is no longer developing the CMU program as individual subsystems. Instead, the program is now managed in four blocks that correspond to the system's major missions: two for missile warning, one for air warning, and one for space surveillance. The block concept focuses on ensuring that all of the subsystems within each block can work together effectively to warn, for example, of a missile launch. This increases the possibility that development, testing, and management will be better coordinated throughout the life cycles of the subsystems. Second, the Air Force has made the CMU program director responsible for CMU during its acquisition and development phases as well as during the subsequent operations and maintenance phases. Having one manager responsible across all phases of the system's life cycle will likely improve the coordination and cooperation among the many offices involved in development, testing, and maintenance. -------------------- \7 Attack Warning: Better Management Required to Resolve NORAD Integration Deficiencies (GAO/IMTEC-89-26, July 7, 1989). FURTHER PROGRAM MANAGEMENT CHANGES ARE BEING CONSIDERED ---------------------------------------------------------- Letter :6.1 In response to the recent determination that CMU will not meet its approved costs and scheduled milestones, the Assistant Secretary of the Air Force for Acquisition chartered a task force of Air Force personnel, known as the Red Team, to review the current CMU program approach. The Red Team reviewed CMU requirements, operations, testing, contractor performance, configuration management, and integration. They then made specific recommendations to improve these activities. Subsequently, the Air Force developed a revised CMU management plan that would consolidate the implementation of fixes and enhancements to CMU subsystems and extend the milestones for integration and testing of the overall CMU system through December 1998. The revised CMU management plan defines a core program of basic capabilities to be achieved in November 1995. After that, three additional phases are planned through December 1998. Each phase includes a single annual delivery of hardware and software for CMU subsystems as well as testing of both individual subsystems and the integrated "blocks" of subsystems that correspond to warning missions. The last phase includes a test of the end-to-end CMU system, which, if successful, will signal the completion of the program and the old systems will be decommissioned. At that point, additional development work is to be undertaken on an incremental, "evolutionary" basis, through a single annual software and hardware upgrade based on priorities set by the users. GAO OBSERVATIONS ------------------------------------------------------------ Letter :7 As CMU development enters its 13th year, schedules continue to slip and costs continue to rise. Serious subsystem development and integration problems continue to slow CMU deployment. The continuing absence of an overall architecture that fully describes system and subsystem requirements, which we reported on previously, increases the risks that CMU will not meet original systems requirements nor be capable of evolving to meet the needs of new missions in the future. The Air Force has already taken some steps to improve program management and is now working on a revised management plan that will provide more time for testing and integration of CMU subsystems. The new plan will require additional funding and a further delay, now projected by the Air Force to be at least $104 million and 3 years. We will evaluate these changes in future work and assess the likelihood that they will effectively address CMU's many complex problems. ---------------------------------------------------------- Letter :7.1 As requested, we did not obtain comments on a draft of this report from the Department of Defense. However, we discussed the information contained in it with Defense officials, including the Vice Commander, Air Force Space Command; the Commander, Cheyenne Mountain Complex; the System Program Director, Electronic Systems Center; and the Office of the Secretary of Defense (Acquisition) at the Pentagon. These officials generally concurred with the facts presented. We are providing copies of this report to the Secretary of Defense; the Secretary of the Air Force; the Director, Office of Management and Budget; and other interested parties. We will also make copies available to others upon request. This work was conducted under the direction of David O. Nellemann, Director for Information Resources Management/National Security and International Affairs, who can be reached at (202) 512-6240. Other major contributors are listed in appendix II. Sincerely yours, Gene L. Dodaro Assistant Comptroller General OBJECTIVES, SCOPE, AND METHODOLOGY =========================================================== Appendix I On March 22, 1994, the Chairman, Subcommittee on Defense, House Appropriations Committee, requested that GAO ascertain the status of the computer upgrades at Cheyenne Mountain. We agreed to examine (1) the current status of the cost and schedule for the CMU program, (2) the status and results of developmental and operational testing of CMU subsystems, (3) the status of the integration of the systems, and (4) management issues affecting future CMU development. We performed work at Air Force Space Command, U.S. Space Command, the Air Force Operational Test and Evaluation Center, the Space Warning and Systems Center, the Site Activation Team of the Electronic Systems Center, and Air Force Materiel Command's Detachment 25 at Peterson Air Force Base in Colorado Springs, Colorado. We also interviewed officials at the Electronic Systems Center at Hanscom Air Force Base in Bedford, Massachusetts. As the basis for this work, we obtained mission status briefings from the six U.S. Air Force Space Command subsystem leads, as well as briefings from six other Air Force and U.S. Space Command organizations involved in the development, test, evaluation, and maintenance of CMU. We reviewed and analyzed test result reports, budget documents, and subsystem position papers. We obtained briefings and additional information from the Red Team assigned by the Assistant Secretary for Acquisition to develop ways to mitigate CMU's projected cost and schedule increases. We also met with officials at Air Force and Department of Defense headquarters at the Pentagon. We discussed the information in this report with appropriate Defense program officials, including the Vice Commander, Air Force Space Command; the Commander, Cheyenne Mountain Complex; the System Program Director, Electronic Systems Center; and the Office of the Secretary of Defense (Acquisition) at the Pentagon. MAJOR CONTRIBUTORS TO THIS REPORT ========================================================== Appendix II ACCOUNTING AND INFORMATION MANAGEMENT DIVISION, WASHINGTON, D.C. -------------------------------------------------------- Appendix II:1 John A. de Ferrari, Assistant Director Keith A. Rhodes, Technical Assistant Director David A. Powner, Evaluator-in-Charge Elizabeth L. Johnston, Assignment Manager DENVER REGIONAL OFFICE -------------------------------------------------------- Appendix II:2 Jamelyn A. Smith, Regional Assignment Manager RELATED GAO PRODUCTS ============================================================ Chapter 0 Granite Sentry (GAO/IMTEC-92-84R, September 21, 1992). Attack Warning: Status of the Survivable Communications Integration System (GAO/IMTEC-92-61BR, July 9, 1992). Attack Warning: Lack of System Architecture Contributes to Major Development Problems (GAO/IMTEC-92-52, June 11, 1992). Computer Technology: Air Attack Warning System Cannot Process All Radar Track Data (GAO/IMTEC-91-15, May 13, 1991). Attack Warning: Costs to Modernize NORAD's Computer System Significantly Understated (GAO/IMTEC-91-23, April 10, 1991). Defense Acquisition: Air Force Prematurely Recommends ADP Acquisitions (GAO/IMTEC-90-7, March 29, 1990). Attack Warning: Defense Acquisition Board Should Address NORAD's Computer Deficiencies (GAO/IMTEC-89-74, September 13, 1989). Attack Warning: Better Management Required to Resolve NORAD Integration Deficiencies (GAO/IMTEC-89-26, July 7, 1989). Space Defense: Management and Technical Problems Delay Operations Center Acquisition (GAO/IMTEC-89-18, April 20, 1989). Attack Warning: NORAD's Communications System Segment Replacement Program Should Be Reassessed (GAO/IMTEC-89-1, November 30, 1988).