Peter Christensen, Director, National Cyber Range, The National Cyber Range: A System Engineering Resource for Cybersecurity R&D, S&T, Testing and Training , October 27 and 28, 2015. Unclassified.
National Security Archive
Christensen’s 2015 briefing turned the National Cyber Range into a cornerstone of DoD acquisition, linking new cyber policies to a joint, realistic test environment.
Source: Peter Christensen, Director, National Cyber Range, The National Cyber Range: A System Engineering Resource for Cybersecurity R&D, S&T, Testing and Training , October 27 and 28, 2015. Unclassified. Date: Oct 27, 2015 Archive: Defense Technical Information Center .
Editorial Analysis
Original analysis by the DriftSeas editorial desk. The complete primary-source document, transcribed from the National Security Archive scan, appears in full below.
A Test Bed for the New Battlefield
In October 2015 Pete Christensen, then director of the National Cyber Range (NCR), delivered a briefing to the 18th Annual Systems Engineering Conference that reads less like a product pitch and more like a diagnostic of a military caught mid‑transition. The slide deck, now declassified, was prepared for senior engineers and acquisition officers who were wrestling with a sudden influx of policy memos—DoDI 5000.02, DoDI 8510.01, the revised DoD 8500.01—and a stark DOT&E annual report that found “significant vulnerabilities on nearly every acquisition program” that had undergone cybersecurity operational test and evaluation (OT&E) in FY‑14. Christensen’s talk was a direct response to those findings: the DoD needed a realistic, repeatable environment where cyber‑enabled weapons systems could be probed with the same rigor afforded to missiles or aircraft.
The document’s opening frames the NCR as a “systems engineering resource” – a phrase that signals a shift from ad‑hoc lab setups to an integrated, acquisition‑lifecycle tool. By positioning the range under the Test Resource Management Center (TRMC), the briefing underscores a bureaucratic hand‑off from DARPA’s experimental research (2009‑12) to a service‑wide operational mandate in 2012. This transition is crucial: DARPA built the technical scaffolding, but only the DoD could embed the range into the formal testing, training, and experimentation pipelines that govern weapon system certification.
From Policy to Practice
Christensen’s slides map the burgeoning policy landscape onto concrete capabilities. The revision of DoDI 5000.02 (Jan 2015) introduced “new/better guidance for both developmental and operational testing of IT,” while DoDI 8510.01 (Mar 2014) codified the Risk Management Framework (RMF) across all DoD information systems. The briefing repeatedly stresses that “cyberspace threats are proliferating” and that the DoD’s own assessments—sixteen cyber‑risk exercises in 2014 alone—still left at least one mission per exercise “high risk” against even beginner‑level adversaries.
What the slides reveal between the lines is a growing anxiety that existing test methods were too abstract. The “six‑phase cyber‑security T&E process” slated for August 2015, and the accompanying guidebooks (July 2015 Cybersecurity T&E Guidebook, draft Cybersecurity Implementation Guidebook), were attempts to institutionalize a “shift‑left” approach: embed security testing early, iterate through materiel development stages, and only then hand off to OT&E. The NCR was presented as the technical linchpin for that process, offering a “real‑world, quantifiable assessment” of cyber‑defensive and offensive capabilities across the acquisition timeline.
The Actors and Their Stakes
Beyond Christensen, the briefing cites two other senior voices: J. Michael Gilmore, Director of Operational Test and Evaluation, whose FY‑14 report bluntly equated cyber threats with missiles, artillery, aviation, and electronic warfare; and the Office of the Secretary of Defense, which issued a memorandum mandating that all oversight systems capable of sending or receiving digital information undergo cybersecurity testing. These citations are not decorative; they signal a top‑down mandate that forces every program office—whether developing a new radar, a communications suite, or a logistics platform—to allocate budget and schedule to cyber‑range events.
The document also hints at inter‑service dynamics. By listing the Joint Test and Evaluation Command, Air Force OT&E Center, Marine Corps OT&E Activity, and Joint Interoperability Test Command as memorandum recipients, Christensen underscores that the NCR is meant to be a joint resource, not a siloed Army or Navy asset. This jointness is essential for the range’s credibility: a test environment that can emulate heterogeneous networks, cloud services, and legacy legacy platforms is only valuable if every service can trust its results.
Why the NCR Still Matters
The briefing’s legacy lies in how it framed cyber‑testing as a core acquisition function rather than an afterthought. The “six‑phase” model, now embedded in the Defense Acquisition Guidebook, still references the NCR as the preferred venue for “adversarial cybersecurity DT&E” and “vulnerability and penetration assessment.” Moreover, the range’s DARPA heritage ensures it retains cutting‑edge emulation capabilities—software‑defined networks, high‑fidelity threat injectors, and scalable cloud‑based environments—that commercial testbeds cannot match in terms of classified data handling.
In the years since 2015, the NCR has been invoked in major joint exercises (e.g., Cyber Flag, Cyber Quest) and in the testing of hypersonic weapons control software, confirming Christensen’s prediction that cyber threats would be treated with the same rigor as kinetic threats. The document thus serves as a snapshot of a pivotal moment when the DoD institutionalized cyber‑range testing, laying groundwork for today’s integrated cyber‑acquisition ecosystem.
TEST RESOURCE MANAGEMENT CENTER DEPARTMENT OF DEFENSE NATIONAL CYBER RANGE Test Resource Management Center The National Cyber Range: A Systems Engineering Resource for Cybersecurity R&D, S&T, Testing and Training Prepared for the 18th Annual Systems Engineering Conference October 27 and 28, 2015 Presented by: Pete Christensen, Director, National Cyber Range peter.h.Christensen.civ@mail.mil 571-372-2699
TEST RESOURCE MANAGEMENT CENTER DEPARTMENT OF DEFENSE
What, Why, How?
NATIONAL CYBER RANGE Test Resource Management Center
- What do we want to accomplish?
- Provide an overview of T&E Policy and Guidance
- Provide an overview of the National Cyber Range (NCR)
- Discuss how programs and organizations can benefit from using the NCR
- Why is this important?
- Cyberspace Threats are proliferating
- Systems Security Engineering (SSE) and Risk Management Framework (RMF)
- Recent policies are emphasizing the importance of increased realism in cybersecurity testing and training
- TRMC and the NCR can help!
- How will we do it?
- Cover some existing DoD cybersecurity guidance and policies
- Explain some of the history behind the NCR
- Provide an overview of NCR technical capabilities
- Discuss what you can do with the NCR and types of events that it supports
- Describe NCR event planning and how customers can get engaged
2
New/Ongoing Cybersecurity Policy and Guidance Activities
- Revision of DoDI 5000.02: Issued 6 Jan 2015
- New/better guidance for both developmental and operational testing of IT
- Revision of DoD 8500.01, Cybersecurity: 14 Mar 2014
- Expanded scope and specificity
- DoDI 8510.01 – Risk Management Framework (RMF) for DoD IT: 14 Mar 2014
- Provides policy, clarity and guidance on the RMF and compliance
- Six Phase Cybersecurity T&E Process: Planned Aug 2015 Incorporated into Defense Acquisition Guidebook Chapter 9
- OSD DOT&E- Procedures for Operational Test and Evaluation of Cybersecurity in Acquisition Programs: 01 Aug 2014
- Formalizes OT&E Phases
- Cybersecurity Implementation Guidebook for PMs: In Draft
- Address Cybersecurity T&E across the acquisition lifecycle
- Cybersecurity T&E Guidebook: Issued July 2015
- Address Cybersecurity T&E across the acquisition lifecycle
Department of Defense DoD Program Manager's Guidebook for Integrating the Cybersecurity Risk Management Framework (RMF) into the System Acquisition Lifecycle
3
DoD Cybersecurity Test Posture and Emerging Requirements
- "Also in 2014, my office conducted 16 cybersecurity assessments in conjunction with Combatant Command and Service exercises...Despite the improved defenses, my office found that at least one assessed mission during each exercise was at high risk to cyber-attack from beginner to intermediate cyber adversaries."
- "DOT&E found significant vulnerabilities on nearly every acquisition program that underwent cybersecurity OT&E in FY14."
- "The cyber threat has become as real a threat to U.S. military forces as the missile, artillery, aviation, and electronic warfare threats which have been represented in operational testing for decades."
- "Operational Test Agencies (OTAs) will include cyber threats among the threats to be encountered in operational testing for DOT &E oversight systems with the same rigor as other threats."
- "All oversight systems capable of sending or receiving digital information are required to conduct cybersecurity testing."
Director, Operational Test and Evaluation FY 2014 Annual Report January 2015 This report satisfies the provisions of Title 10, United States Code, Section 139. The report summarizes the operational test and evaluation activities (including live fire testing activities) of the Department of Defense during the preceding fiscal year. [Signature] J. Michael Gilmore Director
OFFICE OF THE SECRETARY OF DEFENSE MEMORANDUM FOR COMMANDER, JOINT TEST AND EVALUATION COMMAND COMMANDER, AIR FORCE OPERATIONAL TEST AND EVALUATION CENTER COMMANDER, OPERATIONAL TEST AND EVALUATION FORCE DIRECTOR, MARINE CORPS OPERATIONAL TEST AND EVALUATION ACTIVITY COMMANDER, JOINT INTEROPERABILITY TEST COMMAND SUBJECT: Procedures for Operational Test and Evaluation of Cybersecurity in Acquisition Programs The cyber threat has become as real a threat to U.S. military forces as the missile, artillery, aviation, and electronic warfare threats which have been represented in operational testing for decades... Operational Test Agencies (OTAs) will include cyber threats among the threats to be encountered in operational testing for DOT&E oversight systems with the same rigor as other threats. All oversight systems capable of sending or receiving digital information are required to conduct cybersecurity testing.
4
Cybersecurity T&E “Shift Left” – Six Phased Process
MDD Materiel Solution Analysis AOA DRAFT CDD MS A Technology Maturation & Risk Reduction Req Decision Pre-EMD MS B CDD Engineering & Manufacturing Development IATT CDR TRR DT&E Event SVR MS C ATO CPD Production and Deployment Full Rate Production Decision Review OTRR IOT&E Operations and Sustainment ASR SRR SFR PDR DT&E Assessment DT&E Assessment
T&E Phases Understand Cybersecurity Requirements Characterize Cyber Attack Surface Cooperative Vulnerability Identification Adversarial Cybersecurity DT&E Vulnerability and Penetration Assessment Adversarial Assessment O&S
Pre MS A/B Requirements and Systems Security Engineering Analysis
SE/DT&E Evaluate Software and Systems Security Architecture
RMF/DT&E Verify Baseline Cybersecurity Requirements and Vulnerability Assessment
DT&E/OT&E Evaluate Mission Capabilities and Interoperability in a Contested Environment
Training & Exercises Evaluate TTPs in a Contested Environment
OT Focus – Codified in OSD DOT&E Memo dated 01 Aug 2014
5
National Cyber Range – Background
- Originally developed by Defense Advanced Research Projects Agency (DARPA) in the 2009-2012 timeframe
- Transitioned from DARPA to the DoD Test Resources Management Center (TRMC) in October 2012
- TRMC was charged with "operationalizing" the capabilities for use by the DOD test, training, and experimentation communities
National Cyber Range Adaptable, multi-dimensional, heterogeneous cyber test environment The Nation's environment for cyber research
The National Cyber Range is the measurement capability providing a realistic quantifiable assessment of the Nation's cyber research and development technologies, enabling a revolution in national cyber capabilities and accelerate transition of these technologies
The National Cyber Range will allow classified and unclassified researchers to measure their progress... ... in either a classified or unclassified environment, ... against appropriate threats with sufficient timeliness and accuracy, ... to allow corrections and needed new capabilities to be determined.
Leap-ahead research and quantifiable assessment of cyber tools, processes and architectures facilitates; Revolution in national cyber capabilities Rapid technology development Accelerated deployment
Providing the environment to solve the Nation's Cyber problems Unconstrained cyber research environment supporting the CNCI
What is the National Cyber Range? A dedicated cyber testbed to enhance the Nation's ability to defend against cyber attacks
A cyber test center to: Enable leap-ahead advances to defend and exploit the cyber realm Enable revolutionary cyber testing
The National Cyber Range will Provide a dedicated "test bed" to produce qualitative and quantitative assessments of the security of cyber technologies and expansion Provide a revolutionary, safe, instrumented environment for our national cyber security research organizations to test the security of information systems. Revolutionize the state of the art of cyber security testing.
Revolutionary test technologies Automated configuration, sanitization, reconfiguration - Automation Virtualization technology - Scale Simulate human activity - Realism Time dilation & contraction - Efficiency All systems: wired, MANET, control systems, phone, etc. - Completeness Facilitates concurrent, realistic, credible testing
JTASS Aegis Radar Systems Business Networks
6
TEST RESOURCE MANAGEMENT CENTER
DEPARTMENT OF DEFENSE
NATIONAL CYBER RANGE
Test Resource Management Center
# What is a Cyber Range?
**Traditional “Ranges”**
* Physical Environment for:
* Weapon Testing
* Live Training
* TTP Development, ...
* Range Assets Change slowly
**Cyber Range**
* Place to Evaluate:
* Effectiveness of Cyber Defenses
* Effectiveness of Cyber Weapons
* Train Cyber Warfighters
* Rehearse Mission
* TTP Development
* Range Assets Change Rapidly
NCR provides a range solution that can span the entire spectrum of cyber test, evaluation & training needs
7
TEST RESOURCE MANAGEMENT CENTER DEPARTMENT OF DEFENSE DASD(DT&E) / Director, TRMC NATIONAL CYBER RANGE Test Resource Management Center
USD(AT&L) HON Frank Kendall
ASD(R&E) Mr. Stephen Welby (Acting)
DASD(DT&E) / Director, TRMC Dr. C. David Brown
Staff Director Col Bohenek, USAF Chief of Staff Vacant
Principal Deputy, DT&E Dr. Brian Hall (SES) (Acting
Principal Deputy Director, TRMC Mr. Derrick Hinton (SES)
Deputy Director, T&E Competency & Development Tom Simms
Deputy Director, Air Warfare Mike Ginter
Deputy Director, Land and Expeditionary Warfare Steven Lopes
Deputy Director, Naval Warfare Patrick Clancy
Deputy Director, Cyber and Information Systems Andrew Pahutski
Deputy Director, Space and Missile Defense Systems Darlene Mosser- Kerner
NCR is here! Director, National Cyber Range Pete Christensen
Deputy Director, Corporate Operations Sheila Wright
Deputy Director, T&E Range Oversight (MRTFB) Bruce Bailey
Deputy Director, Test Capabilities Development (CTEIP) Gerry Christeson
Deputy Director, Cyber and Interoperability (JMETC) Chip Ferguson
Deputy Director, Technology Development (T&E/S&T) George Rumford
8
NCR – Vision and Mission
- Vision
- Be recognized as the cyberspace test range of choice for providing mission tailored, hi-fidelity cyber environments that enable independent and objective testing and evaluation of advanced cyberspace capabilities
- NCR Mission Statement
- Provide secure facilities, innovative technologies, repeatable processes, and the skilled workforce
- Create hi-fidelity, mission representative cyberspace environments
- Facilitate the integration of the cyberspace T&E infrastructure through partnerships with key stakeholders across DoD, DHS, industry, and academia
9
# BLUF – NCR Key Capabilities
* **Multiple concurrent tests at varying classification levels are supported using a Multiple Independent Levels of Security (MILS) architecture**
* Accredited for testing up to Top Secret / Sensitive Compartmented Information
* Currently support up to 4 events at varying classification concurrently
* **Rapid emulation of complex, operationally representative network environments**
* Can scale up to ~40K high-fidelity virtual nodes
* Red/Blue/Gray support, including specialized systems (e.g., weapon systems)
* **Automation provides significant efficiencies that enable more frequent and more accurate events**
* Reduces timelines from weeks or months to hours or days
* Minimizes human error and allows for greater repeatability
* **Sanitization to restore all exposed systems to a known, clean state**
* Allows assets to be reused even when they are exposed to the most malicious and sophisticated uncharacterized code
* **Supports a diverse user base by accommodating a wide variety of event types (R&D, OT&E, information assurance, compliance, malware analysis, etc.) and communities (testing, training, research, etc.)**
10
TEST RESOURCE MANAGEMENT CENTER DEPARTMENT OF DEFENSE What is the National Cyber Range? NATIONAL CYBER RANGE Test Resource Management Center
Computing Assets/Facility (LMCO Orlando, FL)
Range Operations Center FACTR Wide Situational Awareness FACTR Operations Accreditation Maintenance
Reconfigurable Test Suite 1 2 Operator Rooms 1 Brief/Debrief Conf Room
Welcome and Reception Introductions Visitor Check In
Security Office Security Operations File Storage
High Security Data Center Asset Warehouse MLS Environment
Reconfigurable test Suite 2 2 Operator Rooms 1 Brief/Debrief Room
Range Support Center Software Sustainment Community Outreach Resource Integration
Encapsulation Architecture & Operational Procedures
Cyber Test Team
Integrated Cyber Event Tool Suite
Test Consumer Test Sponsor Test Specification Tool (TST) Asset Descriptions CSTL Test Specification Range Management Tool (RMT) Range Repository Master Resource Manager (MRM) SAN Manager FACTR Operations Team MIL/Other Environments Test Management Verification and Control (TMVC) Test Execution Tool (TET) Range Configuration & Verification Tool (RCVT) Test Participants Event Execution Language (EEL) Sensors Traffic Generation Tools (TG) Data Analysis & Visualization Tools (DAVT) Testbed Sanitization
Secure Connectivity via JIOR and JMETC
Realistic Mission Environments Global Information Grid RSDPs PSDPs JMN Net Enabled Weapon Live Fire Event
11
Facility Overview: On-site or Remote Access
Range Operations Center FACTR Wide Situational Awareness FACTR Operations Accreditation Maintenance
Reconfigurable Test Suite 1 2 Operator Rooms 1 Brief/Debrief Conf Room
Welcome and Reception Introductions Visitor Check In
Security Office Security Operations File Storage
Range Support Center Software Sustainment Community Outreach Resource Integration
Reconfigurable test Suite 2 2 Operator Rooms 1 Brief/Debrief Room
High Security Data Center Asset Warehouse MLS Environment
- Fully accredited SCIF
- Supports at least two independent concurrent events on-site
- Test suites can be utilized at different security levels and contain:
- Two test rooms
- Conference room
- Unclassified Range Support Center
- Wireless Testing Environment
- Remote access currently provided through the Joint IO Range (JIOR) and JMETC MLS
12
TEST RESOURCE MANAGEMENT CENTER DEPARTMENT OF DEFENSE Facility Overview: Support for Wireless Testing NATIONAL CYBER RANGE Test Resource Management Center
Range Operations Center FACTR Wide Situational Awareness FACTR Operations Accreditation Maintenance
Reconfigurable Test Suite 1 2 Operator Rooms 1 Brief/Debrief Conf Room
Welcome and Reception Introductions Visitor Check In
Security Office Security Operations File Storage
Range Support Center Software Sustainment Community Outreach Resource Integration
High Security Data Center Asset Warehouse MLS Environment
Reconfigurable test Suite 2 2 Operator Rooms 1 Brief/Debrief Room
- Wireless environment that supports classified testing (TS/SCI)
- Support for mobile computing: iOS, Android, Windows 8 on tablets, cell phones, and multimedia devices
12:45 f cisco
13
TEST RESOURCE MANAGEMENT CENTER
DEPARTMENT OF DEFENSE
NATIONAL CYBER RANGE
Test Resource Management Center
# Automation Toolkit: End to End Support
Test Consumer
Test Scientist
Test Specification Tool (TST)
Asset Descriptions
CSTL Test Specification
Planning Role
Tools to support event planning
FACTR Operations Team
Range Management Tool (RMT)
Range Repository
Master Resource Manager (MRM)
SAN Manager
MLSEnclave
Tools to define and manage resource requirements
Test Participants
Test Management Verification and Control (TMVC)
Test Execution Tools (TET)
Range Configuration & Verification Tool (RCVT)
Event Execution Language (EEL)
Sensors
Traffic Generation Tools (TG)
Data Analysis & Visualization Tools (DAVT)
Testbed Sanitization
Testbed
Tools to automatically:
* Build, verify and sanitize your environment
* Support event execution
Faster, more reliable, event environment creation and execution
14
NCR Automated Cyber Test Process Start with a common pool of HW/SW Resources and Cyber Tool Set Running a Cyber/Test Evaluation
Step 1: Utilize Test Spec Tool to define end to end aspects of test Define Test
Step 2: Resource Allocation determines what resources from the pool are needed and allocates them to Event Allocate Resources
Step 3: Range Provisioning Tools automatically wire HW to the appropriate configuration Configure the HW
Step 4: Range Configuration (ACORN) tools automatically configure the SW you need to run the event Configure the SW
Step 5: Test Execution Tools are used by the event team along with event-specific systems for execution and data collection/analysis Run Test
Step 6: Sanitization Tool sanitizes HW and "virtually" puts HW resources back in pool Sanitize Resources
15
Cybersecurity T&E "As A Service"
ONE NCR TEAM
TRMC Government FFRDCs Lockheed Martin SETA Contractors
- Services Include, But Are Not Limited To:
- End-to-End Test Support
- Test Bed Design Support
- Cyber and Testing Expertise
- Threat Vector Development
- Custom Traffic Generation
- Custom Sensor and Visualization Support
- Custom Data Analysis
- Integration of Custom Assets
- Software
- Hardware
- Wired and Wireless
- Remote Red/Blue Team Support
The NCR's Most Valuable Resource Is A Diverse and Experienced World Class Cybersecurity Workforce
16
TEST RESOURCE MANAGEMENT CENTER DEPARTMENT OF DEFENSE
Why Use a Cyber Range?
NATIONAL CYBER RANGE Test Resource Management Center
- Requirements to conduct testing that cannot or should not occur on open operational networks due to potential catastrophic consequences, for example full execution of extremely malicious threats on realistic representations of systems and networks (e.g., releasing self-propagating malware)
- Requirements to test advanced cyberspace tactics, techniques, and procedures that require isolated environments of complex networked systems (e.g., movement on the Internet)
- The need to rapidly and realistically represent operational environments at different levels of security, fidelity, and/or scale (e.g., Blue [friendly] force, Red [adversary] force, and Gray [neutral] networks)
- The need for precise control of the test environment that allows for rapid reconstitution to a baseline checkpoint, reconfiguration, and repeat of complex test cases; this would include the need for rapid variation of conditions to quickly evaluate hundreds of scenarios
17
When To Use a Cyber Range?
Across the Acquisition Life Cycle
TEST RESOURCE MANAGEMENT CENTER DEPARTMENT OF DEFENSE NATIONAL CYBER RANGE Test Resource Management Center
MS A Req Decision MS B Pre-EMD IATT MS C ATO Full Rate Production Decision Review
MDD Materiel Solution Analysis AOA DRAFT CDD Technology Maturation & Risk Reduction CDD Engineering & Manufacturing Development CPD Production and Deployment Operations and Sustainment
ASR SRR SFR PDR CDR TRR DT&E Event SVR OTRR IOT&E
DT&E Assessment DT&E Assessment
T&E Phases Understand Cybersecurity Requirements Characterize Cyber Attack Surface Cooperative Vulnerability Identification Adversarial Cybersecurity DT&E Vulnerability and Penetration Assessment Adversarial Assessment O&S
Pre MS A/B Requirements and Systems Security Engineering Analysis NCR Event Cybersecurity Architecture Evaluation
SE/DT&E Evaluate Software and Systems Security Architecture NCR Event Cybersecurity Verification and Validation
RMF/DT&E Verify Baseline Cybersecurity Requirements and Vulnerability Assessment NCR Event Mission Thread Testing with Blue Team
DT&E/OT&E Evaluate Mission Capabilities and Interoperability in a Contested Environment NCR Event Mission Thread Testing with Red Team in a Realistic Threat Environment
Training & Exercises Evaluate TTPs in a Contested Environment NCR Event Large-scale Simulation to Train Cyber Mission Forces and Evaluate Cyber Defensive and Offensive Operations
18
TEST RESOURCE MANAGEMENT CENTER
DEPARTMENT OF DEFENSE
NATIONAL CYBER RANGE
Test Resource Management Center
# What You Can Do With the NCR (1 of 4)
**Question: Does Product “A” close a requirements gap?**
- Does it mitigate a particular set of threats within my operational system?
- How well?
- What is my residual risk?
**What you get:**
- Empirical evidence showing how the technology or product closes the requirements gap in your operational environment
PCAP Analyzer
Span Switch
Corporate Switch
Internet Router
Simulated Internet
Left Enclave
Outer Router
FW
DMZ
Inner Router
Switch
MS Exchange Server
Print Server
MS Domain Controller
AGM x 600
HBSS
Right Enclave
Outer Router
Switch
Win 7 x 400
Test Room Assets
MS Exchange Server
Print Server
MS Domain Controller
HBSS
**How does adding a technology to my existing environment reduce my threat surface?**
Commercial Product / Emerging Technology Evaluation
19
What You Can Do With the NCR (2 of 4)
Question: Will my architecture scale in the field? – Will it handle the expected user load? – What are potential issues that can only be discovered at scale (normally only found very late in the testing process)
What you get: – Minimize unexpected performance failures late in the DT or early OT process – Reduce costly rework – Empirical data to show whether or not the system operates as predicted in a realistic environment
Ground Site Control Station Boundary Defenses Network Tap RT Laptop Emulated Terrestrial Network Packet Capture Emulated Secure Network Secure Network Emulatio DNS RT Laptop Aircraft Simulation Boundary Defenses
Will this architecture scale to support the mission?
Results provide insight into system performance before the design is finalized
20
What You Can Do With the NCR (3 of 4)
Question: How resilient is my system to cyber attacks and faults when connected into the overall system of systems?
– System is a distributed sensing system that has a dependency on an external service to interconnect platforms to ground stations – How does my system behave when there are problems with external systems?
What you get:
– Increased resilience to cyber attack and failures – Reduce costly rework – Empirical data to show whether or not the system operates as predicted in a realistic environment – Understand how the dependencies on the broader DoD environment affect the ability to meet the mission
System Testing During Development
Graphic Source: http://fm.cnbc.com/applications/cnbc.com/resources/img/editorial/2013/02/14/100460031-server-room-cyber-security-gettyp.1910x1000.jpg
21
What You Can Do With the NCR (4 of 4)
Question : How do I generate realistic cyber mission effect within a large scale training exercise safely and securely?
- OCO is destructive
- Cyber weapons and TTPs are often classified at security levels higher than the rest of the exercise
What you get:
- Realistic operator training
- Repeatability to evaluate relative effectiveness of multiple TTPs
- On-demand, low-cost evolution of the environment to represent salient real-world environments
Be able to use unrestricted TTPs
Operate on realistic and complex network topologies
Integrate home base and remote training
Have access to interactive web sites
A safe environment for safely conducting realistic cybersecurity training
Graphic Source: http://www.npr.org/2014/04/30/307963996/whats-the-nsa-doing-now-training-more-cyber-warriors
22
# NCR Supports Many Different Types of Events
* **NCR supports a wide variety of cyber event types**
* R&D testing
* Product evaluation
* Training events
* System emulation
* Target emulation
* Mission rehearsal
* Risk reduction activities
* Architecture analysis
* DT&E
* OT&E
* Malware analysis
* Forensic analysis
* **Events can occur exclusively at NCR, or in conjunction with other Joint Mission Environment Test Capability or Joint Information Operations Range nodes**
* **Level of support from NCR is dependent on customer needs**
23
NCR Operational Support Models
We work with consumer to define tests and then NCR personnel do everything else with periodic review.
We deliver a verified range and support sanitization at end & consumer does everything else.
Plan Test Construct & Verify Testbed Execute Test Closeout
Plan Test Construct & Verify Testbed Execute Test Closeout
Range Staff Consumer
Minimal Consumer Participation Minimal Range Staff Participation
You Select the Desired Level of Support from NCR Staff
24
NCR Planning and Scheduling Procedures
- NCR Director:
- Coordinates with the JMETC PM to review schedules and make decisions
- Owns the NCR Event Planning List and the NCR Range Schedule
- The NCR Event Planning List describes the events that are currently in the discussion/planning phase and scheduled but not yet run
- NCR Range Schedule describes the events to be held on the range
- Monthly Review held to:
- Formally add/move events to the schedule
- Review customer feedback on tests
- Review Event Planning Port
Initial Contact Preliminary Discussions Detailed Discussions Detailed Event Planning Test Development & Configuration Test Execution
25
NCR Event Planning Stages
- Event Pre-Planning & Planning
- Discussions
- Use Case Development
- Event Design
- Goals, Objectives & Assumptions
- Outputs & Data Collection Plan
- Environment Design
- Event Development
- Red Team Operations
- Environment Build & verification
- Event Execution
- Conduct tests and data
- Review results & adapt as needed
- Event Completion
- Data Analysis
- Reporting, Briefings, Next Event Planning
Event Execution Test Specification Briefs Final Report Event Support Plan Data collection Red Team Logs
Test & Verification TRR Event Checklist Full unclassified Verification Daily Range Verification Component/System Integrations Full Classified Verification
Development Acquire any HW/SW New Emulations Adapt Reuse Continuous Integration Full Unclassified Development Configure with classified Data Red Team Vectors/Attack Plan
Define Implementation HW/SW Requirements Define Reuse leverage Define new component effort GFE Requirement from end Customer New Capability Reuse strategy Schedule
Event Vision Domain Discussions Use Case Development Event Focus Development
Information Briefs NCR Briefs JMETC Synergy/Contacts NCR Overview Day Conferences Papers
Example Generalized from Actual NCR Event
26
TEST RESOURCE MANAGEMENT CENTER DEPARTMENT OF DEFENSE NATIONAL CYBER RANGE Test Resource Management Center
How to get engaged
Start/Finish Contact Pete Christensen, TRMC
TRMC Test Director Assignment
Technical Interchange Meetings Technical Scope Definition Resource Planning Identify New Development NCR provides SME support, automated tools, libraries
Event Planned on Master Test Schedule
NCR Environment Development Testbed Construction Integration with Remote Assets on JMETC or JIOR
Execute Event
Data Storage or Purge / Asset Sanitization
Event Report
Customer Survey
27
Summary
- Cyberspace threats to DoD systems are proliferating at an unprecedented rate
- Leadership has recognized that current cybersecurity testing and training needs further improvements
- Leadership is placing increased emphasis on the need to consistently incorporated realistic cybersecurity testing and training at all levels and phases
- Early identification of system vulnerabilities can make them easier and cheaper to fix
- NCR provides customers with a unique set of cybersecurity test, evaluation, and training capabilities
- NCR enables acquisition organizations to conduct system specific cybersecurity test and evaluation events that are tailored to meet program requirements throughout the systems acquisition lifecycle
- NCR enables operational organizations to conduct realistic cybersecurity training in environments that closely replicate the real world
- NCR capabilities have been independently validated and have successfully supported a wide variety of cyber events including
- Developmental Testing
- Operational Testing
- Training/Exercise
- NCR is institutionally funded and cost effective
- Customers only pay for their own personnel, travel, systems under test, special equipment, etc.
28
TEST RESOURCE MANAGEMENT CENTER DEPARTMENT OF DEFENSE
NATIONAL CYBER RANGE Test Resource Management Center
Questions?
Peter H. Christensen Director, National Cyber Range TRMC Office Phone: 571-372-2699 TRMC Email: peter.h.christensen.civ@mail.mil
Address: 4800 Mark Center Drive Suite 07J22 Alexandria, Va. 22350
NATIONAL SECURITY ARCHIVE
National Security Archive, Suite 701, Gelman Library, The George Washington University, 2130 H Street, NW, Washington, D.C., 20037, Phone: 202/994-7000, Fax: 202/994-7005, nsarchiv@gwu.edu
Keywords
Sources & References
- [1]Peter Christensen, Director, National Cyber Range, The National Cyber Range: A System Engineering Resource for Cybersecurity R&D, S&T, Testing and Training , October 27 and 28, 2015. Unclassified.
- [2]http://fm.cnbc.com/applications/cnbc.com/resources/img/editorial/2013/02/14/100460031-server-room-cyber-security-gettyp.1910x1000.jpg