Execute the Security Test

Learning Objectives

After completing this unit, you’ll be able to:

Describe a security testing and evaluation (ST&E) specialist's role in executing test cases.
List four main types of security testing.

Execute Test Cases

Now it’s time to execute test cases to provide data that supports analysis, evaluation, and reporting regarding the system’s security. This is where your team implements the security acceptance criteria that you came up with during planning. These criteria can include security mitigations, bug fixes, or feature changes you identify as you create your secure product or service.

To do this, you create and execute test cases. Test cases help answer these questions:

Has the business analyst interpreted the security requirements correctly?
Has the development team translated the business requirements to functional requirements and to code correctly?
Has the development team ensured that an adversary can’t misuse the system to compromise the security of its data?

Answering these questions during test execution makes you the hero of the system building process.

An ST&E specialist dressed like a superhero tests a system using a magnifying glass with a check mark inside to inspect a gear.

In addition to building test code to validate your use cases, you should also build test cases to validate “abuse cases.” Building an abuse test case requires that you define ways an attacker could compromise your application. You build test cases to simulate that attack and confirm that there are controls in place to mitigate the attack.

After you generate the test case and the test data, you execute the test cases. During this step, you package the code written by the development team into an installable piece of software, and deploy it to the test environment.

You use test cases to mimic high-use and high-risk functional activities. You assemble sequences of test scripts into a suite of tests, execute the test scripts, correct any gaps in the scripts, and report any defects.

To help you, you can use test automation tools to plan and coordinate test execution across multiple systems. These tools are especially useful because usually the same tests need to be run again after every fix and deployment. Examples of automation tools include LambdaTest, TestComplete, and QMetry Automation Studio.

Test execution is the process of executing the code and comparing the expected and actual results. It is the real-time validation of the product’s security features and finding bugs. To test, evaluate, and verify systems, and determine their compliance with defined security specifications and requirements, follow these steps:

Assign the test cases in each test suite to testers for executing.
Execute tests, and collect and analyze data.
Record all discrepancies or unexpected results for investigation and resolution.
Report security bugs.
Resolve blocking issues as they arise.
Re-test.
Complete regression test to ensure fixes you made did not introduce new vulnerabilities.
Manage test assets, test resources, and test personnel to ensure effective completion of test events.
Monitor ST&E processes and recommend changes when they are warranted.

Types of ST&E

Let’s follow along with Anya, an ST&E specialist at a research institute. She is testing a new system that holds sensitive data about the institute’s cutting-edge research on technology policy. She wants to test the system to verify whether it:

Prevents the usage of untrusted removable media (such as personal USB drives) that could exfiltrate sensitive data or introduce malicious software.
Contains any unpatched software flaws that could allow an adversary to make unauthorized changes to the system’s data.
Can be accessed by previously compromised credentials available on the dark web from a previous hack of the organization’s usernames and passwords.

To fully test and evaluate the security of the system, Anya decides to perform four main types of ST&E:

Developmental testing and evaluation (DT&E)
Operational testing and evaluation (OT&E)
Interoperability testing and evaluation (Interoperability T&E)
Information assurance testing and evaluation (IA T&E)

Anya knows that integrating the objectives of these tests into a single test strategy maximizes efficiencies during test execution while minimizing test resource requirements. Let’s take a closer look at what each entails.

Developmental Testing and Evaluation (DT&E)

Anya performs DT&E to test the system’s security while under development. She evaluates design risk, identifies design alternatives, compares and analyzes trade-offs, and estimates satisfaction of operational security requirements. For example, what is the risk of not implementing a technical control to block untrusted removable media? Will implementing such a technical control prevent authorized users from exfiltrating data under a legitimate use case? Instead, could the system scan untrusted removable media for malicious software before allowing it access to the system?

By considering topics like this, Anya uses developmental testing to answer four critical questions:

What security objectives is she trying to meet?
How is she going to meet them?
Where is she now?
How secure are the current systems now?

In performing DT&E, Anya can identify technical capabilities and limitations of alternative concepts and design options under consideration.

Anya also stresses the system to ensure secure design. For example, she focuses on reviewing the system’s code, and tests whether she is able to brute force access to the system by guessing common username and password combinations, or use credentials available on the dark web to access it. By doing so, she demonstrates the system’s performance against threats and the effectiveness of countermeasures, as identified in the system threat assessment. Any impact on technical performance by these threats should be identified early in DT&E, rather than later in OT&E where their presence may have serious repercussions.

Anya also assesses progress toward meeting critical security parameters. For example, her employer has a standard in place that no system is allowed to go live with any publicly facing critical vulnerabilities. She scans the system to determine whether security patches of these vulnerabilities have been implemented successfully.

Anya uses developmental test activities where appropriate prior to conducting full-up testing in realistic environments. This means that she conducts DT&E in a controlled environment. She uses DT&E to provide data and analysis to support the decision to certify the system is safe and ready for OT&E.

Operational Testing and Evaluation (OT&E)

With OT&E, Anya focuses on questions of operational security. She performs OT&E in a realistic environment with operational scenarios that include typical users and logistics support.

For example, she may implement a test case where a user’s password for the system has expired; they are unable to reset it and must contact the help desk. She observes the process the help desk follows to verify the user’s identity and reset the password. She observes that the information the help desk uses includes knowledge-based questions that are easily answered through a Google search (such as the user’s full name, birthdate, and address). Anya notes that an attacker could use this publicly available information to reset a legitimate user’s password and gain unauthorized access to the system. She gives this feedback to the development team to help them enable a different design or technical control to guard against this vulnerability. For example, the help desk could verify the user's identity instead by pushing an authentication request to an application on their phone, that they must verify before proceeding.

While the focus of developmental and operational testing may be different, they have a common objective: ensuring that the user obtains systems that are secure.

Interoperability Testing and Evaluation (Interoperability T&E)

Anya uses Interoperability T&E to test integrated architecture products. She ensures the test conditions reflect an applicable capability environment. For example, the development team may decide to implement multi-factor authentication (MFA) to the system to ensure that usernames and passwords found on the dark web can’t be used by an adversary to access sensitive data. Instead, for system access, they require users to use both username and password, and to accept an authentication request sent to their mobile phone via an authenticator application. Anya tests whether the authenticator application is properly integrated with the target system, and whether an adversary can intercept the authentication request in any way and gain unauthorized access to the system.

Information Assurance Testing and Evaluation (IA T&E)

Finally, Anya performs IA T&E to ensure the system meets certain statutory, regulatory, and contractual requirements for information security. This can include testing to certify compliance with standards such as:

This testing can also be used to better understand the maturity of the system’s security capabilities, using frameworks such as the NIST Cybersecurity Framework or the Building Security in Maturity Model (BSIMM) as a guide. With IA T&E, Anya uses a mix of operational and laboratory environments.

For example, Anya’s employer has a contract with the federal government to receive data about technology policy decisions. That data is subject to NIST 800-171, which contains a requirement to limit system access to the transactions and functions that authorized users are permitted to execute. For example, the system may be required to limit the ability of regular users to exfiltrate data onto a personal USB drive. Anya executes test cases to ensure a technical control is in place to block this action.

Execute the Security Test

Learning Objectives

Execute Test Cases

Types of ST&E

Resources