Control Effectiveness: a Capture-the-Flag Study

As cybersecurity breaches continue to increase in number and cost, and the demand for cyber-insurance rises, the ability to reason accurately about an organisation’s residual risk is of paramount importance. Security controls are integral to risk practice and decision-making: organisations deploy controls in order to reduce their risk exposure, and cyber-insurance companies provide coverage to these organisations based on their cybersecurity posture. Therefore, in order to reason about an organisation’s residual risk, it is critical to possess an accurate understanding of the controls organisations have in place and of the influence that these controls have on the likelihood that organisations will be harmed by a cyber-incident. Supporting evidence, however, for the effectiveness of controls is often lacking. With the aim of enriching internal threat data, in this article we explore a practical exercise in the form of a capture-the-flag (CTF) study. We experimented with a set of security controls and invited four professional penetration testers to solve the challenges. The results indicate that CTFs are a viable path for enriching threat intelligence and examining security controls, enabling us to begin to theorise about the relative effectiveness of certain risk controls on the face of threats, and to provide some recommendations for strengthening the cybersecurity posture.