Cybersecurity Comms: A/B Testing Cuts Phishing by 18% in

Effective cybersecurity communication is a moving target, especially when phishing attacks and social engineering tactics evolve daily. Organizations must ensure their security awareness training resonates, but how do you measure that resonance beyond completion rates? We recently executed an A/B testing campaign designed to precisely quantify the effectiveness of different communication styles in reducing susceptibility to phishing, and the results were illuminating.

Campaign Overview: The “Phishing Defender” Initiative

Our “Phishing Defender” campaign, launched in Q3 2025, aimed to significantly lower the internal click-through rate (CTR) on simulated phishing emails and increase employee reporting of suspicious communications. The client, a mid-sized financial services firm with approximately 1,500 employees across three main offices in Atlanta, Perimeter Center, and Alpharetta, faced a growing challenge with sophisticated spear-phishing attempts. Their existing cybersecurity training, while compliant, lacked measurable impact on real-world behavior.

The total budget allocated for this specific A/B testing phase was $15,000 over a six-week duration. This covered creative development for two distinct communication tracks, licensing for a specialized phishing simulation platform (KnowBe4, for example, offers strong simulation tools), and analytics dashboard access. Our primary metric for success was a reduction in the simulated phishing CTR, with secondary goals including an increase in reported suspicious emails via their internal reporting tool.

Strategy: A Tale of Two Tones

We hypothesized that the tone and framing of cybersecurity alerts and training refreshers would dramatically influence employee engagement and, consequently, their protective behaviors. We designed two distinct communication tracks:

1. Track A: Formal & Procedural. This approach used standard corporate language, emphasizing policy compliance and the abstract risks of cyber threats. Communications were drafted with a neutral, authoritative tone, focusing on “adhering to security protocols” and “maintaining organizational integrity.”
2. Track B: Urgent & Personal. This track adopted a more direct, almost alarmist tone, personalizing the threat (“Your data is at risk,” “You are the first line of defense”) and highlighting immediate, tangible consequences for individuals and the company. Language focused on “critical vulnerabilities” and “immediate action required.”

Both tracks included a series of three email alerts, two intranet announcements, and one short video module over the six-week period. The core content (e.g., how to identify a suspicious link) remained consistent across both tracks. Only the presentation and tone varied. This was a critical distinction, ensuring we isolated the variable we intended to test.

Creative Approach: Visuals and Verbiage

For Track A, visuals were clean, corporate stock images typically depicting abstract digital security concepts or lock icons. The email subject lines were straightforward: “Security Protocol Update,” “Important Security Notice.” The intranet posts were designed with standard company branding. An example email subject line was “Review of Updated Phishing Guidelines.”

Track B employed more impactful, sometimes slightly unsettling, visuals: blurred images of login screens, warning triangles, or stylized representations of data breaches. Subject lines were designed to grab immediate attention: “URGENT: Your Account Security at Risk,” “WARNING: New Phishing Threat Detected.” The video module for Track B featured a brief, dramatic narrative about a real-world data breach and its impact on individuals, using an actor speaking directly to the viewer. This was a conscious decision to move beyond generic warnings and elicit a more visceral response.

The simulated phishing emails themselves were identical for both groups, launched two weeks after the initial communication burst. These simulations mimicked common tactics observed in the financial sector, such as fake HR updates, urgent password reset requests, and seemingly legitimate vendor invoices. We used five different phishing templates to ensure a broad test of susceptibility.

Targeting and Segmentation

The client’s 1,500 employees were randomly split into two groups of 750 individuals each. This randomization was important to minimize pre-existing biases between the groups. We ensured an even distribution of departments (e.g., IT, HR, Sales, Operations) and seniority levels within each group. This level of segmentation allowed us to confidently attribute differences in behavior to our communication variations rather than inherent differences in employee roles or technical savviness.

Emails were delivered via the company’s internal communication system, which allowed for precise tracking of open rates and click-throughs. Intranet announcements were targeted to specific employee groups based on their login credentials. The video modules were hosted on a secure internal learning management system (Foundation OnDemand, for instance, provides these capabilities) with completion tracking.

What Worked and What Didn’t: Data Analysis

The results were stark. The Urgent & Personal (Track B) approach significantly outperformed the Formal & Procedural (Track A) strategy in every key metric.

The most compelling finding was the 18% reduction in simulated phishing CTR for employees exposed to Track B communications. This translated to 17 fewer clicks on potentially malicious links within that group of 750 employees during the simulation period. While this might seem small, in a real-world scenario, those 17 clicks could represent 17 potential breaches, making the impact substantial.

The higher open rates for Track B’s communications (79.5% vs. 68.2%) indicated that the more urgent subject lines and personalized framing successfully captured attention. Plus, the significant increase in reported phishing attempts (188 vs. 125) in Track B suggested a heightened sense of vigilance and a clearer understanding of the reporting mechanism. This is a critical behavioral shift. Simply not clicking is one thing, but actively reporting suspicious emails strengthens the overall security posture.

What didn’t work as well? For Track A, the formal tone often led to communications being perceived as “just another corporate email,” easily overlooked. Anecdotal feedback from a small post-campaign survey indicated that many employees in Track A felt the messages were generic and lacked direct relevance to their daily work. Some even confessed to skimming the content without truly absorbing the warnings. This highlights a common pitfall: compliance-driven communication often fails to drive behavioral change.

Optimization Steps Taken

Based on these clear results, we recommended a complete overhaul of the client’s internal cybersecurity communication strategy, adopting the principles of Track B. Specific optimization steps included:

1. Standardizing Urgent Messaging: All future security alerts, from password change reminders to new threat warnings, now use the “Urgent & Personal” tone. This includes subject lines, email body content, and intranet headlines.
2. Enhanced Visual Cues: Implementing a consistent visual language for security communications that immediately signals urgency and importance, moving away from generic stock photos. This involves specific icons and color palettes.
3. Mandatory Interactive Modules: Replacing passive video viewing with short, interactive modules that require user input and demonstrate understanding. We also integrated micro-learning modules (3-5 minutes) that focus on single, actionable security tips, delivered weekly.
4. Gamification of Reporting: Introducing a leaderboard for departments based on the number of legitimate suspicious emails reported, fostering a sense of healthy competition and encouraging active participation in security.
5. Regular, Shorter Simulations: Instead of infrequent, lengthy simulations, the client now conducts shorter, more frequent (monthly) simulated phishing campaigns, using varied templates to keep employees on their toes.

The cost per simulated phishing click (our proxy for a “negative conversion”) was significantly lower for Track B. For Track A, the cost per click was approximately $15,000 / (750 0.128) = $156.25. For Track B, it was $15,000 / (750 0.105) = $190.48. This seemingly counterintuitive result (higher cost per click for the more effective track) simply reflects that fewer negative actions occurred. The true measure of success here is the reduction in the sheer number of clicks, not the cost associated with each one, as clicks represent failures. The ROAS (Return on Ad Spend, or in this case, Return on Awareness Spend) is harder to quantify directly in dollars for cybersecurity, but the avoided cost of a single breach (which can run into millions for a financial firm, according to reports like IBM’s Cost of a Data Breach Report) makes the $15,000 investment negligible.

This campaign underscored a fundamental truth in communication: people react to what they perceive as directly relevant and immediately impactful. Abstract corporate directives often fall flat. When it comes to cybersecurity, making the threat personal and the defense actionable is not merely a preference. It’s a necessity for fostering a truly resilient human firewall.