Max Heinemeyer, the chief product officer at the U.K.-based cybersecurity firm Darktrace, says that looking at the one-month period since ChatGPT attained 1 million users in early December, there has been little change in the total number of attempted cyberattacks targeting Darktrace customers. But Darktrace has seen a distinct shift in the tactics used by cybercriminals.

Malicious links in phishing emails declined from 22% of cases to just 14%, Heinemeyer says. But the average linguistic complexity of the phishing emails encountered by Darktrace jumped 17%.

The company’s working theory: Cybercriminals are starting to use ChatGPT to craft much more convincing phishing emails—ones that are so good that cybercriminals don’t even need to rely on embedding malware in attachments or links. After all, malicious links or embedded malware can often be detected and stopped by cybersecurity software such as Darktrace’s.

What’s much harder to stop are attacks that rely completely on old-fashioned deception, or “social engineering.” An email that is so convincingly written that the recipient believes it’s from a trusted source is a great to way pull off an authorized push payments fraud, for example. The victim is fooled into sending funds to pay for what they think is a legitimate transaction or invoice, but is in fact sending the money straight to a fraudster’s account.

In some cases, Heinemeyer says, criminals may be setting the stage for longer cons that involve winning the victims’ trust over a period of time and might involve sophisticated impersonations of real executives or customers.

A gift for non-native English speaking hackers

In addition to A.I. writing tools such as ChatGPT, other new generative A.I. tools could be used to abet such scams. A.I. software, such as that from nascent startup Eleven Labs, can now create realistic voice clones after having been trained on recordings of a target’s voice that might only be a few seconds long. Meanwhile, text-to-image generation software, such as Stable Diffusion, can create increasingly realistic deepfakes with a fraction of the training data previously required for other deepfake methods.

Frauds based on compromised business emails have been on the rise for the past four to five years, Evan Reiser, the founder and CEO of cybersecurity Abnormal says. And while he says that his company has not yet seen any increase in these kinds of attacks since ChatGPT debuted, he thinks it is possible criminals, especially those whose native language is not English, may be tempted to use the tool to craft emails that are less likely to raise red flags with potential victims due to ungrammatical or uncolloquial expressions. “Any tool that is perceived by humans as authentic will make [fraud] worse,” Reiser says.

He says this is especially true of systems where they are explicitly trained to produce text in a particular style, synthesized voices, or images with the intent of fooling people. But he also says that often the simplest tricks—just a very short email that seems to come from a trusted person—works well enough and that criminals generally gravitate towards whatever methods are simplest and require the least effort. “You can send silly, stupid emails and make millions of dollars,” he says. “Why go through the trouble and effort to train [A.I.] models to do that.”

In the wake of the release of ChatGPT, some cybersecurity firms raised the alarm that the A.I. might make it fiendishly easy to pull off a cyberattack. Maya Horowitz, the vice president of research at cybersecurity firm Checkpoint, says that her team was able to get ChatGPT to generate every stage of a cyberattack, starting with a better-than-average phishing email, but then carrying on to actually writing the software code for a malware attack and being able to figure out how to embed that code into an innocuous-looking email attachment. Horowitz said she feared ChatGPT and other generative language models would lead to many more cyberattacks.  

But the same kind of large language models that power ChatGPT can also be used to help cybersecurity companies defend against attacks. Abnormal uses some language models, such as Google’s BERT language model, to help determine what the intent of an email is. If an email is aksing a person to pay for something and putting that person under time pressure, saying it is urgent, or needs to be done ASAP, then that could be a red flag, Reiser says. Language models can also read attachments and see if they match the form and style of previous invoices—or if the invoicing company is one that business has interacted with before. It can even see if the account numbers seem to match ones that have been used previously. (Abnormal even analyses things such as whether an email attachment has fonts that match those previously seen from that company and looks at the meta data of documents for potential signals that something fishy is going on, Reiser says.)

Much of what Abnormal does though is look at patterns across a huge number of features and use machine learning models to figure out if they rise to the threshold where the email should be blocked and a company’s security team alerted. There’s almost always something that will give away a phishing attempt if you know where to look, Reiser says. Even in the case where a legitimate business email account has been compromised, the attacker will often take actions, such as running multiple searches through the account’s history, or using an API to control the account rather than a PC keyboard, that will provide a signal that something isn’t right.

Nicole Eagan, Darktrace’s chief strategy officer, says Darktrace itself has been using the same kind of large language models that underpin to ChatGPT to create more believable spear phishing emails that the compay uses in internal “red teaming” excercises to test its own cybersecurity practices. Eagan says she recently fell for one of these, which was inserted directly into the actual email chain she was having with an outside recruiter Darktrace used.

(Darktrace spent much of the past week trying to prove a different sort of pattern didn’t indicate anything fishy was going on: the company’s share price dropped dramatically after short seller Quintessential Capital Management issued a report claiming it had found evidence that the cybersecurity company might have engaged in dubious accounting practices to try inflate its revenues and profitability ahead of its 2021 initial public offering. Darktrace has denied the accusations in the report, saying that the hedge fund never contacted it before publishing its report and that it has “full confidence” in its accounting practices and the “integrity of our independently audited financial statements.”)