Category: Risk Management

Understanding the Top 10 Security Risks Associated with Large Language Models (LLMs)

Introduction

Large Language Models (LLMs) have revolutionized the field of artificial intelligence and natural language processing, but with great power comes great responsibility. As LLMs become increasingly prevalent, it’s essential to understand the potential security risks they pose.

In light of OWASP’s recent announcement of the OWASP Top 10 Risk for Large Language Model Applications, this article aims to explore my perspective on the top 10 security risks associated with LLMs. I am eager to compare and contrast these risks with the ones OWASP will publish.

Cybersec.Cafè Top 10 Large Language Models Security Risks

1. Jailbreaking: Bypassing the security measures of an LLM to gain unauthorized control and exploit it for malicious purposes.
2. Prompt injection: Crafting prompts to influence the model’s output, which can lead to biased, offensive, or harmful text generation.
3. Second-order injections: Advanced prompt injection techniques, where the prompt itself is generated by an LLM, making it harder to detect and prevent attacks. Note: I’m not considering cross-content injections (a type of prompt injection where the prompt is generated in one context and then used to generate text in another context – this can be used to generate text that is relevant to the first context but harmful in the second context) as I consider still as in between of both risk 2 and 3
4. Data poisoning: Injecting malicious data into the training dataset, resulting in biased or harmful outputs. Rigorous validation and monitoring are crucial to mitigate this risk. This is actually a Machine Learning (ML) risk that extend to LLMs being that their training is ML based.
5. Misinformation: Unintentional contribution to the spread of misinformation or support for creating misinformation campaigns.
6. Malicious content generation: Misusing LLMs to generate persuasive or believable text for phishing or social engineering attacks.
7. Weaponization: Misusing LLMs to support coding malware or potentially even for malware detection evasion (still a theoretical threat) by generating malware code that evades traditional endpoint detection and response scanners. For example, an LLM could be used to generate malware code that is not detected by traditional Endpoint Detection and Response scanners as the code is generated by an LLM that provides it via API.
8. LLM-delivered attacks: Using LLMs to deceive users and obtain sensitive information or launch cyber attacks. For example, an LLM could be used to ask a user for sensitive information such as their passwords or credit card number.
9. Abuse of vertical LLM APIs: Exploiting LLMs for purposes outside their intended use cases, potentially undermining the intended business model.
10. Privacy: LLMs are trained on massive datasets that contain also personal information, raising privacy concerns if the models generate text like the confidential data it was trained from. This happens for instance with Inference Attacks or Model Inversion Attacks these attacks attempt to infer or recreate information about the training data from the outputs of an ML model.

Some other thoughts

• Organizations have their specific risks that are not necessarily the same of my Top 10.
• OpenAI’s LLMs are not currently subject to any restrictions on their use. This means that it might be more complex to mitigate any risk at all.
• Of course also the ML risks are part of the LLM ones

Conclusion

While the risks associated with LLMs may seem challenging, we don’t know yet if they are insurmountable. As of today, we still lack comprehensive solutions to mitigate most of these risks compared to other security domains like applications and mobile devices. Additionally, due to the “black box” nature of LLMs, understanding their inner workings presents challenges in determining the appropriate security measures to adopt. Furthermore, regulatory frameworks surrounding LLM use are still evolving, as discussed in my geopolitical analysis of the ChatGPT block in Italy.

LLM security contains a multitude of unknown unknowns, and it necessitates further research and mitigation strategies to effectively safeguard against these risks. Awareness serves as the critical first step towards achieving effective cybersecurity if it will be ever possible to reach it.

Recommended Readings

To delve deeper into the topic, I recommend reading the following insightful resources:

• https://www.wired.com/story/chatgpt-jailbreak-generative-ai-hacking/
• https://themathcompany.com/blog/data-poisoning-and-its-impact-on-the-ai-ecosystem
• https://spectrum.ieee.org/ai-cybersecurity-data-poisoning
• https://www.semianalysis.com/p/google-we-have-no-moat-and-neither
• https://ambcrypto.com/heres-how-to-jailbreak-chatgpt-with-the-top-4-methods-5/
• https://www.techopedia.com/what-is-jailbreaking-in-ai-models-like-chatgpt
• https://www.theregister.com/2023/04/26/simon_willison_prompt_injection/
• https://blogs.itemis.com/en/model-attacks-exploits-and-vulnerabilities
• https://research.nccgroup.com/2022/12/05/exploring-prompt-injection-attacks/
• https://hiddenlayer.com/research/the-dark-side-of-large-language-models/
• https://hiddenlayer.com/research/the-dark-side-of-large-language-models-2/
• https://embracethered.com/blog/posts/2023/ai-injections-direct-and-indirect-prompt-injection-basics/
• https://embracethered.com/blog/posts/2023/ai-injections-threats-context-matters/
• https://www.mufeedvh.com/llm-security/

The Interplay of Risk, Investment, and… Luck in Cybersecurity

Last weekend, I came across a LinkedIn post illustrating how numerous companies were breached despite having SOC2, ISO 27001, and PCI-DSS certifications. This observation prompted me to reflect.

Initially, my thought was that there isn’t a direct correlation. The data set is rather small and doesn’t account for all the certified companies that have avoided breaches. Furthermore, certification is a form of assurance that some level of security is in place, signaling to potential attackers that there is valuable data worth protecting.

In the cybersecurity realm, we frequently emphasize robust defense mechanisms, proactive risk assessments, and constant vigilance. Today, however, I want to navigate less charted territory: “security-by-luck”.

What do you mean with Security-by-Luck?

My definition of “Security-by-luck” would be the situation where a company, despite having weak or inadequate security measures, remains unbreached due to factors outside its control, such as the attackers’ choices, capabilities, or sheer chance.

To clarify, I’m not endorsing this as a strategic approach – that would be reckless. Rather, I aim to highlight a crucial facet of cybersecurity – the constant interplay of risk, investment, and a dose of luck.

In a previous article, I discussed on the challenge of defining ‘how much security is enough’. No matter how much an organization invests in security, the threat of an attack persists. Conversely, not all lightly-defended organizations will suffer breaches, too lightly defended (even if those that are inadequately defended become low-hanging fruit for cybercriminals). However, over-investment in security isn’t the solution either, as organizations have other business objectives to meet. So, the question arises, where do we draw the line?

I’m not suggesting that companies should stop investing in cybersecurity and merely hope for the best. Instead, I want to stress the importance of making calculated risks.

To illustrate this, consider four hypothetical companies, each investing differently in cybersecurity…

The contenders:

• Company A: Does the bare minimum for security (e.g., has an antivirus installed)
• Company B: Complies with statutory requirements and uses common sense
• Company C: Adheres to a cybersecurity standard and has obtained certification (like SOC 2, ISO 27001, PCI-DSS, HITRUST, etc.)
• Company D: Follows all major best practices and has adopted bleeding-edge security solutions

Each of these companies, regardless of their investment level, can either be breached or remain secure. Here’s how:

Vulnerabilities-based Attacks:

• A vulnerability in their system gets exploited – Company A gets breached.
• Company B, which patches vulnerabilities quarterly, gets breached when an attacker exploits a flaw within the time window before it gets patched.
• Even Company C, which patches vulnerabilities monthly, gets hacked, as the attackers were quicker on their feet.
• Company D has no known unpatched vulnerabilities (a near impossibility in real life, but let’s go with it). However, there’s a zero-day vulnerability that they aren’t aware of (I know this is the definition of zero day). An attacker discovers and exploits it – Company D gets breached.

Let’s assume, for a moment, that all these companies understand this risk and decide to have all vulnerabilities patched (again a near impossibility) and are lucky there aren’t any unexploited zero-day vulnerabilities. You might think they’re safe. But what if an attacker targets their people instead?

People-based Attacks:

• An attacker successfully executes a phishing attack on Company A, leading to a breach.
• Despite having good email security and having conducted a phishing simulation last year, Company B falls prey to a successful social engineering attack.
• Company C suffers a sophisticated MFA fatigue attack and gets breached.
• In Company D, an attacker bribes an employee to gain access to the system (including credentials and MFA, as seen in the Lapsus$ attacks last year).

Even if the organization decide to invest in a solid cyber culture and luckily their employees are equipped with strong ethics to resist such attempts, are the potential threats truly over?

Unfortunately, no, the threats aren’t over. They are susceptible to…

Supply Chain Attacks:

The attack surface extends to vendors, giving birth to a new cycle of vulnerabilities and people-based attacks. Hence, even Company D could harbor cybersecurity points of failure within their supply chain.

Luck is Not a Strategy

In essence, cybersecurity isn’t merely about investment levels; it’s also about the complex interplay of factors that contribute to a company’s overall risk profile. Even the most secure organization cannot completely rule out the possibility of a breach. Given the dynamic nature of the landscape, absolute security is a virtual impossibility, making a small element of ‘luck’ an undeniable part of the equation.

Regrettably, many companies have relied solely on this ‘luck’ factor for so long that they’ve now become easy targets.

‘Security-by-Luck’ should not be a strategy in itself, but understanding its role in the broader cybersecurity framework is essential. The goal should always be to optimize investment, maintain a robust defense mechanism, foster employee awareness, and devise sound strategies to mitigate potential risks, including supply chain risks. This involves striking a balance, understanding that no solution offers 100% protection, and ensuring readiness to respond effectively (by having incident response plans and exercises conducted) if or when a breach occurs by conducting regular incident response plans and exercises.

Conclusion

In conclusion, while we can’t depend entirely on luck, or as the Cybersecurity community usually call it, the residual-risk, acknowledging its existence, could make us more attuned to the realities of the ever-evolving cybersecurity landscape. The presence of residual risk is an undeniable part of cybersecurity, and acknowledging without relying on it might encourage a more realistic approach towards cybersecurity strategy and implementation.