A Comprehensive approach proposal
Greetings, readers! Welcome back to our exploration of LLM (Large Language Models) security risks. In my previous posts (here and here), I discussed the significance of understanding these risks. That’s why I am excited to share my participation in the creation of the OWASP Top 10 Risk for Large Language Model Applications 😊.
In this article, we will delve into the challenges involved in defining an approach to create the Top 10 LLM security risk list and propose a holistic approach to address them.
The Challenges in Defining a Top 10 LLM Security Risk List
As we embark on this endeavor, we encounter several challenges that need to be overcome:
- Evolving Landscape: LLMs are rapidly evolving, with new models (including Open ones with no restrictions) and attack techniques emerging. Keeping the evaluation comprehensive to address emerging risks is challenging but necessary.
- Complexity and Interdependencies: LLMs involve various components, including training data, algorithms, infrastructure, and user interactions. Understanding their interdependencies and how risks propagate across them requires careful analysis. Some components are already covered by other Top 10s but they might be so relevant that we might want to include them
- Lack of Standardization: Inconsistencies in terminology and definitions related to LLM security risks can lead to inconsistencies in risk assessment and mitigation. Establishing standardized language and frameworks is vital and luckily OWASP will help a lot in this. A couple of examples below:
- I had a discussion about Intellectual Property Theft. I wrongly assumed that we were speaking only the theft of the LLM model itself, but if we think about it there are other king of IP theft, e.g., the weights are intellectual property, or if some users provide IP to the LLM, the LLM will learn from that and might provide the IP to the next users. As I said I didn’t consider those as for me those were privacy risks… but these are also ML risks
- We had discussions on how we should call the “hallucination” risk (e.g., is this term humanizing LLMs? Shuldn’t something as “Confabulation” be better? Maybe, but hallucination is already LLM Jargon).
- Multidimensional Risks: LLM risks encompass technical, ethical, legal, and societal aspects. Incorporating these perspectives and achieving a holistic understanding is essential.
- Risk Prioritization: Determining the significance of each risk and prioritizing them within the Top 10 list is complex. Professional judgment and a thorough assessment are needed.
- Balance of Granularity: Striking the right balance between granularity and practicality is crucial. The Top 10 list should be concise, understandable, and actionable, while capturing the breadth and depth of LLM security risks.
Addressing the Challenges with TARA
“Necessity makes the method” used to say one of my old bosses, and to tackle these challenges, I propose adopting a TARA (Threat Analysis and Risk Assessment) method, which involves identifying potential threats, analyzing their likelihood and impact, and evaluating associated risks.
First Step: Threat Modeling
We start conducting a comprehensive threat modelling exercise, defining threat categories specific to LLMs and documenting potential threats within each category.
Below you will find my proposal of threat list, it is not supposed to be 100% correct, just to give an idea on how it would look like. To do so I used OWASP v0.1, Adam AI centered Top 10 some of the Cybersec risks and ML risks from this super insightful article.
| Category | Threats | Sub-Threat |
| LLM-specific | Prompt Injection | Direct Prompt Injection Second Order Injection Cross-content injections |
| Machine Learning | Training-Time Attacks | Training Data Poisoning Byzantine attacks |
| Decision-Time Attacks | Inference | |
| Evasion Attacks | ??? | |
| Oracle Attacks | Extraction Inversion Membership Inference | |
| Model Theft | Model Theft Surrogate Model | |
| Statistical Attack Vectors | Bias Drift | |
| Model Hijacking Attacks | Backdoors Trojanized models | |
| User specific | Overreliance on LLM-generated Content | Hallucination Bias Inexplicability |
| Operational | ??? | Inadequate AI Alignment |
| Application / Infrastructure | Insecure development | Inadequate Sandboxing Improper Error Handling |
| Insecure deployment | Unauthorized Code Execution SSRF Vulnerabilities Insufficient Access Controls | |
| Personal Data / Intellectual Property | ??? | Data Leakage IP Theft |
To be more accurate, this exercise leans more towards threat identification rather than threat modelling.
Please note that I’m not sure where all the sub-threats should be. For instance an ML threat might be the root cause of the existence of some User specific or Personal Data/IP threats…
The following TARA Steps
The next steps would be:
- Risk Evaluation: Estimate the likelihood and impact of each identified threat, considering various perspectives and dimensions. Combine these factors to calculate the overall risk level associated with each threat.
- Risk Prioritization: Prioritize risks based on their significance and impact, using professional judgment and a holistic perspective to choose the Top 10.
- Mitigation Strategies: Define appropriate mitigation and prevention strategies to address the identified risks effectively.
Those phases are all straightforward, the only difficult part could be understanding the impact. What angle do we need to consider? For an organization of course many of those threats could result in data breaches, financial losses, reputational damage, legal implications, etc. What if we consider a non-enterprise end-user? And the LLM owner? E.g., the latter would be the only one that wants to avoid model theft…
Conclusion
LLMs are at the forefront of technological advancement, and understanding their risks is paramount for secure adoption. By adopting a comprehensive approach like TARA, we can identify, assess, and mitigate these risks more effectively.
Collaboration, standardization, and a multidisciplinary perspective are key to success in this endeavor. Let’s work together to create a safer LLM landscape and pave the way for responsible and secure deployment.
Join me for future articles as we explore LLM security risks and discuss practical mitigation strategies.
Recent Comments