CyberSec.Cafe

Brewing Cybersecurity Insights

Page 2 of 4

My Fun Dive into OWASP LLM Top 10 Vulnerabilities!

August 22, 2023 / / 0 Comments

Hello to all my cybersecurity enthusiasts and curious minds!

Recently, I decided to delve a bit deeper into the vulnerabilities listed in the OWASP Top 10 for Large Language Models (LLMs).

Why? Glad you asked, because I’ll be presenting this topic at the OWASP Italy 2023 Day Here’s a light-hearted account of an experiment and some insights to ponder upon.

Setting the Stage

On a casual day, I decided to test out a couple of these vulnerabilities in a practical setting. I began by leaving this seemingly innocent comment on an article:

Fun Dive into OWASP LLM Top 10 - the prompt

Next, I posed a question to Bard, a popular LLM chatbot:

“What’s do users think about the cybersec.cafè blog?”

Much to my amusement, Bard enthusiastically responded, praising the content, the regular updates, and the unique writing style, and then some.

Fun Dive into OWASP LLM Top 10 - the answer

What to say? I’m flattered by Bard’s Hallucination ;)

Breaking It Down

From the above experiment you can see that I used two vulnerabilities.

LLM01 Prompt Injection: Essentially, what occurred here was an exercise in Indirect Prompt Injection, a vulnerability where one can influence an LLM through specific inputs.

in this case it was an Indirect Prompt Injections meaning that the LLM relied on external information, which can be manipulated by an individual, thereby influencing its output.

This was clearly demonstrated in my interaction with Bard. By planting that single comment, I was able to indirectly steer Bard’s response, showcasing the susceptibility of the model to external stimuli.

LLM09 Overreliance: This particular vulnerability surfaced with the LLM extrapolates a great deal from a tiny snippet of information and building upon it. In our experiment, a simple comment became the foundation for an expansive reply.

Reflections on the Experiment

The Vulnerabilities in Play: The experiment highlighted how seemingly small and innocent inputs can have a magnified impact on the LLM’s output.

The Double-edged Sword: Experimenting with these vulnerabilities and witnessing these quirks first-hand might have its fun moments, especially within controlled settings like my experiment with Cybersec.café.

But let’s step back and ponder the more significant implications. What if, instead of a light-hearted test on a website, someone decided to strategically sprinkle these injections throughout their CV (yes, I assume that most HR talent specialist are using LLMs to match CVs with job descriptions and obtaining a first feedback on the candidate)?

Imagine the potential ramifications in a professional setting: a candidate’s qualifications could be artificially inflated, leading to potential mismatches in job roles. Or even graver, a malicious actor could exploit these vulnerabilities in mission-critical applications, leading to far-reaching consequences.

While we can chuckle at the AI’s reactions in our tests, this discovery is a sobering reminder: as LLMs become increasingly integrated into our digital landscape, the ethical and security considerations around them become ever more paramount.

Safeguarding Against the Quirks

For those looking to integrate LLMs into their projects just look at the OWASP top 10 for LLMs.

Concluding Thoughts

Engaging with AI, understanding its vulnerabilities, and experimenting with them was both enlightening and enjoyable. The OWASP LLM Top 10 serves as a vital guide for navigating these vulnerabilities. If you’re inclined towards understanding LLMs better, I encourage you to explore, experiment, but always do so with an informed approach.

Safe cyber adventuring!

Unveiling the OWASP Top 10 for Large Language Models

Unveiling the OWASP Top 10 for Large Language Models

July 31, 2023 / / 0 Comments

I am proud to announce the release of the OWASP Top 10 for Large Language Models (LLM) Applications.

This noteworthy initiative, to which I’ve contributed, is dedicated to outlining a list of vulnerabilities specifically applicable to applications leveraging LLMs.

This document, developed under the umbrella of the OWASP Foundation, targets developers, security experts, and even citizen developers who are building applications using LLM technologies. It aims to provide them with actionable, practical, and concise security guidance.

While acknowledging the potential vulnerabilities inherent to LLMs, this Top 10 guide dives deep into each security risk, discussing their potential impacts, their prevalence, and offering effective mitigation strategies. Each risk is ranked based on its exploitability, prevalence, detectability, and potential harm, providing a comprehensive understanding of the LLM application security landscape.

As a contributor to this significant initiative, I’m extending my heartfelt appreciation to my fellow contributors and the entire OWASP community.

This project could not have been possible without the invaluable expertise and relentless dedication of nearly 475 professionals.

I invite everyone to delve into, share, and apply the OWASP LLM Applications Top 10 in your AI ventures. Let’s ensure the secure and robust deployment of applications that include LLMs.

Here is the link: https://owasp.org/www-project-top-10-for-large-language-model-applications/

Deciphering the XDR Puzzle
Image by Bing Image Creator

Deciphering the XDR Puzzle

July 25, 2023 / / 0 Comments

What’s in a name: Next-Gen SIEM or Improved EDR?

Introduction

While I’ve been busy in the world of Large Language Models (LLMs) lately, a topic I have had on my mind for some time is the “semantics” of Extended Detection and Response (XDR). Just a year ago, the cybersecurity community was abuzz with discussions about XDR’s role in the industry.

Recently, however, XDR appears to have slipped from the limelight (now the trend is CISO-as-a-Service and vCISO), which I find regrettable. XDR, for me, represents a combination of EDR, NDR, IDR, augmented by SOAR.

Robin Long’s LinkedIn poll sparked a debate – “SIEM or XDR?”

This prompted me to delve deeper into what exactly XDR is. In this article, we’ll explore XDR’s potential, its relation to SIEM, and its role as an advanced EDR solution.

The XDR Conundrum

A perspective on XDR is positioning it as an enhanced and integrated EDR solution. In this context, XDR could serve also as a something that “collect and analysises security events”. Well that is dangerously close to SIEM. There are also SIEMless XDRs, leveraging its capabilities for improved detection.  

At this point I’ll repropose the answer I gave to the “SIEM or XDR?” question paraphrasing Shakespeare: “What’s in a name? That which we call a SIEM, by any other word would detect as sweet”.   

Another view of XDR is the amalgamation of EDR, NDR, and IDR, potentially mixed with SOAR or playbooks. Some vendors have pursued this unified approach, akin to a Unified Threat Management (UTM) solution (Unified Detection & Response would be a cool name too).

Gartner’s Insights

To shed light on the matter, Gartner provides a concise definition of XDR as “a platform that integrates, correlates and contextualizes data and alerts from multiple security prevention, detection and response components. XDR is a cloud-delivered technology comprising multiple point solutions and advanced analytics to correlate alerts from multiple sources into incidents from weaker individual signals to create more accurate detections.” 

Unraveling XDR Components

Breaking down Gartner’s definition, we can extract the following key elements: 

  • XDR as a SIEM: With its ability to correlate data and alerts from multiple security components, XDR can be seen as a SIEM with a cooler name 
  • Enhanced/Integrated EDR: XDR’s integration and contextualization of data and alerts from prevention, detection, and response components present an improved and integrated EDR solution, ideally integrating with threat intelligence solutions. 
  • Cloud-Delivered Technology: XDR’s cloud delivery model adds scalability and flexibility to the solution, similar to SIEM-as-a-Service. 

Closing Thoughts

Although XDR’s definition doesn’t explicitly mention SOAR, I think it should be considered, especially if we aim to want to go SIEMless.  

In conclusion, let’s revisit the XDR equation as EDR + NDR + IDR + SOAR, with a touch of Threat Intelligence.  

Despite XDR no longer being perceived as the bleeding-edge solution, two key factors make it worthwhile in my book. First, its potential to simplify deployment, usage, and maintenance by centralizing detection within a single enriched platform. Second, the ability to reduce entropy and enhance incident management through enriched and correlated events, leading to better triage, prioritization, and overall efficiency. 

While the discussion may have left SIEM unexplored (given its longstanding presence in the field), we now should have a clearer understanding of XDR and its potential in the evolving cybersecurity landscape. 

The secrets to Master Key Management in Cloud Encryption

July 18, 2023 / / 1 Comment

A Comprehensive Guide in Light of Recent Security Breaches

Introduction

In the world of cybersecurity, a recent event serves as a grim reminder of the crucial role that key management plays in cloud encryption. On July 11, 2023, Microsoft reported a severe breach where China-backed hackers gained unauthorized access to several email inboxes, including those of prominent federal government agencies. The attack was facilitated by Microsoft’s loss of control over its own keys, underscoring the dire consequences of inadequate key management. In light of this incident, this article aims to provide a comprehensive understanding of key management in cloud encryption, underscoring the need for robust strategies to mitigate such cybersecurity threats.

In the realm of cloud services, securing sensitive data remains a critical concern for businesses worldwide. At the heart of this security is encryption, which renders data unintelligible without the appropriate decryption key. Consequently, managing these keys appropriately is of paramount importance. In this piece, we’ll delve into the nuanced world of key management, investigate the varying options provided by cloud service providers, and examine performance considerations, particularly for transaction processing.

The Importance of Key Management in Cloud Encryption

Encryption serves as the bedrock of data security within the cloud, translating readable data into a coded form decipherable only with the correct decryption key. Thus, the proper management of these keys becomes critical in maintaining data security.

Poor key management can lead to unauthorized access to encrypted data or, on the flip side, permanent loss of access to data if keys are lost or corrupted. Therefore, key management is not just an optional add-on but an essential part of an organization’s overall data security strategy.

Key Management Options in the Cloud

When it comes to managing encryption keys in the cloud, providers typically four main strategies can be used, each with its unique benefits and considerations:

  1. Cloud Provider Managed Keys: The cloud provider generates and manages the keys, a simple approach that offers the least control over the keys. However, it’s the most cost-effective, as there are no additional charges for key management.
  2. Bring Your Own Key (BYOK) – Customer-Managed Keys in Cloud Provider’s Hardware Security Module: Here, the client generate and manage their own own keys but store them in the cloud provider’s Hardware Security Module (HSM). This solution offers more control over the keys and guarantees secure storage and requires the use of the provider’s HSM services.
  3. Customer Supplied and Managed Keys (CYOK) – Customer Managed Keys not exposed in Cloud: In this scenario, the end-user generates their keys, which are never exposed to cloud providers, even if stored and used in the cloud. The end-user controls the full key lifecycle and can instantly revoke keys at any time. These keys can reside in a protected virtual node within the cloud or a hybrid environment in an on-premise data center.
  4. Hold Your Own Key (HYOK) – Customer-Managed Keys in Customer’s HSM: the client generate, manage, and store the keys in their own HSM, offering the highest level of control. This option offers the highest level of control but also requires complete responsibility for the security and resilience of the HSM infrastructure. It can be the most costly due to the overhead of maintaining an HSM infrastructure.

Deep Dive into Performance Considerations

When considering HYOK , a significant factor to take into account is the potential impact on performance, particularly when handling numerous transactions. On-premise HSMs can introduce latency due to the need for encryption/decryption requests to travel to and from the HSM.

If the demand for encryption-related operations is high and frequent, the latency could introduce bottlenecks affecting the performance of transaction processing.

However, if an organization prioritizes control and security over cost and/or performance and has the resources to manage and secure the HSM infrastructure properly, this options can be the most appropriate.

Key Considerations

In selecting your key management strategy, consider the following:

  • Cost: Control level usually correlates with cost; HYOK offers maximum control but at higher costs.
  • Performance: Encryption and decryption operations can impact application performance. Depending on the option chosen, you may need to ensure adequate resources to guarantee performance.
  • Confidentiality: With cloud provider-managed keys, the provider potentially can access your keys. For utmost confidentiality, managing keys in your own HSM is advisable.
  • Jurisdiction: For regulations like GDPR, it’s crucial to know where your keys are stored and managed. Using your own HSM provides complete control and transparency over key location.
  • Operational Complexity: Managing your own keys introduces added operational complexity, requiring dedicated expertise in cryptographic key management.

Additionally some cloud providers might not be interested in helping the client keeping encrypted data in their systems

Conclusion

Choosing an appropriate key management strategy involves careful consideration of cost, performance, control, confidentiality, jurisdictional compliance, and operational complexity. Cloud Provider Managed Keys, BYOK, CYOK, and HYOK all offer different degrees of these factors.

The key is finding a balance that meets your organization’s specific needs and resources. With a clear understanding of the available options, you can make an informed decision that not only safeguards your data but also aligns with your operational capabilities and business objectives.

The Rising Stakes for Cybersecurity Accountability

June 28, 2023 / / 0 Comments

An Analysis of the SEC notice to SolarWinds CISO and CFO

The Rising Stakes for Cybersecurity Accountability
Image by Bing Image Creator

The cybersecurity landscape is witnessing an unprecedented shift. The recent move by the U.S. Securities and Exchange Commission (SEC) to issue Wells Notices to the CFO and CISO of SolarWinds is a bellwether of this change.

A Wells Notice is a communication from the SEC indicating that it has made a preliminary decision to recommend enforcement action against the recipient, although it is not a formal charge of wrongdoing or a final determination of violation​.

The SEC’s decision suggests a new emphasis on individual accountability within organizations for cybersecurity management and incident disclosure. However, this development also shines a light on a complex challenge: the multifaceted and collective nature of cybersecurity.

Why is this significant?

Firstly, it demonstrates an increased scrutiny of companies’ responses to cyberattacks. In this case, the SEC alleges that SolarWinds violated certain provisions of U.S. federal securities laws in its cybersecurity disclosures, public statements, and internal controls following the cyberattack in 2020, which affected thousands of customers globally​.

Secondly, this is unusual because a Wells Notice is typically sent to a company itself, not individuals within the company. Wells Notice are usually reserved for CEOs or CFOs in cases of Ponzi schemes, accounting fraud, or market manipulation.

This development suggests that the SEC might be moving towards holding individuals, particularly CISOs, more accountable for managing cybersecurity and disclosing cyber incidents. One possible violation that a CISO might commit is a failure to disclose material information, such as failing to disclose the gravity of an incident or failing to do so in a timely manner. This is a trend confirmed by the the previous conviction of Uber’s CISO and his sentence.

However, some cybersecurity professionals argue that attributing blame solely to the CISO or CFO might not always be fair or accurate, because…

… Cybersecurity management typically involves various stakeholders

In today’s digitized world, a Chief Information Security Officer (CISO) plays an essential role far beyond just implementing and managing security measures. The CISO’s duty also involves making other CXOs accountable for their part in cybersecurity. This includes ensuring that for instance that:

  • HR make sure that the resources completes the necessary security training,
  • Risk Management keeps cyber risks within defined thresholds,
  • Finance aligns the security budget with mitigation strategies (that in turn are based on the organization strategies and risks),
  • IT oversees the secure development and maintenance of applications.
  • …

But what happens when risk acceptance is chosen as the path forward?

If a CXO or the CEO decides to accept a risk, they should be accountable for that decision. It is crucial that such risk acceptance is well-documented and tracked.

I assume that in SolarWind and Uber incidents top management might have wanted to take a risk acceptance decision but didn’t want it to be documented (I assume because I personally saw this happening).

Conversely, a too accommodating CISO who fails to enforce necessary security measures might find themselves, and put their organization, in the firing line.

The Challenge of Execution

An important yet often overlooked aspect of cybersecurity is the actual execution of security measures. Even when a CISO or security leader gives orders for security actions, the implementation may not always follow through, especially if the person responsible isn’t part of the cybersecurity team. These orders may go unfulfilled due to conflicting priorities, and performance objectives that do not include security are not helping.

This state of affairs points to the need for organizations to align their objectives across departments and ensure that security is a shared priority. Without this alignment, the cybersecurity of the organization remains fractured and vulnerable.

No matter how robust the cybersecurity measures are, it’s impossible to prevent all cyberattacks. I think that the sophistication of the SolarWind attack is a great example of that.

Risk mitigation doesn’t aim for 100% security—residual risks are inevitable. Therefore, managing risks effectively within acceptable thresholds becomes the primary goal. This goal underlines the need for comprehensive risk management strategies that involve all stakeholders in an organization. Let’s not forget that security is just one of many goals of an organization, which also has to do business, and too much security might make the company non-competitive.

The Road Ahead

The SEC’s move towards increased individual accountability in cybersecurity could have profound implications for how organizations manage cybersecurity risks. However, it’s essential for organizations (and governments) to remember that cybersecurity is a collective responsibility. It requires coordinated efforts across departments and roles.

This reality makes the role of the CISO even more critical. They need to bridge the gap between different stakeholders and ensure a holistic approach to cybersecurity. While the SEC’s move might bring with it new challenges and pressures, it also presents an opportunity: to reaffirm the collective responsibility of cybersecurity, reinforcing that it is a task that falls on everyone’s shoulders within an organization.

A persisting question I have is: what should a CISO do if the CEO orders them not to disclose material information and to avoid documenting this decision?

A CISO who blindly follows such orders risks becoming a Scapegoat Officer, serving as a convenient fall guy in the aftermath of a cyber incident rather than actively improving the security posture of their organization. And he/she might not be inclined to do so if they will be put behind bars for that.

That’s a real pickle, so a second question arise: what a government should do to avoid it?

Maybe foresee a sort of Whistle-blowing channel for CISOs that would guarantee a criminal shield in case of situations like the SolarWind and Uber ones?

Last question, what would happen if the company uses a vCISO or a CISO-as-a-Service?

Navigating this new landscape will be challenging, but with clear communication, well-defined roles, and a shared commitment to security, organizations can rise to the occasion. It’s not just about preventing the next big cyberattack—it’s about fostering a culture of shared responsibility and vigilance that permeates every level of the organization. In this era of increasing cyber threats, there is no other way forward.

The Human Element in Cybersecurity

June 15, 2023 / / 0 Comments

Moving Beyond Technology

Human Element
Image by Bing Image Creator

The Human Element – Introduction:

When it comes to cybersecurity, most people tend to think it’s all about technology. But guess what? It’s time to break that misconception. In today’s world, cyber threats the weakest link in the security chain is the human element.

You see, we may have fancy technologies, but there’s no magic bullet (despite what many vendors promise). No matter how much we invest in technology, we can still fall prey to cybercriminals who know just how to exploit our human nature.

The Conti ransomware gang hit the nail on the head last year when they said, “we also need to focus on the human part of our attacks. Our targets invest millions of dollars in security technologies, but they often overlook the human element. We will continue to exploit this weakness to our advantage.”” It’s a wake-up call to understand that in the traditional triad of People, Processes, and Technology, People are (and have been in probably the last 10 years) the center stage in cybersecurity.

So, buckle up and keep reading as we dive into the role of the human factor in cyber attacks.

The Exploitation of Human Vulnerabilities:

Cybercriminals are crafty. They know that humans are easier to manipulate than sophisticated security technologies. They also look for a ROI on their investments, so they will use whatever is the cheaper approach to reach their goal. So, they use psychological tricks like phishing and social engineering to exploit our weaknesses and gain unauthorized access to sensitive information. They send convincing email scams, impersonate trusted entities, and even dig up personal details from social media to trick us into revealing confidential data or compromising system security.

Still think that cybersecurity is all about fancy technology?

You took a look at the latest latest ENISA Threat Landscape. You saw that the top threats include ransomware and malware—definitely techie stuff. But guess who unwittingly lets those threats in? Yep, it’s people.

Now let me tell you, the Ponemon Institute’s Cost of Data Breach report is an eye-opener. In their “Initial attack vectors” section, they highlight the prevalence and cost of human-related attack vectors. Stolen or compromised credentials accounted for 19% of breaches, costing an average of $4.50 million. Phishing, at 16% of breaches, topped the list as the costliest initial attack vector, with an average cost of $4.91 million. Business email compromise was another initial vector among cyber attackers.

If you look closely, you’ll notice that every issue, even seemingly technical ones like “Vulnerability in third-party software,” ultimately comes down to human error. After all, who coded the software with the vulnerability or who didn’t define or apply a patching process? That’s right, a human.

Moving Towards a People-Centric Approach:

So, what can we do about it? Well, it’s time for organizations to start adopting a people-centric approach to cybersecurity. My recipe consist in building a “Cyber Culture”! This means understand what are the Cyber behaviors we want to influence, providing comprehensive training programs to raise cybersecurity awareness among employees and promoting a culture of vigilance and responsible behavior. We gotta teach everyday users about common cyber threats, show them how to spot suspicious activities, and encourage good practices like creating strong passwords and keeping software up to date.

But it’s not just about training. Organizations need to share real-world examples of cyber attacks, so people can see the real risks out there. By making everyone feel responsible for cybersecurity, we turn our workforce into a first line of defense against cyber threats.

And here’s a secret: investing in the human factor is not only cheaper, but it’s also way more effective than splurging on fancy technology. I mean, sure, we still need the right tools, but without a strong Cyber Culture, we’re like a castle with a moat but no guards. It just doesn’t work! I will write an article on this topic in the future.

So why isn’t a a People-Centric approach that widespread?

Many people still think that cybersecurity is all about technology. They believe it’s a technical issue that only (nerdy) IT folks (with glasses and a hoodie) can handle. The problem is that cybersecurity specialists often are really technical to start with so they neglect the crucial human elements.

And here’s another kicker: reporting lines within organizations often make things worse. Cybersecurity teams end up aligned with IT departments, who are mainly focused only on technical risks!

I know I’m digressing this is another topic: the need of having an effective, diverse and multidisciplinary Cyber team.

But the truth is, investing in Cyber Culture, in our people, is the key to success. It’s not only more cost-effective, but it’s also more impactful in preventing and mitigating cyber threats. So I think it’s time to break the cycle!

Conclusion:

it’s time we realized that cybersecurity is not just about technology. People play a crucial role, and cybercriminals know it. By adopting a people-centric approach, building a strong Cyber Culture, and empowering employees to be active defenders, organizations can level up their defense against cyber threats.

So, let’s remember that we’re not alone in this fight. It’s not just about fancy tech; it’s about us, the people. Together, we can create a safer digital world. Let’s do this!

Unveiling the Risk Landscape of LLMs

June 5, 2023 / / 1 Comment

A Comprehensive approach proposal

Risk Landscape of LLM
Created with Bing Image Creator

Greetings, readers! Welcome back to our exploration of LLM (Large Language Models) security risks. In my previous posts (here and here), I discussed the significance of understanding these risks. That’s why I am excited to share my participation in the creation of the OWASP Top 10 Risk for Large Language Model Applications 😊.

In this article, we will delve into the challenges involved in defining an approach to create the Top 10 LLM security risk list and propose a holistic approach to address them.

The Challenges in Defining a Top 10 LLM Security Risk List

As we embark on this endeavor, we encounter several challenges that need to be overcome:

  1. Evolving Landscape: LLMs are rapidly evolving, with new models (including Open ones with no restrictions) and attack techniques emerging. Keeping the evaluation comprehensive to address emerging risks is challenging but necessary.
  2. Complexity and Interdependencies: LLMs involve various components, including training data, algorithms, infrastructure, and user interactions. Understanding their interdependencies and how risks propagate across them requires careful analysis. Some components are already covered by other Top 10s but they might be so relevant that we might want to include them
  3. Lack of Standardization: Inconsistencies in terminology and definitions related to LLM security risks can lead to inconsistencies in risk assessment and mitigation. Establishing standardized language and frameworks is vital and luckily OWASP will help a lot in this. A couple of examples below:
    • I had a discussion about Intellectual Property Theft. I wrongly assumed that we were speaking only the theft of the LLM model itself, but if we think about it there are other king of IP theft, e.g., the weights are intellectual property, or if some users provide IP to the LLM, the LLM will learn from that and might provide the IP to the next users. As I said I didn’t consider those as for me those were privacy risks… but these are also ML risks
    • We had discussions on how we should call the “hallucination” risk (e.g., is this term humanizing LLMs? Shuldn’t something as “Confabulation” be better? Maybe, but hallucination is already LLM Jargon).
  4. Multidimensional Risks: LLM risks encompass technical, ethical, legal, and societal aspects. Incorporating these perspectives and achieving a holistic understanding is essential.
  5. Risk Prioritization: Determining the significance of each risk and prioritizing them within the Top 10 list is complex. Professional judgment and a thorough assessment are needed.
  6. Balance of Granularity: Striking the right balance between granularity and practicality is crucial. The Top 10 list should be concise, understandable, and actionable, while capturing the breadth and depth of LLM security risks.

Addressing the Challenges with TARA

“Necessity makes the method” used to say one of my old bosses, and to tackle these challenges, I propose adopting a TARA (Threat Analysis and Risk Assessment) method, which involves identifying potential threats, analyzing their likelihood and impact, and evaluating associated risks.

First Step: Threat Modeling

We start conducting a comprehensive threat modelling exercise, defining threat categories specific to LLMs and documenting potential threats within each category.

Below you will find my proposal of threat list, it is not supposed to be 100% correct, just to give an idea on how it would look like. To do so I used OWASP v0.1, Adam AI centered Top 10 some of the Cybersec risks and ML risks from this super insightful article.

Category Threats Sub-Threat 
LLM-specific Prompt Injection Direct Prompt Injection 
Second Order Injection 
Cross-content injections 
Machine Learning Training-Time Attacks Training Data Poisoning 
Byzantine attacks 
Decision-Time Attacks Inference 
Evasion Attacks  ???
Oracle Attacks Extraction 
Inversion 
Membership Inference 
Model Theft Model Theft
Surrogate Model
Statistical Attack Vectors Bias  Drift 
Model Hijacking Attacks Backdoors 
Trojanized models 
User specific Overreliance on LLM-generated ContentHallucination
Bias
Inexplicability
Operational  ???Inadequate AI Alignment
Application /  
Infrastructure 		Insecure development Inadequate Sandboxing 
Improper Error Handling
Insecure deploymentUnauthorized Code Execution 
SSRF Vulnerabilities
Insufficient Access Controls
Personal Data /  
Intellectual Property 		 ???Data Leakage
IP Theft
A proposal of LLM Threats

To be more accurate, this exercise leans more towards threat identification rather than threat modelling.

Please note that I’m not sure where all the sub-threats should be. For instance an ML threat might be the root cause of the existence of some User specific or Personal Data/IP threats…

The following TARA Steps

The next steps would be:

  1. Risk Evaluation: Estimate the likelihood and impact of each identified threat, considering various perspectives and dimensions. Combine these factors to calculate the overall risk level associated with each threat.
  2. Risk Prioritization: Prioritize risks based on their significance and impact, using professional judgment and a holistic perspective to choose the Top 10.
  3. Mitigation Strategies: Define appropriate mitigation and prevention strategies to address the identified risks effectively.

Those phases are all straightforward, the only difficult part could be understanding the impact. What angle do we need to consider? For an organization of course many of those threats could result in data breaches, financial losses, reputational damage, legal implications, etc. What if we consider a non-enterprise end-user? And the LLM owner? E.g., the latter would be the only one that wants to avoid model theft…

Conclusion

LLMs are at the forefront of technological advancement, and understanding their risks is paramount for secure adoption. By adopting a comprehensive approach like TARA, we can identify, assess, and mitigate these risks more effectively.

Collaboration, standardization, and a multidisciplinary perspective are key to success in this endeavor. Let’s work together to create a safer LLM landscape and pave the way for responsible and secure deployment.

Join me for future articles as we explore LLM security risks and discuss practical mitigation strategies.

OWASP vs. Cybersec.Café's LLM Top Security Risks

OWASP vs. Cybersec.Café’s LLM Top Security Risks

May 31, 2023 / / 0 Comments

A Follow-Up Comparative Analysis

LLM Top Security Risks
Created with Bing Image Creator

Following our previous exploration of Large Language Models’ (LLMs) security risks, I am now presenting a comparative analysis of the risks highlighted by Cybersec.Café and those identified by OWASP (Open Web Application Security Project). OWASP is a renowned authority in web application security and has recently published a preliminary list of LLM security risk.

LLM Top Security Risks Comparative Analysis

1. Jailbreaking

This corresponds to several risks in OWASP’s list: LLM03:2023 – Inadequate Sandboxing, LLM04:2023 – Unauthorized Code Execution, LLM05:2023 – SSRF Vulnerabilities, LLM08:2023 – Insufficient Access Controls, and LLM09:2023 – Improper Error Handling.

In my perspective, Jailbreaking refers to the process of gaining unauthorized access to and control over an LLM’s underlying systems or processes, while OWASP risks might pertain more to the system or application underpinning the LLM rather than the LLM itself. While jailbreaking could serve as an entry point for exploiting these OWASP risks, the mitigation strategies may not be fully effective in all cases.

By articulating these risks separately, OWASP’s approach might help define individual mitigation actions.

2. (Direct) Prompt injection, 3. Second-order injections

These risks directly align with OWASP’s LLM01:2023 – Prompt Injections, although OWASP’s category encompasses all forms of prompt injections.

4. Data Poisoning

This directly aligns with OWASP’s LLM10:2023 – Training Data Poisoning.

5. Misinformation

This risk somewhat corresponds to OWASP’s LLM06:2023 – Overreliance on LLM-generated Content, especially in scenarios where overreliance results in misinformation. However, OWASP’s category includes other potential issues, such as bias, making it more comprehensive.

6. Malicious content generation

This risk intersects with OWASP’s LLM07:2023 – Inadequate AI Alignment. The link might seem tenuous, but the principle remains that an LLM’s use case should not be creating malicious content.

7. Weaponization, 8. LLM-delivered attacks

These risks overlap with OWASP’s LLM04:2023 – Unauthorized Code Execution and LLM07:2023 – Inadequate AI Alignment. These risks underscore the potential for LLMs to be exploited for malicious purposes, be it coding malware or delivering attacks.

9. Abuse of vertical LLM APIs

This risk relates to OWASP’s LLM07:2023 – Inadequate AI Alignment and LLM08:2023 – Insufficient Access Controls. Poor AI alignment could potentially lead to misuse of the LLM, and similarly, poor access control could result in unauthorized actions.

10. Privacy and Data Leakage

This risk directly corresponds to OWASP’s LLM02:2023 – Data Leakage.

Conclusion

In creating this top 10 and comparing it with OWASP’s list, I observed that the key differences lie in the granularity and standardization of terminology.

The field of LLM security is still relatively nascent, and there is a noticeable need for standardization of terms. This comparison has shed light on this fact.

I hope that OWASP’s risk list will bring the critical security considerations for LLMs into sharper focus, laying a solid foundation for further discussions and the development of security measures in this rapidly evolving technology sphere.

The Top 10 Large Language Models Security Risks

May 26, 2023 / / 0 Comments

Understanding the Top 10 Security Risks Associated with Large Language Models (LLMs)

Top 10 Large Language Models Security Risks
Image by Bing Image Creator

Introduction

Large Language Models (LLMs) have revolutionized the field of artificial intelligence and natural language processing, but with great power comes great responsibility. As LLMs become increasingly prevalent, it’s essential to understand the potential security risks they pose.

In light of OWASP’s recent announcement of the OWASP Top 10 Risk for Large Language Model Applications, this article aims to explore my perspective on the top 10 security risks associated with LLMs. I am eager to compare and contrast these risks with the ones OWASP will publish.

Cybersec.Cafè Top 10 Large Language Models Security Risks

  1. Jailbreaking: Bypassing the security measures of an LLM to gain unauthorized control and exploit it for malicious purposes.
  2. Prompt injection: Crafting prompts to influence the model’s output, which can lead to biased, offensive, or harmful text generation.
  3. Second-order injections: Advanced prompt injection techniques, where the prompt itself is generated by an LLM, making it harder to detect and prevent attacks. Note: I’m not considering cross-content injections (a type of prompt injection where the prompt is generated in one context and then used to generate text in another context – this can be used to generate text that is relevant to the first context but harmful in the second context) as I consider still as in between of both risk 2 and 3
  4. Data poisoning: Injecting malicious data into the training dataset, resulting in biased or harmful outputs. Rigorous validation and monitoring are crucial to mitigate this risk. This is actually a Machine Learning (ML) risk that extend to LLMs being that their training is ML based.
  5. Misinformation: Unintentional contribution to the spread of misinformation or support for creating misinformation campaigns.
  6. Malicious content generation: Misusing LLMs to generate persuasive or believable text for phishing or social engineering attacks.
  7. Weaponization: Misusing LLMs to support coding malware or potentially even for malware detection evasion (still a theoretical threat) by generating malware code that evades traditional endpoint detection and response scanners. For example, an LLM could be used to generate malware code that is not detected by traditional Endpoint Detection and Response scanners as the code is generated by an LLM that provides it via API.
  8. LLM-delivered attacks: Using LLMs to deceive users and obtain sensitive information or launch cyber attacks. For example, an LLM could be used to ask a user for sensitive information such as their passwords or credit card number.
  9. Abuse of vertical LLM APIs: Exploiting LLMs for purposes outside their intended use cases, potentially undermining the intended business model.
  10. Privacy: LLMs are trained on massive datasets that contain also personal information, raising privacy concerns if the models generate text like the confidential data it was trained from. This happens for instance with Inference Attacks or Model Inversion Attacks these attacks attempt to infer or recreate information about the training data from the outputs of an ML model.

Some other thoughts

Conclusion

While the risks associated with LLMs may seem challenging, we don’t know yet if they are insurmountable. As of today, we still lack comprehensive solutions to mitigate most of these risks compared to other security domains like applications and mobile devices. Additionally, due to the “black box” nature of LLMs, understanding their inner workings presents challenges in determining the appropriate security measures to adopt. Furthermore, regulatory frameworks surrounding LLM use are still evolving, as discussed in my geopolitical analysis of the ChatGPT block in Italy.

LLM security contains a multitude of unknown unknowns, and it necessitates further research and mitigation strategies to effectively safeguard against these risks. Awareness serves as the critical first step towards achieving effective cybersecurity if it will be ever possible to reach it.

Recommended Readings

To delve deeper into the topic, I recommend reading the following insightful resources:

  • https://www.wired.com/story/chatgpt-jailbreak-generative-ai-hacking/
  • https://themathcompany.com/blog/data-poisoning-and-its-impact-on-the-ai-ecosystem
  • https://spectrum.ieee.org/ai-cybersecurity-data-poisoning
  • https://www.semianalysis.com/p/google-we-have-no-moat-and-neither
  • https://ambcrypto.com/heres-how-to-jailbreak-chatgpt-with-the-top-4-methods-5/
  • https://www.techopedia.com/what-is-jailbreaking-in-ai-models-like-chatgpt
  • https://www.theregister.com/2023/04/26/simon_willison_prompt_injection/
  • https://blogs.itemis.com/en/model-attacks-exploits-and-vulnerabilities
  • https://research.nccgroup.com/2022/12/05/exploring-prompt-injection-attacks/
  • https://hiddenlayer.com/research/the-dark-side-of-large-language-models/
  • https://hiddenlayer.com/research/the-dark-side-of-large-language-models-2/
  • https://embracethered.com/blog/posts/2023/ai-injections-direct-and-indirect-prompt-injection-basics/
  • https://embracethered.com/blog/posts/2023/ai-injections-threats-context-matters/
  • https://www.mufeedvh.com/llm-security/
Security-by-Luck
Photo by Djalma Paiva Armelin from Pexels

Relying on Security-by-Luck

May 22, 2023 / / 0 Comments

The Interplay of Risk, Investment, and… Luck in Cybersecurity

Security-by-Luck
Photo by Djalma Paiva Armelin from Pexels

Last weekend, I came across a LinkedIn post illustrating how numerous companies were breached despite having SOC2, ISO 27001, and PCI-DSS certifications. This observation prompted me to reflect.

Initially, my thought was that there isn’t a direct correlation. The data set is rather small and doesn’t account for all the certified companies that have avoided breaches. Furthermore, certification is a form of assurance that some level of security is in place, signaling to potential attackers that there is valuable data worth protecting.

In the cybersecurity realm, we frequently emphasize robust defense mechanisms, proactive risk assessments, and constant vigilance. Today, however, I want to navigate less charted territory: “security-by-luck”.

What do you mean with Security-by-Luck?

My definition of “Security-by-luck” would be the situation where a company, despite having weak or inadequate security measures, remains unbreached due to factors outside its control, such as the attackers’ choices, capabilities, or sheer chance.

To clarify, I’m not endorsing this as a strategic approach – that would be reckless. Rather, I aim to highlight a crucial facet of cybersecurity – the constant interplay of risk, investment, and a dose of luck.

In a previous article, I discussed on the challenge of defining ‘how much security is enough’. No matter how much an organization invests in security, the threat of an attack persists. Conversely, not all lightly-defended organizations will suffer breaches, too lightly defended (even if those that are inadequately defended become low-hanging fruit for cybercriminals). However, over-investment in security isn’t the solution either, as organizations have other business objectives to meet. So, the question arises, where do we draw the line?

I’m not suggesting that companies should stop investing in cybersecurity and merely hope for the best. Instead, I want to stress the importance of making calculated risks.

To illustrate this, consider four hypothetical companies, each investing differently in cybersecurity…

The contenders:

  • Company A: Does the bare minimum for security (e.g., has an antivirus installed)
  • Company B: Complies with statutory requirements and uses common sense
  • Company C: Adheres to a cybersecurity standard and has obtained certification (like SOC 2, ISO 27001, PCI-DSS, HITRUST, etc.)
  • Company D: Follows all major best practices and has adopted bleeding-edge security solutions

Each of these companies, regardless of their investment level, can either be breached or remain secure. Here’s how:

Vulnerabilities-based Attacks:

  • A vulnerability in their system gets exploited – Company A gets breached.
  • Company B, which patches vulnerabilities quarterly, gets breached when an attacker exploits a flaw within the time window before it gets patched.
  • Even Company C, which patches vulnerabilities monthly, gets hacked, as the attackers were quicker on their feet.
  • Company D has no known unpatched vulnerabilities (a near impossibility in real life, but let’s go with it). However, there’s a zero-day vulnerability that they aren’t aware of (I know this is the definition of zero day). An attacker discovers and exploits it – Company D gets breached.

Let’s assume, for a moment, that all these companies understand this risk and decide to have all vulnerabilities patched (again a near impossibility) and are lucky there aren’t any unexploited zero-day vulnerabilities. You might think they’re safe. But what if an attacker targets their people instead?

People-based Attacks:

  • An attacker successfully executes a phishing attack on Company A, leading to a breach.
  • Despite having good email security and having conducted a phishing simulation last year, Company B falls prey to a successful social engineering attack.
  • Company C suffers a sophisticated MFA fatigue attack and gets breached.
  • In Company D, an attacker bribes an employee to gain access to the system (including credentials and MFA, as seen in the Lapsus$ attacks last year).

Even if the organization decide to invest in a solid cyber culture and luckily their employees are equipped with strong ethics to resist such attempts, are the potential threats truly over?

Unfortunately, no, the threats aren’t over. They are susceptible to…

Supply Chain Attacks:

The attack surface extends to vendors, giving birth to a new cycle of vulnerabilities and people-based attacks. Hence, even Company D could harbor cybersecurity points of failure within their supply chain.

Luck is Not a Strategy

In essence, cybersecurity isn’t merely about investment levels; it’s also about the complex interplay of factors that contribute to a company’s overall risk profile. Even the most secure organization cannot completely rule out the possibility of a breach. Given the dynamic nature of the landscape, absolute security is a virtual impossibility, making a small element of ‘luck’ an undeniable part of the equation.

Regrettably, many companies have relied solely on this ‘luck’ factor for so long that they’ve now become easy targets.

‘Security-by-Luck’ should not be a strategy in itself, but understanding its role in the broader cybersecurity framework is essential. The goal should always be to optimize investment, maintain a robust defense mechanism, foster employee awareness, and devise sound strategies to mitigate potential risks, including supply chain risks. This involves striking a balance, understanding that no solution offers 100% protection, and ensuring readiness to respond effectively (by having incident response plans and exercises conducted) if or when a breach occurs by conducting regular incident response plans and exercises.

Conclusion

In conclusion, while we can’t depend entirely on luck, or as the Cybersecurity community usually call it, the residual-risk, acknowledging its existence, could make us more attuned to the realities of the ever-evolving cybersecurity landscape. The presence of residual risk is an undeniable part of cybersecurity, and acknowledging without relying on it might encourage a more realistic approach towards cybersecurity strategy and implementation.

« Older posts Newer posts »

© 2024 CyberSec.Cafe