Massive hack hits AI servers, exploits Ray framework vulnerability

Researchers have found that thousands of servers, running AI infrastructure, have been hacked in an active attack campaign. This attack targets a reported vulnerability in Ray, a computing framework used by the likes of OpenAI, Uber, and Amazon.

According to analysts at Oligo, a cybersecurity firm, the vulnerability allows attackers to take over the companies’ computing power and leak sensitive data. The blog post stated that this flaw has been under active exploitation for the last seven months, affecting sectors like education, cryptocurrency, biopharma and more.

I discovered an active attack campaign targeting a vulnerability in Ray. Thousands of companies running AI infrastructure are exposed to the attack through a critical vulnerability that is under dispute and thus has no patch.

It all began with a client that was using Ray. pic.twitter.com/BrVr2p8yOu

— Avi (@avi_lum) March 26, 2024

They claim that a trove of sensitive information from compromised servers has been leaked, targeting entities such as OpenAI, Hugging Face, Stripe, and Slack, as well as cloud environments like Amazon’s AWS and Microsoft Azure.

Oligo reports that it discovered hundreds of compromised clusters, with hackers allegedly installing cryptocurrency miners on compromised infrastructure. The researchers explain that attackers choose to compromise these machines because they can obtain valuable sensitive information, and GPUs are very expensive and difficult to obtain.

The company reports that GPU on-demand prices on AWS can reach an annual cost of $858,480 per machine, which means the total amount of machines and computing power that might have been compromised is estimated to be worth almost a billion dollars. Attackers have also installed reverse shells, which are text-based interfaces that allow for remote server control.

In a statement, it continued: “When attackers get their hands on a Ray production cluster, it is a jackpot. Valuable company data plus remote code execution makes it easy to monetize attacks—all while remaining in the shadows, totally undetected (and, with static security tools, undetectable).

Ray, an open-source unified compute framework, simplifies the scaling of AI and Python workloads, including everything from reinforcement learning and deep learning to tuning and model serving.

These applications generally operate on large clusters of servers. A central dashboard serves as an interface for displaying and managing active tasks and applications. Among the programming interfaces accessible via this dashboard is the Jobs API. It enables users to dispatch a list of commands to the cluster through a straightforward HTTP request that doesn’t require authentication.

In November 2023, analysts from the security outlet Bishop Fox spotted a similar vulnerability in Ray, tracked as CVE-2023-48022. Bishop Fox senior consultant Berenice Flores Garcia wrote in a blog post: “In the default configuration, Ray does not enforce authentication. As a result, attackers may freely submit jobs, delete existing jobs, retrieve sensitive information, and exploit the other vulnerabilities described in this advisory. “

A spokesperson for Anyscale, the developer behind Ray, confirmed to ReadWrite that there were a number of issues it was fixing. In a statement, they said: “In light of reports of malicious activity, we have moved quickly to provide tooling to allow users to verify proper configuration of their clusters to avoid accidental exposure.”

They also said the firm was providing a client-side script and server-side code and that it had pre-configured the defaults of the client-side script to reach out to a server it has set up, simplifying the process of determining whether or not ports are unexpectedly open.

Last October, the company had initially denied the reports stating that four of the five reported bugs had already been fixed in November. They disputed the term “vulnerability,” referring to it as a bug instead.

It added: “We recognize that reasonable minds can differ on this issue, and consequently have decided that, while we still do not believe that an organization should rely on isolation controls within Ray like authentication, there can be value in certain contexts in furtherance of a defense-in-depth strategy, and so we will implement this as a new feature in a future release.”

Featured image: DALL-E

The post Massive hack hits AI servers, exploits Ray framework vulnerability appeared first on ReadWrite.