Red Hat announced updates to Red Hat OpenShift AI and Red Hat Enterprise Linux AI (RHEL AI), with a goal of addressing the high costs and technical complexity of scaling AI beyond pilot projects into full deployment.
New capabilities include the ability to split AI inference across multiple GPUs and nodes, integration of InstructLab into OpenShift AI pipelines, and a new AI guardrails orchestration framework.
In addition, Red Hat is now offering new free online training courses for its customers.
IDC predicts that enterprises will spend $227 billion on AI this year, embedding AI capabilities into core business operations.
But companies are facing challenges when it comes to integrating the components required to run AI applications, as well as moving from experimentation to deployment.
Enterprise-scale AI deployment
RHEL AI lets companies deploy workloads where they need them, including cloud, on premises, or on the edge, IDC analyst Michele Rosen says.
“AI applications need to live as close as possible to their data,” she says. “For security, latency, and what have you. So you want to have a hybrid cloud infrastructure.”
RHEL AI lets companies avoid platform lock-in, especially as AI providers continue to leapfrog one another in performance, features, and cost efficiency.
And organizations already using Red Hat Linux or OpenShift can use their existing investments and expertise, she adds.
According to Red Hat, the new release of RHEL AI and OpenShift AI help companies with various aspects of AI deployment, including training, model evaluation, inference, and guardrails.
InstructLab is Red Hat’s tool for fine-tuning and customizing models with a company’s own data and expertise. Until today, it was only available on a single server.
That meant that training runs could take a long time, says Jeff DeMoss, Red Hat’s director for AI product management. “If you’re operating on a single server, an end-to-end run on InstructLab can take over 24 hours. But if you take it into a distributed environment, you can speed up the overall performance.”
There’s also a new user interface to help subject matter experts more easily contribute to model development, as well as integration into the OpenShift pipeline, to make it easier to take models from training to deployment.
Better AI guardrails
Today, commercial AI platforms and hyperscalers are ahead when it comes to offering ready-to-go guardrails for generative AI.
Red Hat’s latest release includes an AI guardrail orchestrator, but not the actual guardrails themselves.
“Right now we support a bring-your-own-detector,” says DeMoss. “I expect that we will provide more packaged options as we continue to enhance guardrails.”
The idea is that customers will have a framework to customize the behavior of the model and filter out certain kinds of information both at the input and output stage, he says. “You can ensure it doesn’t have hateful or abusive speech, filter out personally identifiable information, or filter out competitor information — you don’t want your AI to produce information about competitors’ products.”
Red Hat will continue to enhance this area through the year, he adds.
More AI models, more problems
RHEL AI now supports IBM’s latest Granite 3.1 8B model, which is multi-lingual and has a larger context window.
In addition, OpenShift AI supports a “bring-your-own-model” approach, though it doesn’t yet have a curated model garden or ready-to-go options for customers, and the company had no comment about its plans to support other AI models.
“We’ll have more to stay about that in the future,” says DeMoss.
However, Red Hat does now offer model evaluation. This allows enterprises to check the performance of a particular model — whether their own custom model or an off-the-shelf one — against industry benchmarks. In addition, companies can also create their own tests, based on their own business requirements, to evaluate models against.
The industry benchmarks are the same ones as those used in the Hugging Face LLM leaderboard, says William Caban Babilonia, Red Hat’s senior principal product manager for generative AI.
“But the leaderboard doesn’t tell me that it’s good for my business,” he says. “I might care that my model knows about my financial department — not about the stars in the universe. That’s where the value is.”
The model testing is integrated right into the pipelines, so enterprises can do the evaluation right at the point of deploying the model, says DeMoss.
The model evaluation tool will be particularly useful as Red Hat expands beyond the Granite models, says IDC’s Rosen.