There is significant pressure on heads of infrastructure and operations (I&O) to drive the implementation of generative AI (GenAI) and AI solutions. Deploying these two emerging technologies has risen to be the highest on the list of CIO priorities in the next year.
However, there are six barriers to GenAI and AI implementation that pose unique challenges to I&O leaders.
The following insights will help heads of I&O support their teams in overcoming those barriers.
Power/Cooling Requirements
AI server requirements are different from typical x86 servers. The high power draw of AI-focused hardware like GPUs and accelerators causes a roughly fivefold to twentyfold difference in power consumption.
The locations where these AI servers reside will likely need modifications to accommodate these special requirements. Heads of I&O must lead their teams to realistically estimate and forecast the power in each location where they plan to deploy, due to their significant power consumption.
Greater power consumption at these higher levels also means greater heat production.
Data center modification requirements and costs must also be considered to support an AI business case. Newer AI chip generations will require direct-to-chip cooling, which takes the heat from the chip to the outside of the chassis.
Heads of I&O must not forget that heat must ultimately be taken from the chip to the atmosphere, so supporting infrastructure and its associated costs will be required.
To optimize AI power/cooling requirements, heads of I&O should mandate a “performance-per-watt” precision standard: Selecting an optimized 8-bit or quantized format instead of a 16-bit precision can reduce AI-related total cost of ownership by a factor of more than four.
AI Governance
To support an on-premises GenAI or AI solution, heads of I&O must ensure that their teams are guarding against any lack of transparency, data protection and privacy issues, model evaluation and validation, and security threats that can come with any AI-related implementation.
Heads of I&O must lead their teams to adopt effective AI governance to overcome this AI barrier, or risk project failure.
A successful governance framework balances enforcement, enablement and engagement in governance. I&O leaders should look to gain foresight, build the value case, execute strategies and scale to maximize value.
Use Case Identification
The pressure to implement GenAI and AI is increasing, and it can be challenging to ensure that AI is not implemented just for AI’s sake.
Heads of I&O must help drive the selection of the right use cases for on-premises AI or GenAI implementations to generate the optimal returns from what is likely to be a significant infrastructure investment. This can be done by discovery of new use cases or evaluation and prioritization of existing use cases.
Focused Purchase Timing
Demand far outweighs the current supply of the chips required for GenAI and AI implementations.
To compound this, demand for large purchases from hyperscale providers may take precedence over smaller-sized server purchases that enterprises make.
As a result of the significant demand, some providers are offering shorter windows for purchasing decisions when AI server supply is available. Typical response windows for infrastructure proposals from providers have traditionally been 30 days.
Heads of I&O should not be surprised if an AI server provider reduces that decision time to 15 days, essentially cutting it in half. Heads of I&O must evaluate this timing to ensure that purchase timing aligns with implementation requirements, thereby avoiding missing critical inflection points on their GenAI and AI timelines.
Downtime Tolerance
AI infrastructure is often evaluated and implemented in a distinct and separate process from typical infrastructure.
To avoid the consequences of potential unplanned downtime, heads of I&O should ensure that AI infrastructure is subject to or incorporated into an existing business impact analysts (BIA), so that the required levels of availability are maintained and negative business impacts are avoided.
The results of an AI-specific BIA can allow it to be knitted into existing disaster recovery tiering and plans. These results can also expose dependencies with other business functions and their supporting applications.
Limited Skills Availability
Limited skills are one of the most challenging barriers for heads of I&O to overcome.
The evaluation, purchase and implementation of on-premises AI infrastructure is still relatively new. There is not a plethora of skilled I&O professionals from which to draw.
To develop a successful plan that incubates, trains and acquires the right skills to create and support AI or GenAI infrastructure implementation, heads of I&O must work with their HR counterparts.
For example, Kubernetes skills become essential for enterprises deploying and managing GenAI and AI workloads at scale, as they provide the robust infrastructure needed to handle AI’s resource-intensive and dynamic nature.
This means that skill requirements planning by heads of I&O must be a key part of a GenAI or AI project.
Failure of a head of I&O to drive a practical approach to increasing AI infrastructure skills within their teams puts any related project at a high risk of ultimate failure.
I&O leaders must proactively address the unique infrastructure challenges of AI and GenAI by planning for increased power and cooling needs, establishing strong governance, selecting high-impact use cases, acting quickly on purchase opportunities, integrating AI into business continuity plans and investing in critical skills development.
By tackling these barriers head-on, leaders can position their teams for successful AI adoption and deliver lasting value to their organizations.
Trusted insights for technology leaders
Our readers are CIOs, CTOs, and senior IT executives who rely on The National CIO Review for smart, curated takes on the trends shaping the enterprise, from GenAI to cybersecurity and beyond.
Subscribe to our 4x a week newsletter to keep up with the insights that matter.
