Investing in artificial intelligence (AI) has become crucial for enterprises aiming to stay competitive. However, the key to a successful AI strategy lies in effectively measuring and optimizing the return on investment. In this first post, I’ll be focusing on the investment side of that calculation. I’ve spoken to many organizations that have embraced AI, and have observed what’s working and what isn’t. The key concept executives must understand is the total cost of ownership (TCO). Here's a guide to help you navigate this complex landscape.
When you think about AI investments, the TCO encompasses several factors beyond the initial purchasing decision. Key among these are the compute resources you select, the talent needed to support these resources, and all those essential, albeit "unsexy," components like governance and education.
External API Approaches: If your security posture allows it, using externally hosted models via APIs is a great way to hit the ground running. It allows for rapid prototyping and proof-of-concept work without hefty upfront investments. However, beware of the costs growing as your project scales. For instance, using methods like retrieval-augmented generation (RAG) can quickly drive up costs due to increasing input token counts. Like most managed services, while these solutions are convenient, reaching a certain scale usually makes self-hosting the more cost-effective path. But keep in mind that transitioning to self-hosting can complicate how you quantify your investment.
Host Internally: Hosting models internally gives you control over security, performance and costs, but it requires a careful evaluation of your infrastructure capacity and capabilities. This path can optimize long-term expenses but requires upfront investment and the right team to manage it.
Right-size Your Model: Always align your AI capabilities with your primary use-cases. Avoid over-investing in a complex model (think Ferrari) when a simpler one (think golf cart) could meet your needs just as effectively and at a lower cost. Security, as always, is an important factor. If you’re hosting this model, how much overhead do you want to take on? Also, how are you actually using these models? It is important to understand your use-cases and determine if you actually need all those extra billion parameters.
Funny story: Yurts recently performed a bakeoff between two competing models for a customer that needs to self host. These models have very different parameter sizes. In this case, we defined a series of tasks, and asked humans which model performed better. The smaller model won on many of the tasks, and was on par for the rest. Bigger isn’t always better, but it’s almost always more expensive.
Support Staff for Compute: Hosting internally requires system administrators, data engineers, AI specialists, devops, MLops, and project managers to oversee development and resource allocation. Salaries and administrative staff are significant costs. Factor these into your overall investment. And please, pretty please, with sugar on top, don’t say, “we already have those people.” Opportunity costs are real.
Strategic Development: Utilize your internal AI and tech talent thoughtfully. As AI technologies become increasingly commoditized, the opportunity costs of developing undifferentiated tools internally can be significant. Focus your team’s efforts on projects that are bespoke and aligned with your enterprise's core competencies, and consider off-the-shelf solutions for everything else. This allows your organization to remain agile and on the cutting edge without unnecessary internal development strains.
Finally, successful AI implementation isn’t just about the technology and people—you also need to spend real resources on:
Investing in AI is as much about strategic planning as it is about technology. By considering the full TCO, including compute resources, human capital, and the foundational elements of a strong AI strategy, you can optimize your investments to deliver sustainable value. Keep these factors in mind, and you’ll be well-positioned to navigate the complexities of AI in the enterprise landscape, ensuring that your organization not only keeps pace with changes in technology but leads the charge.
Request a Demo to discover how Yurts can help your organization enhance efficiency and achieve its AI-driven goals.
