Llm Adoption Considerations

Four experts discuss some issues people should think about when adopting LLMs and how they can make the best choice for their specific use case. This TensorBlue analysis is based on reporting and source material from InfoQ (https://www.infoq.com/articles/llm-adoption-considerations/).

What Happened

InfoQ Homepage Articles Virtual Panel: What to Consider When Adopting Large Language Models

Virtual Panel: What to Consider When Adopting Large Language Models

When choosing between an API-based vs. self-hosted model, consider that the API solution is simpler for early rapid iteration, but self-hosted may be better for long-term cost and privacy concerns.

Before fine-tuning a model, try prompt engineering and retrieval augmented generation (RAG).

Smaller open models may not match the performance of large closed models like GPT-4 in all scenarios, but they are often good enough for many tasks and are worth trying.

Hallucination is a common risk of LLMs, but RAG using trustworthy sources is a good mitigation.

When adopting LLMs, organizations should invest in the education and training of their employees, especially focusing on the capabilities and limitations of the models.

Large Language Models (LLMs) are a general purpose AI solution that can handle a wide range of tasks: answering questions, summarizing long documents, even writing code. Many organizations would like to adopt this technology, but with the fast pace of innovation, it can be difficult to keep up with the different LLM options available, each with its own benefits and risks.

Most people are familiar with the big models available over the web via APIs, such as ChatGPT. Many are also familiar with the risk

Meryem Arik: There are a few main reasons why you should be working with self-hosted models rather than API based models, and we covered them in a recent blog post. Firstly, control and data privacy. For a lot of enterprises, their LLM applications will be touching fairly business-sensitive data, and for them it may be important that they control the model that sees that data. Secondly, customizability. When you self-host models you control all of the weights in the model. This means you can fine-tune or adapt the model as you wish. This can give better results even with smaller models. Thirdly, cost and scalability. It is true that when experimenting, API based models tend to be cheaper, as there is no need to set up the infrastructure of self-hosting. However, at scale, it is cheaper and more scalable to self-host with a highly efficient inference stack. Now that open-source models like Llama 3 have come out, there is very little reason to not build up the capabilities to self-host. Llama 3 is on par with the leading API-based models but comes with the additional benefit of being able to deploy it privately and in your own environment. Even if not all of your applications use self-hosted models, every enterprise must build up the capabilities to self-host otherwise they are seriously missing out.

InfoQ

Implications for Product and Engineering Teams

For TensorBlue readers, the useful question is not just what happened, but how this changes product architecture, engineering priorities, AI delivery, observability, team workflows, or executive decision-making.

• Review whether this changes your AI roadmap, platform architecture, or engineering operating model.
• Identify the specific workflow, reliability, governance, or developer-productivity lesson that applies to your organization.
• Convert the lesson into a small production experiment with measurable quality, latency, cost, adoption, or risk metrics.
• Document source assumptions clearly so teams do not overgeneralize from incomplete public information.

TensorBlue Takeaway

The practical opportunity is to turn this signal into a concrete implementation decision: better AI systems, stronger product instrumentation, more reliable automation, and clearer technical governance. Teams that connect public technology shifts to their own delivery systems will move faster without adding unnecessary complexity.