The rapidly evolving landscape of artificial intelligence and the integration of AI technologies into business processes has become an incredibly hot topic.  Most notably the inclusion of large language models (LLMs), into various tools and applications, like chatbots and virtual assistants to name a few, has started to become much more commonplace. 

As engineers and analysts rush to harness the power of LLMs to help drive business value, a critical challenge emerges: how to ensure these models deliver both optimal and accurate outputs? This question becomes especially acute if LLMs have only been trained on public data. This can cloud responses to business-specific questions. Two methods for achieving this outcome are prompt engineering, and fine-tuning. Knowing which approach is right for your business means understanding what is required for each to be effective and how easily they can be adopted. 

In this blog, we’ll take a look at the differences between prompt engineering and fine-tuning, provide insights into each approach, and ultimately make the case that prompt engineering is all you need to solve the majority of enterprise use cases.

Prompt engineering vs. fine-tuning explained

When working with generative AI applications such as Large Language Models (LLMs) to deliver meaningful output, prompt engineering and fine-tuning have emerged as approaches. 

Before we go further, let’s take a peek into how LLMs work. First a user initiates an interaction with a model in the form of an input called a prompt.  A prompt for an LLM is like a question or sentence that tells the AI what you want to talk about or get information on. The LLM then uses its training to recognize the words, context, and relationships in the input, and then generate a response that matches the prompt’s intent and context.

In order for the model to do all of this, it is trained using very large volumes of textual data, both structured and unstructured. It then uses pattern matching to identify relationships between words, phrases, and concepts in the prompt and its training data to formulate a response. Finally, it uses natural language generation to respond in a way that is understandable, in context, and resembles human-generated text.

When it comes to prompt engineering, LLMs generate responses based on prompts provided by the operator, or end user. This is a precision-focused approach that empowers end users to more easily create optimal interactions with LLMs that result in highly accurate responses without needing advanced skills or experience with AI models. To obtain different responses from an LLM using prompt engineering, all an end user has to do is modify the prompt and provide more context. Context can be given by including new information, clarifying the question, making requests in sequence, or even including more data. Data itself can actually be used to provide context to a prompt. 

For example: if an e-commerce platform deploys a generative AI application such as a chatbot for customer inquiries, and a customer asks, "What laptops do you have?" The model, with the context of the user's browsing history and previous purchases, can provide laptop recommendations that closely match their price range and brand preferences. This context-driven prompt engineering is influenced by data and ensures a personalized and relevant response. By crafting tailored prompts, users can develop queries to extract highly relevant insights from complex data. This technique eliminates ambiguity, streamlines information retrieval, and ensures accurate results. 

On the other hand, fine-tuning involves making changes to the underlying data sets themselves, or the parameters used to train the LLM. By adjusting these parameters, an LLM can then be trained to deliver different responses. Fine-tuning may offer slightly more control over the model's behavior, but this approach is very resource intensive and requires skill sets that are often well beyond the reach of data teams, from startup to the enterprise.

Now let’s take a closer look at each of these in practice.

Building a generative AI application based on the prompt engineering approach

As mentioned, prompt engineering is a highly effective approach for ensuring you are receiving meaningful output from generative AI applications such as LLMs. This involves the development of input prompts to guide the output of LLMs instead of having to touch the models themselves. 

Diving a little deeper, prompt engineering for generative AI applications involves three essential components that should be done in succession: building a knowledge warehouse, populating the knowledge warehouse; and finally, building a generative AI application that uses the knowledge warehouse. Let’s take a look at each of these.

Building a knowledge warehouse

Knowledge warehouses store unstructured data sources such as documents, Slack messages and support tickets and serve as the repository for the relevant information utilized by a generative AI model. When it comes to choosing a knowledge warehouse, there are several excellent options available. These include advanced vector databases like Pinecone, Weaviate, and Milvus, known for their ability to manage complex data. Then, the process of getting data into the knowledge warehouse, or vector database, is a data integration or ETL problem that Prophecy helps simplify. 

A knowledge warehouse serves three main purposes:

• Document storage: the knowledge warehouse stores a vast array of data in documents, all with their own unique ID numbers called vector embeddings. These vector embeddings make it easy and efficient to organize and retrieve information from the store (much like unique keys in a database). 
• Document search: once documents and data are in the knowledge warehouse, you can then leverage vector embeddings to efficiently search for similar or relevant documents. This is a way to retrieve documents that closely match specific criteria or requirements.
• Indexing: Indexing is a technique where a special roadmap is created that points to where specific pieces of data or documents are stored. Indexing is employed to enhance search speed within the knowledge warehouse, making it much faster to find the exact information you're looking for.

By building a knowledge warehouse, prompt engineers will have a vast resource to leverage when developing effective prompts. This will act as a foundation for generating contextually relevant and informative prompts, ultimately improving the quality and precision of AI-generated content.

Prompt engineering for end users

Now that we have our knowledge warehouse, it’s time to consider how best to educate and enable those responsible for prompt engineering on how to achieve the best results. Developing effective prompts involves following several rules of thumb. 

First, prioritize clarity and specificity in your prompts. It is very important to provide hints in the prompt to help guide style and output. Clearly state the desired outcome you’re looking for and provide explicit instructions such as “reply with 3 to 5 sentences” instead of “use only a few sentences”. Incorporate relevant context and details to guide the model and to help it generate responses as contextually accurate as possible. 

Next, pay attention to formatting and structure, ensure prompts are well-organized and aligned with your desired output and also use the appropriate keywords and phrases as cues for the model. These keywords and phrases are crucial as the gen AI application will look for similar documents in the knowledge warehouse to send with the prompt, along with the question. 

Consider including examples and variations to cover different scenarios and increase the chances of obtaining desired outputs. And definitely avoid ambiguity by crafting prompts that leave no room for misinterpretation. 

Finally, approach prompt engineering as an iterative process and regularly test and refine your prompts. This helps those that are responsible for prompt engineering to rapidly improve their skills, and it will also help to effectively train the model to respond quickly and accurately. 

A look at fine-tuning the underlying model 

While prompt engineering clearly offers a user-friendly and lower cost approach, the process of fine-tuning can also help deliver accurate outcomes, but is highly specialized and is not always economically feasible. This method involves updating the parameters of the LLM – of which there can be many – using task-specific data. This allows the model to become very good at generating responses that are in context in complicated scenarios. When the goal is to train an LLM for a particular task, and your organization has the time, skills, and resources to invest in this approach, fine-tuning could be a reasonable choice. Let’s take a closer look.

There are a variety of methods to fine-tuning LLMs that can be considered to improve model performance and behavior: feature-based, transfer-learning, and full fine-tuning.

In the “feature-based” method, a pre-trained LLM is loaded and applied to the specific target dataset. A simple example of this would be if you want to teach a model to identify whether a sentence is talking about a dog or a cat, you can use the knowledge it already has. Instead of starting from scratch, you can take the features it learned about animals and train a smaller model to tell the difference between dogs and cats using those features. You're actually building on what the model already knows to teach it a specific skill.

With the “transfer-learning” method, the focus lies on only updating the model's output layers (which are the final layers in the model's architecture responsible for generating actual responses or predictions). This allows a model to be repurposed, or fine-tuned, to perform a different but related task from what the model was originally intended to do. For example, if you have a pre-trained model designed for language translation, you can fine-tune it for sentiment analysis by modifying the output layers to classify text as positive, negative, or neutral based on sentiment. 

Lastly, using the “full fine-tuning” method requires a comprehensive update across all layers of a model, which enhances its adaptability and responsiveness to specialized tasks. A use case here would be taking a pre-trained model designed for general language understanding and fine-tuning it for text summarization by adjusting all of its output layers to focus on extracting key information and generating concise summaries. It should be noted that fine-tuning more layers results in better performance, but also comes with increased costs, so while full fine-tuning offers better results, it will also be the most expensive approach. 

Considerations for fine-tuning

Fine-tuning Large Language Models (LLMs), while effective, has unique challenges. LLMs can be massively complex, containing as many as billions of weights and parameters. Updating these parameters to achieve desired results requires expertise that can be difficult to hire as well as retain. And, fine-tuning LLMs can be very resource-intensive, in terms of both time and computing power. Due to this, it may not always be realistic for organizations to fine-tune more than a limited number of parameters at any given time.

The issue of limited resources for fine-tuning is a particularly difficult challenge for enterprises. As they deal with large-scale applications and diverse use cases, the need for customizing LLMs to fit specific requirements really becomes crucial. While fine-tuning can help organizations to adapt models to their unique needs or tasks, the cost and complexity of fine-tuning at scale can be a major barrier to success for many enterprises.

Conclusion

Prompt engineering and fine-tuning are both powerful strategies for enhancing the proficiency of large language models in generating precise and tailored responses. However, prompt engineering emerges as a superior approach not only due to its direct influence over the input context, leading to more controlled and contextually aligned outputs; but also due to the ability for data users to more quickly develop the skills to effectively develop these prompts. Central to the success of any generative AI project is having an adaptable platform that can enable the success of these efforts by ensuring the largest cross-section of users can effectively contribute. 

Prophecy’s Generative AI Platform puts the power of generative AI in the hands of every user in an organization and provides the ability to train AI applications against their own enterprise data. We invite you to see for yourself. 

Getting started with Prophecy Generative AI

To get started with Prophecy Generative AI and explore more information, you can:

• Start a free trial
• Learn more about building powerful generative AI applications with Prophecy