Structured output while working with LLMs

[sandesh_arote]

Have you ever faced issues with insufficient control over your large language model's output when developing applications?  Often, models are not configured to produce results tailored to their specific needs, leading to outputs that are less useful or relevant. This oversight is a recurring issue in many projects. Developers frequently neglect to design their prompts in a way that guides the model to generate outputs that can be practically applied to their use case. This misstep can result in inefficiencies and additional work to manually adjust or interpret the model’s responses.

One could easily solve this problem by specifying the output format in the prompt. One should be precise what output is required and its structure. Passing the structure of the output helps a lot in such situation.

Langchain addresses this problem by offering a comprehensive suite of tools known as Output Parsers. These tools are specifically designed to help developers manage and format the model's output effectively. Output parsers enable precise control over the structure and content of the responses generated by the language model, ensuring that the outputs align closely with the intended application. By utilizing output parsers, developers can streamline their workflows, reduce errors, and enhance the overall usefulness of their applications. This makes Langchain a powerful resource for anyone looking to harness the full potential of large language models in a controlled and productive manner.

Lets get started with techniques to address this issue effectively:

1. Structured output without Langchain
2. Structured output with Langchain (Pydantic parser)

Structured output without Langchain

In this technique, we pass the structure of the output we expect LLM to give. This is simply passed in prompt.

CODE

import requests
output_format = {
    "product":"Product mentioned in Paragraph",
    "price_in_rupees":"Price of the product in rupees",
    "price_in_dollar":"Price of the product in dollars",
    "credit_card_number":"Credit card number provided"
}
prompt = f"""You are given a paragraph. You have to extract information from the above given paragraph.\
    Paragraph : I saw Nike's sneakers on amazon. They are for 67 american dollars which is approximately equivalent to \
    5600 Indian rupees. Here is the credit card with number XXXX-ZZZZ-AAAA-IIII.

    Output Format : {output_format}

    Please don't give any analysis from your side. Just return the required result in given format.
    """
url = "<your-endpoint>"
headers = {
    "Authorization": "Bearer <token-here>",
    "Content-Type":"application/json",
    "AI-Resource-Group":"default"
}
payload = {
  "messages": [
    {
      "role": "user",
      "content": prompt
    }
  ]
}
response = requests.post(f"{url}",
                                     headers=headers,
                                 json=payload)
print("\n\nStructured Output :")
print(response.json()["choices"][0]["message"]["content"])

OUTPUT

This is the simplest way one can achieve structured output with any framework. But this approach has one problem, the result is returned in string format. We need to pre-process the data before we use it in code. There can be chance where output can have escape characters or any other unwanted characters which we need to handle manually.

Structured output with Langchain (Pydantic parser)

It's important to remember that large language models are leaky abstractions. This means that they might not always produce perfect results, and the quality of the output can vary depending on the complexity of the task and the capacity of the model being used. To generate well-formed JSON or other structured outputs, it's crucial to use an LLM with sufficient capacity and to design prompts that clearly specify the desired format.

By utilizing output parsers and carefully crafting prompts, developers can significantly improve the reliability and usefulness of the outputs from large language models, making their applications more efficient and effective. This makes Langchain a powerful resource for anyone looking to harness the full potential of large language models in a controlled and productive manner.

Prerequisite:

Create a requirements.txt file 

requirements.txt

generative-ai-hub-sdk==4.1.1
langchain==0.3.0
langchain-aws==0.2.1
langchain-community==0.3.0
langchain-core==0.3.28
langchain-google-genai==2.0.1
langchain-google-vertexai==2.0.1
langchain-openai==0.2.9
langchain-text-splitters==0.3.0

Install the dependencies using below command:

pip install -r requirements.txt

CODE

from gen_ai_hub.proxy import GenAIHubProxyClient
from gen_ai_hub.proxy.langchain.openai import ChatOpenAI
from langchain.output_parsers import PydanticOutputParser
from pydantic import BaseModel, Field
from langchain.prompts import PromptTemplate
from typing import List
gen_ai_hub_proxy_client = GenAIHubProxyClient(
                                    base_url=<AICORE_BASE_URL>,
                                    auth_url=<AICORE_AUTH_URL>,
                                    client_id=<CLIENT_ID>,
                                    client_secret=<CLIENT_SECRET>,
                                    resource_group=<RESOURCE_GROUP>
                                )
chat_model = ChatOpenAI(proxy_client=gen_ai_hub_proxy_client,
                                     proxy_model_name= "gpt-4",
                                     temperature= 0)

class Product(BaseModel):
    products: List[str] = Field(description="Products mentioned in Paragraph", default=[])
    price_nike: int = Field(description="Price mentioned for Nike sneakers", default=0)
    price_LV: float = Field(description="Price mentioned for LV sneakers", default=0.0)
    credit_card_number: str = Field(description="Credit card number provided", default="None")
    shipping_address: str = Field(description="Address for shipment of product", default="7th street, Hill Road")


parser = PydanticOutputParser(pydantic_object=Product)

prompt = PromptTemplate(
    template="You are given a paragraph. You have to extract information from the above given paragraph. If no data is available to extract \
            just return output with default value\n{format_instructions}\n{paragraph}\n",
    input_variables=["paragraph"],
    partial_variables={"format_instructions": parser.get_format_instructions()},
)

paragraph = f"""I saw Nike's sneakers on amazon. They are for $67. I also saw LV's sneakers, they were for $1499.67\
      Here is the credit card with number XXXX-ZZZZ-AAAA-IIII."""
chain = prompt | chat_model | parser

response=chain.invoke({"paragraph": paragraph})
print("Structured Output :")
print(response)

OUTPUT

One notable feature of pydantic parsers is the ability to set default values. Paragraph provided to the LLM does not include a shipping address. Despite this, the output can still contain an address by using default values specified in the pydantic model. This is incredibly useful for handling cases where certain information is either not extractable by the LLM or simply not available in the input data. By providing default values, developers can ensure that the output remains complete and usable.

Additionally, pydantic parsers allow for specifying the data type for each key in the output. This ensures that the extracted data is not only present but also in the correct format, reducing the risk of type-related errors downstream in the application. The overall output object is an instance of a pydantic class, "Product" in this case, encapsulating all the specified keys and their corresponding values.

In summary, controlling the output of large language models is critical for developing useful and reliable applications. Langchain's output parsers provide an essential toolset for achieving this control, with features like pydantic models, default values, and data type specifications. By leveraging these tools and understanding the inherent limitations of LLMs, developers can create more robust applications that truly meet their needs. Langchain stands out as a valuable asset for anyone aiming to maximize the potential of large language models in their projects.

NOTE : The output varies from model to model. This blog was written considering GPT-4 as a LLM model.