Artificial Intelligence and Machine Learning Blogs
Explore AI and ML blogs. Discover use cases, advancements, and the transformative potential of AI for businesses. Stay informed of trends and applications.
Showing results for 
Search instead for 
Did you mean: 
Product and Topic Expert
Product and Topic Expert
NOTE: The views and opinions expressed in this blog are my own

In this blog I will demonstrate how you can quickly build a custom LLM Chatbot Demo with Gradio in under 100 lines of code, running on SAP BTP. Gradio's agility makes it a quick solution to present your machine learning models and share with colleagues.

Gradio is commonly used to demonstrate open source models on Hugging Face.

For an example see Meta's recent opensource Llama2 chatbot.


If you get to the end of this blog you will be able to build the following LLM Chatbot demo on BTP, using OpenAI API's:

Click to Enlarge


The pre-requistes are:

  • BTP Account..  (even a free trial account works)

  • Cloud foundry (cf) cli installed

  • OpenAI API key

  • Basic Python knowledge


The build includes the steps to make the following:

openai_service Cloud Foundry User Provide Service to store your OpenAI API Key
manifest.yaml Details of the App, need for deployment to Cloud Foundry
requirements.txt Python library dependencies 100 lines of code

NOTE: in this example I've focussed on deploying on the BTP CF runtime environment, but with some minor tweaks to the steps the Kyma (K8S) runtime could also be used.




Create the following:
#Command line (Mac flavour)

cf cups openai_service -p '{"username":"api_key","password":"<your OpenAI API Key goes here>"}'


In BTP you should see something like the following.

Click to Enlarge



In your local machine create a file with the following:
- name: llm-chatbot
random-route: true
path: ./
memory: 128M
buildpack: python_buildpack
command: python
- openai_service




In your local machine create a file with the following:

In your local machine create a file with the following:
import os, cfenv
import gradio as gr
import openai

port = int(os.environ.get('PORT', 7860))
print("PORT:", port)

# Create a Cloud Foundry environment object
env = cfenv.AppEnv()

# Check if the app is running in Cloud Foundry
# Get the specified user-provided service by name
service_name = "openai_service"
service = env.get_service(name=service_name)

# Access the service credentials
credentials = service.credentials

# Access the "openai_api_key"
openai_api_key = credentials['password']
os.environ['OPENAI_API_KEY'] = openai_api_key

openai.api_key = openai_api_key

print("OpenAI API Key assigned")
except cfenv.AppEnvError:
print(f"The service '{service_name}' is not found.")
print("The app is not running in Cloud Foundry.")

prompt = " "

#Call the LLM Api's with a prompt/question and return a response
def generate_response(llm, prompt):
if llm == 'OpenAI':
completion = openai.Completion.create(
model = "text-davinci-003",
prompt = prompt,
temperature = 0,
max_tokens= 500,
stop=[" Human:", " AI:"]
return completion.choices[0].text
return "Unknown LLM, please choose another and retry."

#Handle the Chatbox call back
def llm_chatbot_function(llm, input, history):
history = history or []
my_history = list(sum(history, ()))
my_input = ' '.join(my_history)
output = generate_response(llm, my_input)
history.append((input, output))
return history, history

#Define the ChatBot using Gradio Elements
def create_llm_chatbot():
with gr.Blocks(analytics_enabled=False) as interface:
with gr.Column():
top = gr.Markdown("""<h1><center>LLM Chatbot</center></h1>""")
llm = gr.Dropdown(
["OpenAI", "Llama-2-7b-chat-hf"], label="LLM Choice", info="Which LLM should be used?" , value="OpenAI"

chatbot = gr.Chatbot()
state = gr.State()
question = gr.Textbox(show_label=False, placeholder="Ask me a question and press enter.")
with gr.Row():
summbit_btn = gr.Button("Submit")
clear = gr.ClearButton([question, chatbot, state ])

question.submit(llm_chatbot_function, inputs=[llm, question, state], outputs=[chatbot, state]), inputs=[llm, question, state], outputs=[chatbot, state])

return interface

llm_chatbot = create_llm_chatbot()

if __name__ == '__main__':



Deploy the App to SAP BTP

With the above service and files created your are ready to deploy the app.

From the same directory you created the files, run the following:
cf push


In BTP you should now see your app running:

Click to Enlarge


You can find  the url (application route) to your app when you navigate further on the llm-chatbot:

Click to Enlarge



Where to Next?

As you can see it only takes a few simple steps to deploy a machine learning demo app onto the BTP platform. Perfect for iterative testing and development.

So where do you you think it make sense to explore this further?

Ideas I have are:

  • Add additional Open Source LLM's

  • Add Human Feedback to the Chatbot, to be used with fine tuning of open source models

  • Add user level security (application routing)

  • Integrating with Corporate Data


I welcome your additional ideas in the comments below.


SAP notes that posts about potential uses of generative AI and large language models are merely the individual poster’s ideas and opinions, and do not represent SAP’s official position or future development roadmap. SAP has no legal obligation or other commitment to pursue any course of business, or develop or release any functionality, mentioned in any post or related content on this website.