Build an LLM Chatbot with OpenAI API and Ollama – Step by step Tutorial

I this tutorial, you will learn how to build an LLM agent that would run locally using various tools. First we’ll build a basic chatbot the just echoes the users input. Next would enhance it to use OpenAI API and finally we’ll further refine it to used LLM running locally.

Tools

• Streamlit framework for basic Web UI
• Ollama for downloading and running LLMs locally
• OpenAI API for making requests to OpenAI API

We would cover the following

1. Setup Ollama Locally and Test
2. Create a StreamLit Project
3. Create the Basic LLM
4. Modify to Make API Call to OpenAI
5. Modify to use LLM Locally (mistral and llama3.2
6. Enhance the UI with Styles

1. Setup Ollama Locally and Test

What is Ollama? Ollama is an open-source tool that lets you run large language models (LLMs) locally on your own machine, removing the need for cloud-based services.

How to setup Ollama

macOS

brew install ollama

Linux (Ubuntu/Debian)

curl -fsSL https://ollama.com/install.sh | sh

Windows

• Download the installer from: https://ollama.com/download
• Run the .exe file and follow the installation prompts.

Run a Model

After installation, open a terminal and run:

ollama run llama3

This command:

• Downloads the model (first time only)
• Launches a chat session locally
• You can start typing prompts directly

2. Create a StreamLit Project

1. Set Up Your Environment

a. Create a new project folder

mkdir basic-app

cd basic-app

b. (Optional) Create a virtual environment

python -m venv venv

source venv/bin/activate


# On Windows
venv\Scripts\activate

2. Install StreamLit

pip install streamlit

3. Create Your First App

Create a Python file named app.py:

import streamlit as st
st.title("My First Streamlit App")

st.write("Hello, world!")

st.text("This is a simple text output.")
# Add an interactive slider

number = st.slider("Pick a number", 0, 100, 25)

st.write("You picked:", number)

4. Run the App

In your terminal:

This will open a browser window at http://localhost:8501 showing your app.

3. Create the Basic LLM Chatbot

Create a new file called basic.py (you can give it any name). Add the following code inside:

st.title("My Basic Chatbot")

if "messages" not in st.session_state:
    st.session_state.messages = []

for message in st.session_state.messages:
    with st.chat_message(message["role"]):
        st.markdown(message["content"])

if prompt := st.chat_input("What's up"): # Get user input
    with st.chat_message("user"):
        st.markdown(prompt)
    st.session_state.messages.append({"role": "user", "content": prompt})
    response = f"Echo: {prompt}"
    
    with st.chat_message("assistant"):
        st.markdown(response)
    st.session_state.messages.append({"role": "assistant", "content": response})

In the command line enter the command below to run the chatbot:

This will open a browser window at http://localhost:8501 showing your app and you can test that the bot works

4. Modify to Make API Call to OpenAI

In the previous section, we created a basic chatbot that just echoes the user input.  Let’s now enhance it to return actual response by call to OpenAI API.

• Create a new file and use the content from this file (you can get the complete project in the Github repo)
• Create a new directory in your project called .streamlit
• Inside this directory, create a file called secrets.toml.
• Obtain your OpenAI API key and add it to the secrets.toml file (see the video if you have any issues)
• Run the file as before and interact with the chatbot

5. Modify to use LLM Locally (mistral and llama3.2)

Create a new file called chatbot_with_ollama_v1.py and add the content. Get the content from the Github Repo as well.

Notice the key difference: instead of calling client.completions.create(), we now make a post request directly using the requests library. You will have to install this.

with st.spinner("Thinking..."):
    response = requests.post(
        "http://localhost:11434/api/chat",
        json={
            "model": MODEL_NAME,
            "messages": [{"role": m["role"], "content": m["content"]} for m in st.session_state.messages],
            "stream": False  # Set to True for streaming support (more complex handling)
        }
    )

You can then run the file as before.

6. Enhance the UI with Styles

The final modification is adding some stying to the UI. Get the file from here (available in the repo)