Our organization does not allow ChatGPT

When I ask participants in my Generative AI training workshops if they use ChatGPT for their daily work, this is a common response.

While ChatGPT has some the best performing LLM models that can boost productivity by 30-50% in most tasks, it's unfortunate that many organizations block access to it (though there are valid reasons for this, which I'll cover in another article).

On a side note, if your organization uses Microsoft Enterprise licenses, you might have access to Microsoft Copilot, which uses GPT models from OpenAI. So, in a way, you're still getting access to the premium version of ChatGPT.

Overview

Read part 2 of this tutorial series here: System Requirements for Working with Different Types of Models.

This tutorial is for those who don’t have access to online LLMs like ChatGPT, Gemini, Claude, etc., but are still interested in using LLMs to boost productivity (there’s a lot you can do with LLMs—just don’t tell your boss!).

Several open-source LLMs (e.g., Llama 3, Mistral, Qwen) have performance very close to GPT models. These models can run completely offline on your local machine - no internet required.

Plus, with these models running locally, there are no security concerns about confidential data leaving your organization's firewall.

Who Is This Tutorial For

Anyone with a PC or laptop can follow this tutorial. All you need to know is:

How to download and install software.
How to copy-paste and run commands in a terminal or command prompt.

By the way, if you're a non-technical professional, don’t worry about the API stuff I talk about below. Just follow the installation steps, and you are good to go.

Software Required

To use open-source LLMs locally, we'll need to install a few software tools:

1. Ollama: Think of Ollama as an app store for LLMs. It lets you download and manage LLM models and run them on your computer. Ollama is a wrapper (aka LLM runner) that helps us to manage and use these LLM models.

If you're a software developer, Ollama has an API to integrate LLM models with other systems. Did I mention Ollama is fully open source?

2. Open WebUI: While Ollama is a wrapper, Open WebUI provides a user-friendly ChatGPT-like interface to interact with the local LLM models. It communicates with Ollama through its API.

Open WebUI can also connect to online LLMs like ChatGPT via the OpenAI API (you’ll need an OpenAI API key for this). Open WebUI has many other features (check out the website), but we'll stick to the basics in this tutorial.

3. Docker: Docker allows you to run apps in isolated environments so that they don’t affect other stuff on your computer. Open WebUI is available as a Docker container, so we'll need to install Docker to run it.

Alright, let’s get started and install these 3 tools.

Software Installation

Install Docker

1. Visit Docker website, download Docker Desktop, and install it.

2. Verify the installation by launching Docker Desktop - you should see a UI like this:

Install Ollama

1. Visit Ollama website, click the "Download" button, and follow the installation instructions for your OS.

2. Verify the installation by opening a terminal and typing ollama. You should see an output similar to this:

Start Ollama Service

In a command prompt, run the command ollama serve to start the Ollama service, allowing Open WebUI to communicate with it.

Note: If the service is already running, you might see an error message like below. This is fine and you can ignore it.

Error: listen tcp 127.0.0.1:11434: bind: Only one usage of each socket address (protocol/network address/port) is normally permitted.

Install Open WebUI

Open WebUI is available as a Docker image, so you'll need to run the following command to download and launch it. You can find more installation instructions in the OpenWeb UI GitHub repository.

Open the command prompt and run:

BASH

Copy code

docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:main

If you're a developer using frameworks like NextJS or SvelteKit, be aware that these often use port 3000 by default. To avoid port conflicts, change port 3000 in the command above to another free port like 5000 (e.g., I run Open WebUI on port 6001 as you see in the screenshot above).

Download and Run LLM Models

Launch Open WebUI

Open your browser and go to http://localhost:3000 (or whatever port you changed to, e.g., http://localhost:6001).

On first launch, you'll see the Open WebUI login page:

Create an account. The first account you create becomes the administrator.

Open WebUI stores all your information locally on your machine. No data is sent to any external server, ensuring complete privacy.

Downloading an LLM Model

When you installed Ollama, you only installed the wrapper without any models. Verify this by running the below command in terminal:

BASH

Copy code

ollama list

You should see an empty list.

There are a few things to consider when choosing which models to download use, but we’ll cover that in Part 2.

For now, let’s download the llama3.2 3b model, a good choice that runs well on laptops with 8GB or more RAM.

There are two ways to download the model:

Option 1: Run the below command in terminal:

BASH

Copy code

ollama pull llama3.2

Option 2: In Open WebUI, click your account icon (top right) > Admin Panel > Settings > Models section. Enter llama3.2 in the “Pull a model from Ollama.com” box.

I’ll use the command line to download the model:

Start Chatting

Return to Open WebUI in your browser and start a new chat. You you should see the downloaded models listed in the Select a Model dropdown.

Choose a model and enter a prompt like:

Tell me a joke

That should get back a response from the LLM model:

Conclusion

The steps we covered in this tutorial are a one-time setup. Once done, you can download many other models and chat with them. You can view all the models available in Ollama here.

Note: Not all models can run on all machines. In Part 2, we'll explore the different types of models and what hardware configuration is needed to run them.

Run AI Models Locally and Securely with Ollama: No Internet Needed!