Introduction

This is part 2 of a tutorial series on Ollama and using local LLM models.

Read part 1 here: Installing Ollama and Open WebUI

In part 1 of this tutorial series, we learned how to install Ollama and Open WebUI - this allowed us to download and use open-source models on our own machine. Now that we have the installation completed, lets understand what are models, and how models work.

We will also explore the different types of models and the system requirements needed to run them effectively. This is extremely important because not all models are the same - some models need more computational power than others, depending on their size and complexity.

What is a Model in Machine Learning?

Think of a model as a trained artificial brain that works somewhat similar to a human brain in the context of understanding a subject and being able to recollecting it later. A model contains knowledge about what it has learned during its training phase and it is capable of reproducing this information based on new queries or inputs.

It is like teaching a person how to write software, and then asking that person to write software to do some new tasks after the training has completed. For example, when you enter a prompt in ChatGPT, the prompt is processed by a GPT model. This model has been trained to generate large amounts of text on different subjects.

What Exactly is an LLM Model?

A Large Language Model (LLM) is a type of machine learning model specifically trained to understand and generate human-like text. These models are trained on vast amounts of text data from books, websites, and other sources. The goal of these models is simple - to predict some text based on a prompt we send to it.

When given a prompt, LLMs generate a response based on the data they’ve been trained on. The larger the model, the more data it has been trained on and the better it is at producing human-like text.

An LLM is like a sophisticated text generator that can help you in a variety of tasks like writing reports, articles, code etc.

Different Types of LLM Models

LLM models come in various types, depending on the tasks they’re designed to handle. Here are the main types:

Text Models

Text models are the most common type of LLMs. These models focus on generating and understanding text. They are used in applications like chatbots, content generation, translation, summarization etc. Examples include opensource models like Llama3.2 1B and 3B, Qwen2.5, Mistral 7B, Gemma 2 etc.

Multimodal Models

Multimodal models are more advanced - they can understand not just text, but also other types of data like images, video and audio. This makes them suitable for tasks like describing images, generating captions, or answering questions about a video. Examples of opensource multimodal models are Llama 3.2-vision, Phi-3.5-vision etc.

Note There are thousands of opensource text and multimodal models that are available to run offline on our own machines. You can explore many of these models on huggingface.co.

Model Sizes (e.g., 3b, 8b, 70b)

LLM models are categorized by their size, which is measured by the number of parameters within the model (represented by "b" for billions). For example, a 3b model has 3 billion parameters, while a 70b model has 70 billion.

The size of a model defines its capability and the system resources required to run it. Large models (like 70b or bigger) are great at understanding and generating complex language across a variety of subjects, but they also require more computational power and RAM. Smaller models (like 1b and 3b) are lightweight and can run on lower-end systems or even mobile devices, but might not be as accurate in handling complex queries.

Due to the variety of model sizes and their varied capabilities, it can be difficult to choose which one to use for a specific use case. One way to understand which model to use is by looking at and comparing their performance benchmarks for our use case. You can read this comprehensive guide to understand more about how to make sense of the benchmarks published by many open source models.

System Requirements to Run Different Models

The hardware needed to run an LLM model depends on the model’s size. Here’s a general breakdown of system requirements based on model size:

Small Models (e.g., 3b, 7b)

RAM: These models can run on systems with as little as 8GB of RAM.

CPU: A decent multi-core CPU is sufficient. However, performance will improve significantly if you have a dedicated GPU or an NPU.

Storage: Usually between 1GB to 10GB, depending on the model.

Medium-Sized Models (e.g., 13b, 30b)

RAM: Atleast 16GB to 32GB of RAM is required to handle models of this size.

CPU/GPU: A strong multi-core CPU is required, and a GPU with at least 8GB of VRAM is recommended for better performance.

Storage: These models are much larger and might require between 15-50GB of storage.

Large Models (e.g., 65b, 70b)

RAM: These models require 64GB or more RAM to run effectively.

GPU: A powerful GPU with at least 24GB of VRAM is essential to run models of this size.

Storage: You might need more than 80GB of free space depending on the model.

The general rule is, the larger the model, the more hardware resources you will need. You can consider upgrading your system or using cloud-based services if you’re working with very large models. If you are in an enterprise environment, you can setup dedicated servers to run large models.

Managing Models in Ollama

Ollama provides a simple command line based interface for managing local models. Here are the basic commands you can use to manage local models:

Pulling Models

To download (or pull) a model, use the following command:

BASH

Copy code

ollama pull <model_name>

For example, to pull the Llama3.2 model, we used this command in part 1 of this tutorial series:

BASH

Copy code

ollama pull llama3.2

You can visit ollama.com/library to explore various models that you can pull and use on your machine.

Running Models

Once a model is pulled, you can start using it. Run the following command to load the model in memory and interact with it:

BASH

Copy code

ollama run <model_name>

You can also interact with a model through Ollama’s API or simply use Open WebUI as we did in part 1 of this tutorial series.

Upgrading Models

To upgrade a model to a newer version, just use the pull command again. Ollama will automatically check for updates and download the latest version of the model.

BASH

Copy code

ollama pull <model_name>

Deleting Models

If you no longer need a model or want to free up disk space, you can delete a model by running the following command:

BASH

Copy code

ollama delete <model_name>

Working with Different Models in Open WebUI

Open WebUI gives us a nice web-based user interface to work with both, text and multimodal models.

Using Text Models

Once you’ve pulled a text model through Ollama, you can start chatting with it in Open WebUI. In the dropdown menu where you select a model, choose the text model you’ve installed (e.g., llama3.2) and start a conversation by writing a prompt.

Open WebUI lets you interact with the model just like you would with ChatGPT. You can ask questions, get summaries, write articles etc - all securely within your local environment.

Using Multimodal Models

If you’ve pulled a multimodal model, you can interact with it in a similar way. Multimodal models in Open WebUI can process inputs like images alongside text. For example, in the prompt input, you can upload an image and ask the model to describe it or answer questions about the image.

Conclusion

In this tutorial, we explored the basic concepts behind models in machine learning, as well as an overview of Large Language Models (LLMs), their sizes and system requirements. We also walked through how to manage these models using Ollama.

With the addition of Open WebUI, working with text and multimodal models becomes very easy, allowing you to make the best use of these models locally on your machine.

By understanding these concepts and tools, you can explore and leverage open-source models for a variety of different tasks within an organizational environment securely, and without concern of leaking confidential information.

System Requirements for Working with Text and Multimodal Models