How to (Easily) Run Local LLMs on Your Computer 💻

5-min Read | Advertise here

Privacy is important in AI.

That’s why I found a way to run an LLM (Large Language Model) on my local computer.

For free.

Here’s how to do it:

I’ve always been cautious about uploading private information to ChatGPT, Gemini, or any AI model online.

There’s one big concern: data privacy.

These big tech companies use the data we input to train their models, so I never upload any sensitive content.

But there has been a compelling solution: locally run LLMs.

In this segment, I’ll explore a new open-source interface: LM Studio.

From its beautiful interface, you can discover open-source models, download them, and run them locally on your computer or a localhost.

But does LM Studio collect any data?

No. One of the main reasons for using a local LLM is privacy, and LM Studio is designed for that. Your data remains private and local to your machine.

Let me show you how.

Step 1: Download LM Studio

The first thing you need to do is head to LM Studio’s website and download it.

It’s open-source, which means it’s free.

Step 2: Search and Download an LLM

In this tutorial, I’ll just search for “Mistral”.

It will bring up a list of Model Cards with Mistral in it, which are like repositories for LLMs from an open-source ML sharing website called Hugging Face.

I picked this one by TheBloke.

On the right, you’ll see a bunch of files displayed on the Model Card.

You can also view the Model Card in your browser (Huggin Face) by clicking on a button at the top right.

These files differ in their quantization, which determines the fidelity of the results.

LM Studio will make a compatibility guess based on your machine’s specs.

Any LLMs with the words “FULL GPU OFFLOAD POSSIBLE” in green mean that you can probably run them.

I downloaded this one: mistral-7b-instruct-v0.1.Q3_K_S.gguf

As the file sizes are huge, give it some time to download.

Step 3: Initialize Model and Configure Settings

Click on the AI Chat bubble on the left panel.

Select a model to load at the top. I’ll select the model I just downloaded.

The model will take a while to load.

Before we start, please click on Markdown at the top so that you can get pretty text responses.

Next, on the right, you can set a system prompt.

You can do some advanced configuration for these parameters:

• Context length

• Temperature

• Tokens to generate

and other settings.

I’ll leave it as default, but you can play around with them to get your desired result.

Step 4: Chat with LLMs Offline

Let’s get chatting!

I entered this prompt and hit enter:

Here’s its response:

Seems like the responses aren’t amazing (compared to GPT-4o), but it works great!

What I’d do next is try out some newer open-source LLMs by Meta (LLaMa 3) or choose a model with a higher quantization number.

Also, note that your chats will be saved on the left side, similar to ChatGPT’s interface!

Have fun running this model for your private use!

Feel free also to explore using your models on a local host or a multi-model session in the Playground tab.

This tutorial was adapted from this YouTube video:

That’s all for this week.

See you next time!

Justin Chia

Find me on LinkedIn or shoot me an email at [email protected]

If you have any questions or feedback, reply to this email. (I'll personally reply! :)