My Hybrid AI System in Obsidian: Local and Cloud AI Integration

Using AI with notes gives superpower to your notes. But should you give AI access to your knowledge base?
Warp Terminal

AI is rapidly gaining popularity across various fields, and I've found that integrating it into a personal knowledge management system like Obsidian offers far more benefits than one might initially expect.

In this article, I'll show you exactly how I've enabled both local AI and cloud AI within Obsidian, configuring them for different vaults to suit my specific needs.

How I Choose AI

AI technology, while powerful, often comes with inherent privacy concerns.

When you opt for a corporate AI solution, which typically offers faster queries and results, you're risking your privacy.

On the other hand, if you decide to use local AI, it can be quite taxing and slow, especially on moderate hardware.

Given this dilemma, I asked myself: why not use both? With Obsidian, setting up such a workflow is surprisingly easy.

I personally use a local AI for my most private notes and journals, ensuring my personal thoughts remain entirely on my machine.

For my work-related documents and learning journeys, where speed and broader knowledge access are key, I leverage Gemini chat (because it's free). This approach gives me the best of both worlds: privacy where it matters most, and powerful cloud capabilities for everything else.

If you want more segregation, have separate vaults for your work and personal notes.

📋
It's worth noting that for local AI to work seamlessly and efficiently, a decent GPU is a crucial component.

Use Copilot for Obsidian extension

Don't go by the name. It's not exclusive to Microsoft Copilot.

To integrate AI directly into Obsidian, I rely on Copilot for Obsidian, a fantastic community extension. This plugin supports both local and cloud AI, and it even offers its own Plus features for those who want to enhance their experience further.

Here's how to install this extension in Obsidian:

Go to the Obsidian settings by pressing CTRL+,. Inside the settings, go to Community Plugins -> Browse.

Click on Browse inside Community Plugins tab in Obsidian settings.
Click on Browse

Search for Copilot and install the extension.

Search and install the Copilot extension for Obsidian.
Install Copilot

Enable it right away using the Enable button.

Setting Gemini AI in Obsidian

Now that Copilot for Obsidian is installed, let me show how I configure Gemini AI within Obsidian.

Create API Key

Go to the Google AI Studio website. Click on Create API key button.

Click on Create API key in the Google AI Studio webpage.
Create API Key

Here, give the key a name and click on Create a New Project.

Give a name for the API key and then create a new project.
Create a New Project

Give the project a name and click on Create Project

Give a name to the Project and click on Create Project button.
Create Project with a name

Now, click on Create Key.

Create API key within the project.
Click on Create Key

Use the Copy key button to copy the API key.

Click on Copy key in Google AI studio to copy the key details to clipboard.
Copy the API Key

Configure Obsidian Copilot

To configure Copilot, open Obsidian and go to settings (CTRL+,). Here, in the sidebar, scroll down and select Copilot.

In the copilot settings, go to the Basic tab, click on Set Keys button.

In the Copilot settings, under Basic settings, click on Set Keys button.
Click on Set Keys

Here, paste the copied API key into the designated field and then select your preferred Gemini model from the dropdown. I have added the Gemini 2.5 Flash model in this particular setup.

Add API key and add a model using the dropdown list.
Add key and model

Set the default chat model to the added one.

Set the default chat model in Obsidian Copilot
Set default chat model

By now, Copilot should be ready and available in the ribbon menu. Simply click on its icon there to open it in the right sidebar.

Copilot in sidebar active and available to chat.
Copilot in Sidebar

Follow the same setup for other AI providers like Claude, GPT, etc.

Setting up Local AI

The Copilot plugin isn't just for cloud AI; it can also be used to set up local AI directly within Obsidian.

🚧
Again, Local AI is a resource-intensive process. It's recommended to use higher models only if you have a dedicated GPU and sufficient RAM available.

Here, I'll explain how to get local AI working using Ollama.

Install Ollama

Ollama is a command-line tool that allows us to download and run open-source Large Language Models (LLMs) like Llama3, Phi-3, Mistral, CodeGamma, and many others, right on my local machine.

To run AI locally on Linux with Ollama, you should have a GPU on your system. While it's possible to run AI on a CPU, the experience won't be ideal. A GPU is highly recommended for better performance. If you have a TPU/NPU, that would be even better but I am not sure if Ollama would support all TPU/NPU.

For this tutorial, I'll be using the Llama3.2 3B model. Even though it can be taxing on a CPU-only machine, it still offers noticeably faster performance compared to larger models without a dedicated GPU.

You can, of course, choose other, heavier models based on your system's specifications.

To install Ollama, use the command:

curl -fsSL https://ollama.com/install.sh | sh

Once Ollama is installed, it's a good idea to check if it's running correctly. By default, it operates on a specific port number on localhost.

To verify its status, open a web browser and enter:

localhost:11434

It should show the message, "Ollama is running".

Install llama3.2 3b model

Now that you have Ollama up and running, install the llama3.2 3b model using the command:

ollama run llama3.2:3b

This will download almost 2GB+ data and enable the model. Once the command execution is completed, you will be automatically entered to a llama prompt, where you can chat with the model from terminal.

To exit the prompt, use the command:

/bye

To stop the model, use the command:

ollama stop llama3.2:3b

Note the name of the AI

In a terminal, run the ollama command:

ollama list

This will list the installed models in your system. Copy the name of the llama3.2 model exactly as shown in the output.

Copy the exact name of the llama model from ollama list command.
Ollama List Models

Configure Obsidian Copilot

Open Obsidian and go to the Copilot settings as done for the Gemini method.

Here, go to the Model tab and select Add Model button.

Click on the Add model to add a new local AI model to the Obsidian Copilot.
Click on Add Model

Fill the fields appropriately. The Model Name, marked with a red asterisk, should be exactly the name you copied in the last step.

The Display name can be any identifiable name you choose. Set the provider to Ollama, and you can leave the Base URL and API key fields as their default values; there's no need to add anything there.

Add the Ollama llama model in Obsidian Copilot.
Add Ollama Model


You can enable Vision in Model Capabilities if you want to allow image additions to the chat.

Once you've filled everything out, you can press the Test button to check for any configuration errors. If no errors are reported, use the Add Model button to add this model to your available list.

Now, go back to the Basic tab and set the default chat model to Llama.

That's it! From now on, you can use the local AI to chat with your notes.

Using the AI interface

Copilot for Obsidian comes packed with a lot of cool features. Let's take a look at some of the cooler ones that you may find yourself using frequently.

Context aware chat

As we've seen, Copilot features a neat chat interface that appears in the right sidebar when enabled. This sidebar offers several cool options that can significantly improve your notes.

For example, by default, the current note is automatically selected as the context for your chat. However, by pressing the @ symbol, I can easily select multiple notes to serve as context for my conversation, allowing for more comprehensive and interconnected AI interactions.

Chat with multiple notes

Send images in chat

At times, you may want to chat along with a screenshot. For example, when I am writing some configs, I can take a screenshot of an error and ask the LLM what the error is right then and there.

If the model supports processing images, you can do this directly via the sidebar interface. This is shown in the recording below.

Chat with Images

Add chat to note

What is the use of an AI assistant if it cannot add content directly to your notes? Copy and paste is not always a good solution.

To solve this, each result in the sidebar has an Insert/Replace at cursor button. Clicking on it immediately adds the chat result to the exact place where your cursor is currently located in your note or document, making the workflow incredibly seamless.

Adding Chat Content to Note

Chat backups

Copilot extension adds all chat sessions to a dedicated folder (copilot/copilot-conversations) in your vault as standard Markdown documents. The chat file is cleanly structured in a simple format:

user: user's prompt
ai: ai reply

The best thing about this setup is that it is properly backlinked. A backlink to the chat note is available right within your active note, which is incredibly good for getting a quick refresh on your thought process later on.

Wrapping Up

Using AI in a note-taking application could be a controversial choice, and it is completely a user's personal choice. While AI can hallucinate and add wrong data, I never suggest someone use it for research purposes.

The core learning and note-taking should always be done manually, but you can leverage the power of AI to add some helpful context to your note. Also, you can use it to format your note into a better Personal Knowledge Management (PKM) format, as doing that completely manually is often just an unwanted burden.

Do you prefer using AI in your Obsidian? Feel free to mention your thoughts in the comments section.

If this helped you, consider supporting It's FOSS

It's FOSS has been helping people use Linux for the past 14 years. Help us stay independent from big tech. Become a Plus member, enjoy ad-free reading and get 5 eBooks.

Plus yearly

Ad-free, FREE ebooks

Join yearly

Buy us a coffee

Any amount, no commitment

Support on Ko-fi
About the author
Sreenath

Sreenath

Linux tinkerer by day, Lord of the Rings fanatic by night. When I'm not busy conquering the command line, you can find me kicking back and watching some football while quoting JRR Tolkien

Become a Better Linux User

With the FOSS Weekly Newsletter, you learn useful Linux tips, discover applications, explore new distros and stay updated with the latest from Linux world

itsfoss happy penguin

Great! You’ve successfully signed up.

Welcome back! You've successfully signed in.

You've successfully subscribed to It's FOSS.

Success! Check your email for magic link to sign-in.

Success! Your billing info has been updated.

Your billing was not updated.