# New Plug: OpenAI / LLM AI integration
**URL:** https://community.silverbullet.md/t/new-plug-openai-llm-ai-integration/211
**Category:** Plugs and Libraries
**Created:** [February 20, 2024, 3:11am UTC](https://community.silverbullet.md/t/new-plug-openai-llm-ai-integration/211 "2024-02-20T03:11:48Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@justyns](https://community.silverbullet.md/u/justyns)
#### Post date: [February 20, 2024, 3:11am UTC](https://community.silverbullet.md/t/new-plug-openai-llm-ai-integration/211/1 "2024-02-20T03:11:49Z")
Hello!
This is something I’ve been working on for a few days and have already shared on the discord (thanks for the feedback there!). I feel like this plug is at a point where it’s usable enough if others want to try it out:
(Please backup your space! This does insert and replace text, so better to be safe than risk your data)
> **[GitHub - justyns/silverbullet-ai: Plug for SilverBullet to integrate LLM...](https://github.com/justyns/silverbullet-ai)**
>
> Plug for SilverBullet to integrate LLM functionality
It’s still pretty early, but here are a few things it can do:
- Create and insert a summary of either selected text or the whole note
- Generate tags for a note and set them in the frontmatter
- Generate images using Dall-E and save them locally
- Send selected text (or the note) and use it as a user prompt, putting the response in the open note
- Have an interactive user/assistant chat inside of a note so you can keep your chat history in SB
- Support for changing the openai base url, allowing local LLMs - I’ve only tested this briefly with Ollama so far
The readme has a list of available commands available through the command palette. There are no slash commands or keyboard shortcuts yet, but I’m open to suggestions for them.
Here’s an example of the chat feature:
On a side-note: I’m not really a javascript dev, so I’d be more than happy if anyone wants to help make it better. Feel free to suggest changes/features/etc too.
---
### Author:  [@zef](https://community.silverbullet.md/u/zef)
#### Post date: [February 20, 2024, 7:05am UTC](https://community.silverbullet.md/t/new-plug-openai-llm-ai-integration/211/2 "2024-02-20T07:05:15Z")
This is very cool! Thanks for building this!
---
### Author:  [@paletochen](https://community.silverbullet.md/u/paletochen)
#### Post date: [February 20, 2024, 7:28am UTC](https://community.silverbullet.md/t/new-plug-openai-llm-ai-integration/211/3 "2024-02-20T07:28:59Z")
Amazing!
Thanks for doing this.
How difficult/easy would it be to adjust this Plug to use Gemini?
---
### Author:  [@nikolasdi](https://community.silverbullet.md/u/nikolasdi)
#### Post date: [February 20, 2024, 12:58pm UTC](https://community.silverbullet.md/t/new-plug-openai-llm-ai-integration/211/4 "2024-02-20T12:58:08Z")
Would this work with Mistral’s API?
---
### Author:  [@zef](https://community.silverbullet.md/u/zef)
#### Post date: [February 20, 2024, 1:43pm UTC](https://community.silverbullet.md/t/new-plug-openai-llm-ai-integration/211/5 "2024-02-20T13:43:20Z")
The OpenAI API seems to be more or less a standard for other LLMs to implement, so just check if they’re OpenAI compatible.
Llamafile implements it for instance: [GitHub - Mozilla-Ocho/llamafile: Distribute and run LLMs with a single file.](https://github.com/Mozilla-Ocho/llamafile)
---
### Author:  [@nikolasdi](https://community.silverbullet.md/u/nikolasdi)
#### Post date: [February 20, 2024, 2:03pm UTC](https://community.silverbullet.md/t/new-plug-openai-llm-ai-integration/211/6 "2024-02-20T14:03:33Z")
I just checked and it is. I must be setting up something wrong in SETTINGS or SECRETS.
---
### Author:  [@justyns](https://community.silverbullet.md/u/justyns)
#### Post date: [February 20, 2024, 4:38pm UTC](https://community.silverbullet.md/t/new-plug-openai-llm-ai-integration/211/7 "2024-02-20T16:38:54Z")
It doesn’t look like Gemini has an openai compatible api, so it wouldn’t be supported right now.
Maybe you could try using the proxy server from [GitHub - BerriAI/litellm: Call all LLM APIs using the OpenAI format. Use Bedrock, Azure, OpenAI, Cohere, Anthropic, Ollama, Sagemaker, HuggingFace, Replicate (100+ LLMs)](https://github.com/BerriAI/litellm) ? Docs: [Quick Start | liteLLM](https://docs.litellm.ai/docs/proxy/quick_start)
I haven’t used it, but it says it supports Gemini.
Overall I’d rather support one type of API, but I do plan on adding support for configuring multiple models and switching between them, so I’ll do some planning around how that could extend to different APIs too if there’s demand for it
---
### Author:  [@justyns](https://community.silverbullet.md/u/justyns)
#### Post date: [February 20, 2024, 4:47pm UTC](https://community.silverbullet.md/t/new-plug-openai-llm-ai-integration/211/8 "2024-02-20T16:47:37Z")
[Mistral’s API](https://docs.mistral.ai/api/) does look like it should be compatible. I don’t have an account to test with yet, but what error(s) are you getting?
In SECRETS, you should only need
```yaml
OPENAI_API_KEY: "mistral api key here"
```
And in SETTINGS, it should have:
```yaml
ai:
defaultTextModel: gpt-4-0125-preview
openAIBaseUrl: https://api.mistral.ai/v1
```
You’ll have to change defaultTextModel too.
And once you change either of those files, you’ll probably need to run the “System: Reload” command and/or do a hard refresh so it picks up the new settings.
---
### Author:  [@nikolasdi](https://community.silverbullet.md/u/nikolasdi)
#### Post date: [February 20, 2024, 7:03pm UTC](https://community.silverbullet.md/t/new-plug-openai-llm-ai-integration/211/9 "2024-02-20T19:03:32Z")
> [@justyns](#):
>
> `https://api.mistral.ai/v1`
I tried Mistral using these settings:
```yaml
ai:
defaultTextModel: mistral-medium
openAIBaseUrl: https://api.mistral.ai/v1
```
and Ollama with these:
```yaml
ai:
defaultTextModel: phi
openAIBaseUrl: http://127.0.0.1:11434
```
For Mistral, does the SECRETS page has to look like this:
```yaml
OPENAI_API_KEY: "aaaaaaaaaaaaaaaaaaaaaaaa"
```
---
### Author:  [@nikolasdi](https://community.silverbullet.md/u/nikolasdi)
#### Post date: [February 20, 2024, 7:07pm UTC](https://community.silverbullet.md/t/new-plug-openai-llm-ai-integration/211/10 "2024-02-20T19:07:53Z")
When trying the chat command I get “page does not match the required format for a chat”.
Other commands show a “Contacting LLM, please wait” but nothing happens.
---
### Author:  [@justyns](https://community.silverbullet.md/u/justyns)
#### Post date: [February 21, 2024, 6:23am UTC](https://community.silverbullet.md/t/new-plug-openai-llm-ai-integration/211/11 "2024-02-21T06:23:45Z")
Thanks for trying it out! I signed up for a mistral.ai account and was able to get it working, using the same settings as you:
SETTINGS:
```yaml
ai:
defaultTextModel: mistral-medium
openAIBaseUrl: https://api.mistral.ai/v1
```
SECRETS:
```yaml
OPENAI_API_KEY: "xxxxx"
```
Right now, the only thing required in SECRETS is that `OPENAI_API_KEY`. Even for Ollama.
One thing I noticed is that running the “System: Reload” command doesn’t seem to be enough for settings changes to take effect. It seems like I need to run both “System: Reload” and then do a hard refresh of the page I’m on. Can you give that a try?
For Ollama, can you try these settings (with the /v1 on the end of the url)?
```yaml
ai:
defaultTextModel: phi
openAIBaseUrl: http://localhost:11434/v1
```
---
### Author:  [@justyns](https://community.silverbullet.md/u/justyns)
#### Post date: [February 21, 2024, 6:26am UTC](https://community.silverbullet.md/t/new-plug-openai-llm-ai-integration/211/12 "2024-02-21T06:26:34Z")
> [@nikolasdi](#):
>
> When trying the chat command I get “page does not match the required format for a chat”.
I should make this easier, but that command expects the page to look like this:
```auto
**user** : What's the population of new york?
```
To get it to work on an existing page, basically just add ` **user** : ` to the bottom of the page. It’ll ignore anything above the first ` **user** :` it finds though.
> [@nikolasdi](#):
>
> Other commands show a “Contacting LLM, please wait” but nothing happens.
Can you check if the javascript console has any errors in it? If there’s an error, it’ll log it to the console but doesn’t flash a notification on the page currently
---
### Author:  [@nikolasdi](https://community.silverbullet.md/u/nikolasdi)
#### Post date: [February 21, 2024, 1:08pm UTC](https://community.silverbullet.md/t/new-plug-openai-llm-ai-integration/211/13 "2024-02-21T13:08:43Z")
Thanks a lot! I got mistral-medium working. I still had no success with ollama running on my machine.
---
### Author:  [@justyns](https://community.silverbullet.md/u/justyns)
#### Post date: [February 22, 2024, 8:17am UTC](https://community.silverbullet.md/t/new-plug-openai-llm-ai-integration/211/14 "2024-02-22T08:17:46Z")
Are you using Chrome by any chance? I was testing with Firefox originally and Ollama worked, but on Chrome I ran into CORS errors.
I pushed a few changes ([release notes](https://github.com/justyns/silverbullet-ai/releases/tag/0.0.5)), but try this for ollama (the new requireAuth: false):
```yaml
ai:
defaultTextModel: phi
openAIBaseUrl: http://localhost:11434/v1
requireAuth: false
```
I added this example to the readme too.
Two other big changes:
- You can use the “chat on page” command on any page now, and use cmd+shift+enter or ctrl+shift+enter to trigger it
- SETTINGS and SECRETS now reload automatically when changed, so no messing about with reloading/etc
---
### Author:  [@nikolasdi](https://community.silverbullet.md/u/nikolasdi)
#### Post date: [February 23, 2024, 3:29pm UTC](https://community.silverbullet.md/t/new-plug-openai-llm-ai-integration/211/15 "2024-02-23T15:29:28Z")
Thanks a lot! Ollama still does not respond for me, but it might have to do with my machine. Anyway, mistral works great and it is relatively cheap, so I am covered 😊
---
### Author:  [@zef](https://community.silverbullet.md/u/zef)
#### Post date: [February 25, 2024, 2:05pm UTC](https://community.silverbullet.md/t/new-plug-openai-llm-ai-integration/211/16 "2024-02-25T14:05:06Z")
I just love this thing. The fact that OpenAI returns content in markdown just makes this perfect for SilverBullet.
---
### Author:  [@justyns](https://community.silverbullet.md/u/justyns)
#### Post date: [March 16, 2024, 10:09pm UTC](https://community.silverbullet.md/t/new-plug-openai-llm-ai-integration/211/17 "2024-03-16T22:09:03Z")
I haven’t updated this thread in a while, but wanted to share a few recent highlights from recent releases!
The latest release is [0.0.11](https://github.com/justyns/silverbullet-ai/releases/tag/0.0.11). Some of the big changes since I originally posted this thread:
- Support for [multiple model configs](https://ai.silverbullet.md/Configuration/), with a command to switch between different LLMs even in the same chat context
- Expand wiki links automatically and provide the linked note as context
- [Custom enrichment functions](https://ai.silverbullet.md/Configuration/Custom%20Enrichment%20Functions/), e.g. to detect and fetch remote urls automatically
- Started a docs site: [https://ai.silverbullet.md/](https://ai.silverbullet.md/)
- Slash commands/completion for templated custom ai prompts
- Direct [support for Gemini](https://ai.silverbullet.md/Providers/Google%20Gemini/)
- [Templated Prompts](https://ai.silverbullet.md/Templated%20Prompts/)
- Support for [custom chat instructions](https://ai.silverbullet.md/Configuration/Chat%20Instructions/) similar to chatgpt
Thanks to everyone who’s provided feedback so far!
---
### Author:  [@zef](https://community.silverbullet.md/u/zef)
#### Post date: [July 14, 2024, 3:08pm UTC](https://community.silverbullet.md/t/new-plug-openai-llm-ai-integration/211/18 "2024-07-14T15:08:19Z")
Let me just bump this one for visibility. @justyns is just doing an amazing job progressing on this project. There’s just so much opportunity in this one that I wish I had more time to play with it.
Note there’s a whole website (published using silverbullet-pub) as well:
[https://ai.silverbullet.md](https://ai.silverbullet.md)
---
### Author:  [@justyns](https://community.silverbullet.md/u/justyns)
#### Post date: [July 15, 2024, 8:59am UTC](https://community.silverbullet.md/t/new-plug-openai-llm-ai-integration/211/19 "2024-07-15T08:59:38Z")
If anyone is interested in recent additions, I’ve also been adding quite a bit related to vector embeddings and search:
- [Configuration/Embedding Models — SilverBullet AI Plug](https://ai.silverbullet.md/Configuration/Embedding%20Models/)
- [Commands/AI: Search — SilverBullet AI Plug](https://ai.silverbullet.md/Commands/AI:%20Search/)
It’s in the main branch, but I haven’t made a release yet since I’ll be out of town for a few days and want to make sure nothing’s going to blow up. If anyone is interested in testing out these changes, please feel free!
One of the features I’m pretty excited about is enabling RAG in the chat mode, so I can ask about things without having to actually link directly to a note now.
---
### Author:  [@justyns](https://community.silverbullet.md/u/justyns)
#### Post date: [September 16, 2024, 7:12am UTC](https://community.silverbullet.md/t/new-plug-openai-llm-ai-integration/211/20 "2024-09-16T07:12:36Z")
Version 0.4.0 of the AI plug is now released. There are quite a few changes, you can see the changelog on the fancy new docs website here:
[https://ai.silverbullet.md/Changelog/#040-2024-09-16](https://ai.silverbullet.md/Changelog/#040-2024-09-16)
A few important changes:
- Support for space-config
- Add support for [post-processors](https://ai.silverbullet.md/Templated%20Prompts/#post-processors) to templated prompts
- Several new options for the `insertAt` option, like replacing a line/selection/list.
- New prompts in the AICore library to demo some of the new features
- [https://ai.silverbullet.md](https://ai.silverbullet.md) is now being created using a combination of silverbullet-pub+mkdocs+mkdocs-material and looks much better than before imo.
As always, please let me know if you run into any issues or have any ideas!
[Next page](https://community.silverbullet.md/t/new-plug-openai-llm-ai-integration/211.md?page=2)