Thoughts on applications of AI or "smarts" in SilverBullet

Let me start by saying that silverbullet-ai written by @justyns exists, and you should all try it, it’s cool!

That said, inspired by Apple’s recent announcements of their “Apple Intelligence” and “personal intelligence” (which us Europeans may not get any time soon) I’ve been thinking a bit about a “SilverBullet approach” to it, from first principles.

What would be a suitable implementation technology wise look like, and what would its main use cases be? Let’s brainstorm a little bit.

Since SilverBullet is largely about self hosting and owning your own data, it seems obvious that “SB smarts” should be built from the ground up to work with a local LLM of sorts. If we can offer a default choice “just use this docker-compose file” type of deal that’d be great. It’s private data we’re dealing with and handing that off to OpenAI or Google should at least not be the default. So support for a locally hosted LLM and perhaps have an opinion on which one that should be would be good.

In the context of note taking use cases I see a few high impact ones:

First: better search. Let’s face it, SB’s full text search is pretty bad. I would suspect that if we use vector based search, based on LLM generated embeddings, we would get WAY better search. On every page change, the page’s text would get vectorized and stored in our index. That index can then be used for various use cases: search would be one.

Second, the vectors can be used to surface related content (based on vector distance again). So imagine below your “Linked Mentions” a “Related Pages” (or whatever) section, with pages thematically close (not based on explicit links or tags, but based on content-relatedness determined by vectors).

Third: smartly naming notes. I’d like my workflow to become: create an empty note with SB without giving it a name. Do some typing, then hit a key to auto name the note for me. Done. Next.

Then fourth would be what I’m using @justyns’s AI plug for now primarily, which is to have a chat with e.g. ChatGPT and capture its output in a page directly. Often I just like to have a note with a definition of some sort, and the way I get that now is to create an empty page with the name of the term, type in “define ‘term’”, and let AI: Chat on current page give me the definition, strip away the redundant stuff. Done.

Fifth “rewrite based on prompt”. I’m not sure I would use this much myself, but I can see that people want to do the “let me just write this aggressively as draft, and then let AI rewrite it to sound nice and professional” type of use case. This is also one of the features of Apple Intelligence: “rewriting this to sound professional, nice, funny” etc.

What do you all think? Where do you stand in the whole AI/LLM discussion? Do you want AI in your SB (it would always be optional, obviously) or want to keep it far away from it? What do you see as the use cases?

9 Likes

Thank you for the thoughtful post. I’ve just started using Silverbullet and have used local LLMs in my previous note taking app. I entirely agree with your approach of local first and strong integration with vector search etc. I’ve had good success with the smaller phi models by Microsoft using ollama and serving them via their openAI compatible local server. The usefulness of LLMs in note taking apps depends (in my opinion) entirely on the UI. If I have to write out a long prompt every time I want a summary of a note it often takes longer than reading it. I’d advocate for a community prompt library that can be triggered easily via slash commands.

3 Likes

Excellent topic, though I may be biased :wink:

I’m interested in everyone else’s thoughts, but did want to share a couple things that come to mind.

#1 and #2 both require some sort of vector database. An easy solution would be to pick one to recommend and have some docker-compose/etc examples - but what about people who don’t want to use any sort of ai/llm? We’d be making the vector db a hard dependency. While I’d be one person who wouldn’t mind, I also value how “simple” SB is to get started with.

Something that has been on my radar for a while is sqlite-vec, which could potentially be embedded into SB without an external dependency (from the user’s point of view). There are other similar projects that could be alternatives too.

There is the same problem for the LLM itself; I doubt(?) we’d want to embed it into SB itself. Offering recommendations/documentation around a 3rd party tool like Ollama would probably be the way to go, but at least it’d be an optional dependency.


Back to the actual question, some things I’d only be comfortable doing with local-only llms, but would be cool imo:

  1. Auto completion in SB using LLM. Like gmail’s smart prediction, or even an IDE’s auto complete - start typing and then it can suggest the rest of the sentence/paragraph. Or auto fill in your frontmatter based on previous notes.
  2. Context-aware search. I want to be able to ask/tell SB “Search my notes for books I’ve read and list when I started and finished them in a markdown table.” or “What did I do last month?” and have it summarize all of the relative notes from last month.
1 Like

Are Templated Prompts closer to what you’re wanting? I use one similar to the example on that page where I can trigger it with a slash command and have the summary inserted at the cursor. You could have multiple templates for different types of notes/styles/etc.

I’d advocate for a community prompt library that can be triggered easily via slash commands.

+1 I’d love that. I put together a few prompts, but mostly as examples. Having a library of real-world use-cases from others would be way better.

1 Like

First of all, I think this should be fully optional so nothing around this should be built into the SB core, including indexing based on vectors. SilverBullet is setup to support implementing this in a plug, e.g. by listening to the page:index event which would allow for custom indexing (e.g. into a vector database) from a plug.

I also don’t believe the tech or capabilities of LLMs are ready to run this completely in a JS engine. Although, apparently you can embed an LLM into a font… so this would by all practical means mean we’d need to add a dependency or two.

I care a lot about ergonomics, so this shouldn’t be too hard to setup. I’d say that optimizing for a docker-based setup would be acceptable, so anything we can ask people to put in their docker-compose file would be fine. In my mind that could be an additional LLM container and a vector database container if that makes sense.

What I had in mind originally was to implement the vector search using SilverBullet’s built in data store in the dumbest way possible:

  • Use an LLM to generate a vector for a page, resulting in 120 floating point numbers (I think?)
  • Store that vector in the data store (as an object like any other)
  • Upon search, translate the query into a vector using an LLM (again, resulting in an array of floats)
  • Simply scan through all the vectors in the data store one by one and compare their distance using e.g. cosine similarity (there are JS implementations) one by one

Performance may be an issue, but I’d imagine that with a space of say a thousand pages, performing a thousand calculations on a 120 float vector could be fast enough not to require a specialized database. This is just an assumption, if not perhaps dedicated vector DB is better.

The advantage of keeping the vector search in-house (in process) is that people could opt into using e.g. OpenAI for generating the vector embeddings, storing them locally, and using the local vector search without having to add any dependency on a vector database.

3 Likes

Absolutely. That looks amazing. We could have a separate git repo for community prompt templates

2 Likes

Btw, @justyns is fast, so already added the feature to intelligently suggest a new page name to the AI plugin: Add page rename command by justyns · Pull Request #29 · justyns/silverbullet-ai · GitHub

CleanShot 2024-07-04 at 18.50.47

6 Likes

Some new random ideas here, again to support the “let me take a quick note and clean it later” workflow.

An AI: Enhance command that could suggest:

  • An appropriate name
  • Suggest tags (based on existing ones, or suggest new ones following a similar pattern)
  • Extract useful front matter and put it in as actual front matter. For instance if it detects that the page represents a person (and has tagged it as such), could attempt extracting a birth day, first name, last name. Since the tag was detected, the prompt can be extended with attributes that we’ve seen from other pages tagged with it. This may not work at all, but it just might…

I know @justyns plug already supports some of these, but combining them may be powerful. Not sure what the UX could look like yet.

Thanks for the ideas! This is added in the main branch now:

The frontmatter command is still pretty experimental at this point. It works, but I’m not sure about the results. The LLM is a little too eager with giving out a bunch of new frontmatter attributes. My current prompt is here, if anyone has ideas for improving it.

edit: And I didn’t even mention, generating embeddings and a new AI: Search command to search those embeddings are part of 0.2.0.

2 Likes

Haha, just tried this on a quick note with the text “This is a quick and fun note”

This is what it generates using the phi3 model that I run locally

---
tags:
  - inbox
  - metaphor
  - humor
  - writing
  - style
  - simple
  - casual
  - tone
  - internet
  - culture
  - jargon
  - summer
  - update
  - current
  - affairs
  - recent
  - events
  - lifestyle
  - trends
  - personal
  - growth
  - memes
  - travel
  - experiences
  - leisure
  - activities
  - pop
  - references
  - technology
  - updates
  - gadgets
  - fashion
  - blogging
  - online
  - entertainment
  - news
  - viral
  - content
  - engaging
  - storytelling
  - digital
  - age
  - communication
  - Here
  - is
  - a
  - concise
  - list
  - of
  - thematic
  - "tags:"
  - developments
  - stories
---
This is a quick and fun note

Quite elaborate tagging :smiley:

3 Likes

lol that might be a bit of overkill

1 Like

well you certainly should be able to find that note if you don’t remember the exact tag :sweat_smile:

Try setting this?

ai:
  promptInstructions:
    tagRules: |
      ONLY use existing tags. Don't create new tags.
      ONLY add relevant tags. Favor a small number of tags instead of many. 

I haven’t added it to the default prompt, but it’s what I use and helps some. I’m considering changing it to “Only use these tags: --list of tags I want it to use–” instead of any existing tag.

We had an AI Hackweek at the company last week, which resulted in some thinking and reflection (in a SilverBullet note, obviously) which I will copy and paste here now:

The “SilverBullet Way”

This is always evolving, but this is how I tend to approach problems in the context of SilverBullet and its positioning as a note taking app (platform) for people with a hacker’s mindset:

  1. Build a solid foundation that enables end-users to expand and be creative with experimentally (examples: templates, queries, but AI could add a few more)
  2. Let users go crazy
  3. Codifying what works in Libraries (and if necessary: plugs) and make it reusable that way

This is the way.

So from that perspective, let’s have a look at LLMs and where they could fit in.

Core concepts in using LLMs

  • Models: this is rapidly evolving field, both for in locally hosted and cloud based version, so SB should be ready to expand here quickly. Luckily there are “standards” at the API level (for chat, embeddings) so this is fairly easy to stay up to date with and silverbullet-ai seems to have a nice “provider” model already.
    • For different use cases you want to select different models, sometimes for performance (quality, speed) and sometimes for privacy reasons. For instance, I would not send my entire space to OpenAI for generating embeddings (I’d use a local LLM for this), but when I use chat I want the latest and greatest, so perhaps gpt-4o or Claude. This may be different for other people.
  • Prompts: Figuring out the right prompts is not always trivial and subject to experimentation, on top of it is model specific. What works for one model, won’t for another. This means that ideally prompts should not be hardcoded.
  • Use cases: These are still to be determined and ideally, new use cases can be discovered/experimented/built simply by building on the foundation.

With this in the back of our heads, let’s see if we can classify some of the interactions we may have with an LLM.

Types of interactions

Chat

I see different use cases for chat.

Some that come to mind:

  1. Learning about a topic via a chat conversation, e.g. I used it at some point to learn more about end-to-end encryption and how to implement it. I can talk to a top performing model like gpt-4o on this and learn a lot, and rework my chat to a page as a note for future reference. Which makes SilverBullet a useful environment to do this in. This use case is already well supported.
  2. GPTs: ChatGPT has this concept of GPTs. Effectively what this is a way to start a new chat with a pre-prompted model. You start the chat, but that model has already been given instructions that are not visible to you and may tap some additional resources (which are vector searchable). In the context of SB these may be references to pages in your space. While a simple concept, it’s actually very powerful. OpenAI has a GPT store, we can could imagine building similar things with a Library of such prompts.
  3. Chat with my space. This would somehow cleverly pull pages in from my entire space based on the conversion. Not sure how this would work, but I think there can be something here.

To make this more concrete, let me throw in some concept code to show how we may specify this:

Example, a “Practice Dutch” GPT (we should really come up with different branding for this):

in Library/AI/Practice Dutch:

---
tags: template
hooks.ai.chat:
   model: gpt-4o
   name: Practice Dutch
   kickOff: | # Need a better name for this
      **user:** |^|
---
You are a helpful assistant to help the user practice their Dutch skills. Therefore, only interact with the person in Dutch, never give an answer in any other language.

When the user struggles with a word, be helpful and translate the word to Dutch for them.

Below is a list of Dutch words that can be used to practice:
![[Library/AI/Practice Dutch/Words]]

(the point of the translusion example is the emphasize that all SB features like templates can be leveraged for generating the prompt)

You could then fire this GPT with a AI: Chat command, for instance allowing me to select “Practice Dutch” which would then kick off a chat page along the lines of:

---
ai.chat: "[[Library/AI/Practice Dutch]]"
---

**user**: 

Page-based interactions

Some use cases that apply to a single page use case.

Transform

In place transformation (entire page, or selection):

  • “Rewrite this less aggressively”
  • “Expand these bullet points into prose”
  • “Translate to German”

How these could be specified, conceptual code:

In Library/AI/Rewrite Less Aggressively:

---
tags: template
hooks.ai.transform:
  model: phi3
  name: Rewrite less aggressively
---
Propose an alternative phrasing of the following text, keep it roughly the same length but phrased less aggressively:

|^|

(where |^| would be the placeholder for the page content or selection to run the prompt against)

You’d select some text, run AI: Transform, pick “Rewrite less aggressive” and the text would be replaced in place.

Extract/query/question

This would be for use cases where you don’t need any operation to happen on the text, but you’d like some sort of feedback or question answered about it. “Query” would be a good name for this, but since we already have queries this may be confusing. From a UX perspective, perhaps a side panel (RHS) can be used to render the result.

Some example use cases:

  • Summarize
  • Determine tone
  • Give me writing feedback

In Library/AI/Summarize:

---
tags: template
hooks.ai.query:
  model: phi3
  name: TL;DR
---
Summarize the following into a list of bullet points:

|^|

You’d select some text, run AI: Query, pick “TL;DR” and would get the result in a side panel.

Action

This is one more open ended and perhaps harder to generalize, and perhaps dangerous, but worth exploring.

Use cases I’m thinking of are:

  • Rename the page based on a prompt
  • Add frontmatter based on prompt

These are now hardcoded into the plug, but perhaps there’s a way to generalize them. The big question would be: what actions would we (and should we) expose. All commands?

Honestly, this area I haven’t really figured out yet.

Space-based interactions

This is another space where I don’t have concrete suggestions, I just sense there are use cases here. Perhaps just in the chat use case of “talking to your space”. The building blocks here would be vectorization of your entire space, which is already supported, pulling in relevant content based on a chat or prompt and then… doing stuff with it. Again I think chat is the most obvious UX to do this.

Anyway, some food for thought.

And I expect that @justyns will have implemented all this by the morning :rofl:

1 Like

Proposed branding for the SilverBullet version of GPTs:

SilverBots
Or more colloquially S-Bots (“ass-bots”)
Or just SBs.

SilverBots is pretty good. It might also be good to keep it similar to Space{Script,Theme} and call it SpaceBots. You still get to retain “ass bots” :wink: .

Other suggestions:

  • BulletBots
  • SilverAI

Recently I’ve found myself using Notion AI at work to search through our documentation and it has been awesome.

Combining search proposal here and providing that as context to do llm chat would be awesome. I don’t write down a lot of things because I know I won’t be able to find anything if I have too much content to search through. This would solve that.

Right. I think the Notion AI Q&A use case is what makes sense for SB as well. I think this is doable via a mix of vector search to select the right content, combining it and prompting it properly.

I’ll take some more time to digest these ideas, but I like them. Templated Prompts were my first attempt at making this more user customizable to avoid hardcoding prompts/etc. I just wasn’t sure how to go from “Render this template and send it to the llm” to “use the result for the page title, or tags, or etc”.

Honestly I think a lot of this is just finding the right UX. Having a fleshed out library to import would help with this too.

Tool/function calling would be pretty powerful. I could potentially see silverbullet-ai and silverbullet itself having more utility-type functions like to help modifying frontmatter/etc, and then some sort of translation layer to turn them into tools that can be used by the llm.

E.g.:

---
tags: template
hooks.ai.transform:
  model: gpt-4o
  name: Rename and tag page
  allowed_syscalls:
  - editor.renamePage
  - editor.setTags
---
Review the below content, choose a new name for this page and generate appropriate tags.

|^|

We could expose space scripts the same way.

1 Like

And one other naming suggestion:

Silver Bullet Assistants (SBAs)

1 Like

Yes, I liked that concept and tried to push it a bit further. I think this is how we discover things, by experimenting and seeing what works. I feel this could evolve into a killer app for SB though. There’s a lot of potential here.

Agreed.

Yep agree. I would hope that Commands would be the answer here. I can imagine enriching commands with more documentation (e.g. in a description field), that AI could as well as regular users could benefit from. Explicitly listing those command (again, I’d go with commands, not syscalls) in a template could be a sane way to keep some control over what this the LLM is going to be able to do.