Library management

[zef (Zef Hemel)]

A long standing challenge I never managed to tackle is how to properly distribute libraries (space lua, space styles etc.) and plugs. For plugs there’s a bare bone plug manager + a config setting but… it’s not a great and very nice experience.

I’m currently working on a new approach to this problem…

Here are my work in progress notes, open to feedback. Sorry these are rough notes, I hope they make sense to people “in the know” (e.g. @Mr.Red who’s been looking into this). There’s a demo GIF at the end of the post.

Libraries are SilverBullet’s mechanism to build and distribute extensions to SilverBullet.

Libraries can consist of:

• Pages containing:
  • Space Lua scripts (Lua extensions)
  • Space style scripts (CSS extensions)
• Plug files (JavaScript extensions)
• Any other documents (images, PDFs, whatever)

The root of a library is a page tagged with #meta/library. This makes them eligible for updates.

Libraries become discoverable via repositories that are just pages (that can self update from a remote source) containing lists of library names with URIs to download them from.

Libraries are kept under Library/ Repositories under Repository/.

Distributed with SB itself:

• Repository/Std: a “blessed” list of installable libraries that will be updated with new SB releases.
• Library/Std: this will have all built-in plugs as files as well as all existing std pages.

Frontmatter used for pages tagged with #meta/library:

• version: used for update checking
• files: for any additional files, with paths relative to the library page

Changes required core to enable this

• Load all files in space ending with .plug.js as plug, not just under _plug/ prefix

Commands

• Library: Add Repo (asks URI)
  • Downloads under Repo/<<name>>
• Library: Update Repos (fetches all #meta/repo pages, reimports based on uri)
• Library: Remove Repo (picker, removes page)
• Library: Install
  • List all #meta/library/remote items
  • Pick, install under Library/<name>
• Library: Update All:
  Update process:
  • Find all #meta/library items, extract name (relative to Library/)
  • Look up in #meta/library/remote fetch remote, compare versions, replace
• Library: Remove
  • Asks for #meta/library object to remove
  • Remove process:
    • Remove library file
    • Remove linked files

Repositories

Repositories are pages tagged with #meta/repo with frontmatter:

• uri: for self updates

Objects tagged with #meta/library/remote, can be data blocks, or tagged items:

• Schema:
  • name
  • uri

Library

Best practices

Simple libraries can be single page artifacts. Many can be put in a single Github repo (e.g. zefhemel/silverbullet-libraries), they can also be distributed as gists. Such a repo can contain a REPO.md file which serves as the repositories linking to all the contained libraries.

More elaborate libraries (such as the built in Std), may contain a very elaborate list of additional files with a mix of plugs, documentation pages and space Lua scripts, e.g.

---
tags: meta/library
files:
- core.plug.js
- editor.plug.js
- API/Something.md
---

Plugs are generally distributed in a separate GitHub repo where the README.md page can serve also as the root of the library, typical frontmatter:

---
tags: meta/library
files:
- plug.plug.js
---

Examples:

README.md

Git.md

Demo of current state

​
Note that this also “wraps” plugs as libraries (like SilverSearch, Mermaid), so there’s just one type of thing to install and manage. This can also replace the whole plug manager that’s currently in place.

So my rough plan is: I’ll distribute one “repository” page with SilverBullet standard distribution, with some of the more popular libraries/plugs, so those are installable with a push of a button. Then people can create their own repository pages (still need to build an installer/updater for those) with additional libraries of their own.

[Mr.Red]

Looks great didn’t understood all of it. But I’m sure if I will have my hands on, it will makes much more sense. Looks really thought trough​:+1:

[malys]

Sound great @zef
But have you imagined something to store the list of repositories provided by community?
You are managing installation and probably updates of plugs from known repositories but it could be interesting to have a list of known libraries and plugs repositories.

[MrMugame]

So if I’m understanding this correctly, it’s very similar to most Linux package managers, like apt? Where Repositories map to sources/repositories and libraries to packages?
This kind of leads me to the question if a dependency system is necessary, but I guess I couldn’t really come up with a good reason, would probably just breach the scope. Second thing that comes to mind from this comparison is stuff like checksums/signing which probably goes hand in hand with versioning. I don’t really know how that exactly works with apt for example, but I guess at least having a way to verify what version someone is running would be great for debugging in the future.

[zef (Zef Hemel)]

@MrMugame

So if I’m understanding this correctly, it’s very similar to most Linux package managers, like apt? Where Repositories map to sources/repositories and libraries to packages?

Yes. I was thinking of adopting that terminology, but not sure it would help. We already use the library term.

As to dependency management and signing: Maybe one day. I’d really like to postpone making this more complicated than it needs to be until it needs to be.

[thepaperpilot (Anthony Lawn)]

I like this idea! Does this mean libraries would all be read-only, similar to the current builtin library? If so, could/should there be a way to “detach” a file in order to modify it, including the js files?

[zef (Zef Hemel)]

No they wouldn’t be read only in my current thinking, but this is risky in the sense that when you make local edits and then update them all your changes would be reverted…

[zef (Zef Hemel)]

Current state, getting close I think:

[Mr.Red]

WEN public beta?

[zef (Zef Hemel)]

I’ll see if I can merge it onto main (= edge) tomorrow.

[zef (Zef Hemel)]

It just landed on main, see CHANGELOG

[zef (Zef Hemel)]

What needs to be done now is that plug authors, especially those listed here: Plugs need to “wrap” their plugs in a library. Once that’s done, I can put them in e.g. the Repositories/Std to make them super easy to install.

To do this, you only need to update your README.md file and add your something.plug.js to the files key and set tags: meta/library like here: silverbullet-mermaid/README.md at main · silverbulletmd/silverbullet-mermaid · GitHub

[Mr.Red]

First of all Congratulations, it’s a well thought out system and it works well

And now to my first feedback and suggestions:

1. you forgot to mention in the documentation that the repository needs to have the meta/repository tag, but I will do a complete suggestion for the documentation, because currently not everything is clear

2. For the Repo Name, it would be nice to be pre filled with a suggestion eg.: username Repo

3. Is there a way to filter out my own local Libraries/Repos (which I use as source to publish to Github) to not show up in the Library Manager ?

4. Is it possible to add “pageDecoration.prefix” to be visible in the custom DOM Tables in library manager? So we can add some emojis to our Library names, to help the user get a first glance what’s the library about:

   For example:
   for Tools & Widgets
   or for Styles / Themes
   for Repositories
   for deprecated libraries

5. In the “Available Libraries” Table, it would be nice to click on the library name and that would open the Library on GitHub so the user can get a first glance at the code without first installing it.

6. in the Library: Add Repository command there is a typo:
   editor.flashNotification "Library installed"
   instead of:
   editor.flashNotification "Repository installed"

7. it would be nice to right-align all the buttons to be aligned and not over the place (see below)

8. Add classes to the buttons: Update, Remove Install, so it can be styled/customized easier: see below

VS:

Modified DOM Table Building functions with added classes

-- priority: -1
function widgets.button(text, callback, attrs) --added attrs argument
  local buttonAttrs = {
    onclick = callback,
    text
  }

--addtrs to add custom classes to buttons for easier styling
  if attrs then
    for k, v in pairs(attrs) do
      buttonAttrs[k] = v
    end
  end

  return widget.html(dom.button(buttonAttrs))
end

function library.installedLibrariesWidget()
  local rows = {}
  for lib in query[[from index.tag "meta/library"]] do
    table.insert(rows, dom.tr {
      dom.td { "[[" .. lib.name .. "|" .. lib.name .. "]]" },
      dom.td {
        not lib.share and "" or dom.span{
          widgets.button("Update", function()
            local updated = library.update(lib.name, true)
            if updated then
              editor.flashNotification "Updated."
              reloadEverything()
            else
              editor.flashNotification "No update required."
            end
          end,{ class = "btn-update" }), --added custom class
          widgets.button("Remove", function()
            if editor.confirm("Are you sure?") then
              library.remove(lib.name)
              editor.flashNotification "Done!"
              reloadEverything()
            end
          end, { class = "btn-remove"}) --added custom class 
        }
      }
    })
  end
  return widget.htmlBlock(dom.table {class = "manage-lib-table", --added custom class
    dom.thead {
      dom.tr {
        dom.td {"Library"},
        dom.td {"Action"}
      }
    },
    dom.tbody(rows)
  })
end

function library.installableLibrariesWidget()
  local rows = {}
  for lib in query[[from index.tag "meta/library/remote"]] do
    local installed = library.getInstalled(lib.uri)
    if not installed then
      table.insert(rows, dom.tr {
        dom.td { lib.name },
        dom.td { "[[" .. lib.page .. "|" .. lib.page .. "]]" },
        dom.td {
          widgets.button("Install", function()
            library.install(library.libPath(lib.name), lib.uri)
            editor.flashNotification "Done!"
            reloadEverything()
          end,{ class = "btn-install" }) --added custom class
        }
      })
    end
  end
  return widget.htmlBlock(dom.table {class = "manage-lib-table", --added custom class
    dom.thead {
      dom.tr {
        dom.td {"Library"},
        dom.td {"Repository"},
        dom.td {"Action"}
      }
    },
    dom.tbody(rows)
  })
end

function library.installedRepositoriesWidget()
  local rows = {}
  for repo in query[[from index.tag(library.repositoryTag)]] do
    table.insert(rows, dom.tr {
      dom.td { "[[" .. repo.name .. "|" .. repo.name .. "]]" },
      dom.td {
        repo.share and dom.span {
          widgets.button("Update", function()
            if share.sharePage(repo.name) then
              editor.flashNotification "Repository updated"
              reloadEverything()
            else
              editor.flashNotification "No changes"
            end
          end, { class = "btn-update"}),
          widgets.button("Remove", function()
            if editor.confirm("Are you sure?") then
              space.deletePage(repo.name)
              editor.flashNotification "Done!"
              reloadEverything()
            end
          end, { class = "btn-remove"}) --added custom class
        } or ""
      }
    })
  end
  return widget.htmlBlock(dom.table {class = "manage-lib-table", --added custom class
    dom.thead {
      dom.tr {
        dom.td {"Repository"},
        dom.td {"Action"}
      }
    },
    dom.tbody(rows)
  })
end

Button Styles

:root {
  /* UPDATE (green) */
  --btn-update-bg: oklch(50% 0.25 160);
  --btn-update-text: oklch(100% 0.05 160);
  --btn-update-border: oklch(70% 0.20 160);
  --btn-update-hover: oklch(70% 0.25 160);

  /* REMOVE (red) */
  --btn-remove-bg: oklch(50% 0.25 30);
  --btn-remove-text: oklch(100% 0.05 30);
  --btn-remove-border: oklch(70% 0.20 30);
  --btn-remove-hover: oklch(70% 0.25 30);

  /* INSTALL (blue) */
  --btn-install-bg: oklch(55% 0.20 260);
  --btn-install-text: oklch(100% 0.05 260);
  --btn-install-border: oklch(70% 0.15 260);
  --btn-install-hover: oklch(70% 0.25 260);
}

/* Base button style */
table.manage-lib-table td button {
  padding: 6px 12px !important;
  margin-inline: 4px;
  font-size: 0.95em;
  border-radius: 8px;
  cursor: pointer;
  border: 1px solid transparent;
  transition:
    background 0.25s ease,
    transform 0.15s ease,
    box-shadow 0.25s ease,
    border-color 0.25s ease;
}

/* UPDATE button */
table.manage-lib-table td button.btn-update {
  background: var(--btn-update-bg);
  color: var(--btn-update-text);
  border-color: var(--btn-update-border);
}
table.manage-lib-table td button.btn-update:hover {
  background: var(--btn-update-hover);
  border-color: color-mix(in oklch, var(--btn-update-hover), white 5%);
  box-shadow: 0 0 5px color-mix(in oklch, var(--btn-update-hover), white 15%);
  transform: scale(0.95);
}

/* REMOVE button */
table.manage-lib-table td button.btn-remove {
  background: var(--btn-remove-bg);
  color: var(--btn-remove-text);
  border-color: var(--btn-remove-border);
}
table.manage-lib-table td button.btn-remove:hover {
  background: var(--btn-remove-hover);
  border-color: color-mix(in oklch, var(--btn-remove-hover), white 5%);
  box-shadow: 0 0 5px color-mix(in oklch, var(--btn-remove-hover), white 15%);
  transform: scale(0.95);
}

/* INSTALL button */
table.manage-lib-table td button.btn-install {
  background: var(--btn-install-bg);
  color: var(--btn-install-text);
  border-color: var(--btn-install-border);
}
table.manage-lib-table td button.btn-install:hover {
  background: var(--btn-install-hover);
  border-color: color-mix(in oklch, var(--btn-install-hover), white 5%);
  box-shadow: 0 0 5px color-mix(in oklch, var(--btn-install-hover), white 15%);
  transform: scale(0.95);
}

/* Press feedback */
table.manage-lib-table td button:active {
  transform: scale(0.90);
  box-shadow: 0 0 4px color-mix(in oklch, black, var(--btn-remove-bg) 20%);
}


/* Right-align the Action column header (thead uses <td> in your markup) */
table.manage-lib-table thead td:last-child {
  text-align: right;
}

/* Right-align every last column cell (where your buttons live) */
table.manage-lib-table tbody td:last-child {
  text-align: right;
}

[malys]

Fantastic!

To my mind, It’s missing a little description for each scripts.

[zef (Zef Hemel)]

Ok people, I just pushed some breaking changes to libraries. They’re all documented on the website pages (which I’ll publish soon), but let me list them here:

I removed the restriction that libraries have to be put under Library/ and repositories under Repositories/. That’s still a good convention, but no longer enforced.

Libraries

I now require a frontmatter name key in all libraries. This name has to match the full page path (without .md). If it doesn’t, you will see a validation error. The reason for this is to give libraries a predictable installation target, which enables you all to write code that assumes a library (and its additional files) are going to be installed in a predictable place. I think @MrMugame and other had use cases for this.

I have updated a few of my own libraries with these new rules, e.g. silverbullet-libraries/Ghost.md at main · zefhemel/silverbullet-libraries · GitHub and silverbullet-libraries/Git.md at main · zefhemel/silverbullet-libraries · GitHub and the mermaid plug: silverbullet-mermaid/PLUG.md at main · silverbulletmd/silverbullet-mermaid · GitHub

Repositories

Repositories now have the following fields for each meta/library/remote object:

• name (mandatory, but no longer determine where the library will be installed, since this is decided by the library itself)
• uri (mandatory): as before
• description (optional): a brief markdown description of what the library does
• website (optional): a website link for the library

Those who already adopted some of this stuff, you’ll have to make some changes, but I think they’re for the better.

@Mr.Red I contemplated including your styling changes because they look good, but decided not to right now. I’d rather see them as a PR to the main repo where they are applied to all buttons, so they all look and behave consistently.

Here’s a screenshot of how things look now:

Feedback welcome!

[thepaperpilot (Anthony Lawn)]

I recently uploaded my first library, and I’m realizing that I didn’t name/place it in a convenient location. The main file is Library/CBT-Journal/README, which makes the library name “README” instead of something more useful. I have all the other files inside the CBT-Journal folder for nice organization, but it sounds like perhaps I should have made the main file under Library/thepaperpilot/CBT-Journal.md, and then the files it requires under Library/thepaperpilot/CBT-Journal?

If that all sounds correct, is there an easy way to handle moving files with the new share feature, or should I just clear out the github repository and re-share each file from its new location?

[zef (Zef Hemel)]

Yes that sounds like better naming. As for support to effectively remotely rename pages — makes a lot of sense, but there is no support for that yet. Need to think how that could work, so until then indeed I’d just reshare under a new name (you can just update the URIs manually in the frontmatter and share again) and then delete the old ones some other way.

[Mr.Red]

@zef

@Mr.Red I contemplated including your styling changes because they look good, but decided not to right now. I’d rather see them as a PR to the main repo where they are applied to all buttons, so they all look and behave consistently.

function MrRed_reply(context)
  local buttons = {"save", "cancel", "delete", "panic"}
  for _, btn in ipairs(buttons) do
    btn = btn .. "_style"
    print("Applying questionable style to: " .. btn)
  end

  if context == "agreed" then
    print("Mr.Red: Fair enough. Consistency must prevail... for now.")
    print("Mr.Red: I’ll open a PR so every button can suffer equally.")
  else
    print("Mr.Red: Oh? You actually liked the styling? Shocking.")
  end
end

-- usage
MrRed_reply("agreed")

[ChenZhu-Xie (Xiè Chén-Zhú)]

2 small feedbacks:

missing remove lib botton on 2.2.1-ec9fcd1ced6dac6b6a5513497930747df28d6533 from rows under # Available libraries header in Library Manager page for external libs

beneficial to add batch remove buttons for the external libraries (associated with the author) under the # Repositories, # Installed libraries, and # Available libraries headers? and to include batch install buttons (for certain author’s libraries or standard ones) under # Available libraries header?

---

emm, remove Repository, remove libs… OK, no problem -_-||