Brainstorm: Library sharing

[zef (Zef Hemel)]

One thing I still haven’t really figured out how to best do is sharing of templates, useful pages, space script etc. I’ve tried a few things:

Initially a lot would simply be built into SB: slash commands such as /task. This worked, but meant to add anything you’d have to dive into the (typescript) code, add a command, and maybe put it in a plug for distribution. Advantage: distribution is simple, as once a new template is added to SB, everybody automatically gets it on the next upgrade, or users use the plug and upgrade it from time to time. Still, the barrier to entry is fairly high and I become a bottleneck of reviewing these things and deciding what’s good enough to include and what is not.

Then, I upgraded the template system to no longer require having to modify SB itself (or create a plug) to do a lot of things and you could just do them in your space: templates, useful pages (such as Library/Core/Page/Maintenance) and more recently space script.

That’s all cool, but then the challenge became: ok, should I just rip everything out of the core SB distribute and figure out distribution of these templates later? I decided yes, this is the way. So now, SB itself ships with as little built in as possible (although I could probably remove even more) and moving as much as possible to “user space”.

My short term solution to distribution of these templates is now Libraries. I think this is a nice and simple model, but the open question remains: how do you update and maintain these things?

In the current iteration, I suggest people just “import” a library, which means to simply keep a copy in their space. They just have to press a button, so it’s easy enough. This is nice and simple. However, work will be done on this core library. You may want to pull a newer version. The best way to do this is to simply import again, but that will then overwrite any local changes, which I also encourage people make.

Another option, which already works but I just am not advertising it is using Federation. Using federation, you say: “this other place on the Interwebs is part of my space, please consider it as such”. So you can add silverbullet.md/Library/Core to your federate list and automatically get everything there (virtually) synced into your space and kept up to date. Great! But… these federated bits are read only. You cannot make changes. This would be fine, but there may be templates you want to disable or tweak slightly, and… you can’t. Potentially, some users may also not like that their SB instance is pinging a central server (silverbullet.md in this case) to pull these templates. Now it’d be possible to conceive of a system where you can influence what templates do without modifying them e.g. through SETTINGS, but that complicates them quite a bit.

So… what would be a better system? How do we balance the flexibility of “owning your templates and tweak them however you want” with “but also give me the latest and greatest updates”?

Ideas?

[devzero]

Could there be a third option? Allow periodic or user controlled updates of their libraries by adding an annotation of a semantic version somewhere in them? Then you could keep track of origin and installed version and make it a user choice to update all the time or just know what version they are running? I’m weary of federated libraries a little bit given the abitrary command execution powers of syscalls on your server that could be trivially used by library code, especially with space scripts. It would also appear to damage the offline and self hosted value of the system. One option to address that might be to grant user selectable security policies to external library code? But then people might be forced to decide between wanting a functioning git plugin and having a current plugin, which isn’t great.

[Maarrk (Marek Ł.)]

My idea, fortunately or not, touches on some other parts of the system:

First, introduce a way to run a command periodically, a bit like cron does, but configured in yaml, on some page like JOBS. That is definitely doable with a Plug, since Git Plug essentially does that. So you could make the automatic updates yourself if you put the Import Library command there, same as in a button. That would make pinging some remote server opt-in with a configurable frequency (daily, weekly etc.), and you can do exactly the same thing manually if you want, since it’s implemented as a command.

Side effect: Git Plug and other backup tasks only need to provide a command, there is a common reusable scheduling system. Also may be useful for {[Share: Publish]}

Maybe there is a version in the frontmatter, and the update is done when it changes? When the update happens unattended, it should wait with showing the overwrite prompt until the user opens the space next time.

But I think a cooler way would be to improve the conflict resolution system for Sync mode and also use it for this. I think the dream would be something more Git-like: diff of specific lines, and buttons to accept either version of the block, both, or manually edit as you wish.

How VS Code does it

from: Source Control with Git in Visual Studio Code - Merge Conflicts

Using a tweaked library and getting updates for it looks like a problem of merging your local changes with remote updates. And I think software world collectively agrees that Git is the way. So even if my specific suggestion isn’t good or feasible, I’d look for more inspiration there.

[paletochen]

Maybe we could use technics used in the Linux world, like with systemd.

systemd has most configurations in /etc/ which are read-only for the normal user.
However, the user is allowed to have a user space systemd config, either a full one, overwriting the default/system one, or one which adds/enhances the default one with extra configurations.

We could have SB offering everything like now, through Library and, if a user wants to update a specific part of that file, they can do it in their personal space, and only for that particular setting, maybe in Library/Personal/Core for the Core one.

For example:

• let’s say the user has a better Linked Mentions query that he wants to use. He could maybe add in Library/Personal/Core/Widgets/Linked Mentions the following file
  (probably the exact file may be different, this is just an example)

where: "false"

Which would be the part needed to disable the core widget.
And then use their own.

When the Core library updates, the user can pull the library as normal and assess if this new version suits their needs or not.
Hopefully changes are backwards compatible and, if no action, the Linked Mentions widget is still disable after the upgrade.

But this way the user can accept all new changes and always decide if they want to overwrite them or not.

Example on how I overwrite the user in transmission-daemon, a daemon running in my machine

>sudo systemctl cat transmission-daemon

# /usr/lib/systemd/system/transmission-daemon.service
[Unit]
Description=Transmission BitTorrent Daemon
Wants=network-online.target
After=network-online.target

[Service]
User=debian-transmission
Type=notify
ExecStart=/usr/bin/transmission-daemon -f --log-level=error
ExecReload=/bin/kill -s HUP $MAINPID
NoNewPrivileges=true
MemoryDenyWriteExecute=true
ProtectSystem=true
PrivateTmp=true

[Install]
WantedBy=multi-user.target

# /etc/systemd/system/transmission-daemon.service.d/run-as-user.conf
[Service]
User=myuser

Default User is overwritten from debian-transmission to myuser

In this case I have the run-as-user.conf also inside /etc but, i could have it somewhere else in my space

[zef (Zef Hemel)]

@Maarrk

First, introduce a way to run a command periodically, a bit like cron does, but configured in yaml, on some page like JOBS. That is definitely doable with a Plug, since Git Plug essentially does that. So you could make the automatic updates yourself if you put the Import Library command there, same as in a button. That would make pinging some remote server opt-in with a configurable frequency (daily, weekly etc.), and you can do exactly the same thing manually if you want, since it’s implemented as a command

This seems orthogonal to the discussion on how to share libraries, but I agree this is a gap in space script. I see two obviously missing hooks we need to add, both are available to plugs but not space script yet:

• Cron (so periodic execution of logic), I can imagine introducing a silverbullet.scheduleCron(“* * * * *”, () => { console.log(“Look at me, I run every minute”); }) type of API
• Event listening, e.g. silverbullet.addEventListener(“editor:click”, (e) => { console.log(“Clicked in the editor!”, e); })

Ah wait, now I see your reference to auto updating libraries using this. Yeah, I suppose that’s one way to do it.

[zef (Zef Hemel)]

Right, I’ve been thinking about this type of “overlay” model as well. In its most basic form it could just be a mechanism to override templates defined elsewhere as a whole. A more advanced version would indeed allow to override specific attributes e.g. in frontmatter (and I suppose the body as well).

Maybe something along the lines of

—-
tags: template
overrides: Library/Core/Widgets/Linked Mentions
hooks.top.where: “false”
—-

[paletochen]

Exactly,

I think that is a way that could work and would not be difficult to understand and learn.
We just need an easy way to relate the target that we want to override and the page where we are doing it.

Your frontmatter example would work and it would allow the user freedom to
have those files wherever they want them.
My proposal was more strict, enforcing a structure and naming convention. Maybe easier and faster to process, but prone to errors if users don’t follow the right pattern

[davidbeijinho]

If i can add my 2 cents, I also think that the idea of overwrite works very well, I have used a lot in https://tiddlywiki.com/ and it makes it very easy to share notes/tiddlers/scripts with anyone and to make changes to that scripts to fit your needs