Real talk: SilverBullet 2.0

General
1

You may have seen hints here and there that I’ve recently spent a lot of time on the Lua side of things and I’m extremely happy with what this will unlock and elegance it can bring to the system. As I probably have mentioned elsewhere, if I would have seen this 3 years ago (I started SB about 3 years ago) I would have designed things very differently.

If you haven’t already, watch this video for a glimpse. A lot of this is already somewhat out of date, but you’ll get the gist:

Up to now I developed the Lua stuff in “parallel universe” mode, there are now “Lua ways” to do many things that previously required more specific functionality.

For instance:

  • Templates can be implemented using the ${lua expression} syntax (that you can use anywhere) and a few Lua APIs. No need for a special template fenced code block syntax + language
  • Queries can be implemented with a slight overloading of Lua syntax and become much more elegant and general purpose this way: Space Lua/Lua Integrated Query so no more need for a special query fenced code block syntax + language.
  • Most (though not all) safe things you could do with Space Script you can also in Lua, but often slightly nicer (for instance because in Space Lua we abstract away from all async/await JS stuff).
  • Live Template Widgets can now also be implemented in Lua by listening and responding to specific events (not publicly used yet, but believe me: it’s more elegant), so they don’t need a special hooks template mechanism anymore either.
  • Live Embeds are unnecessary, because they can simply become Lua expressions, like ${embed.youtube "url"}. No need for specific syntax here either.
  • Transclusions also become unnecessary because, again, you can just do this with a Lua call (not implemented yet, but would be trivial), e.g. ${embed.page[[other page]]} without the need for custom syntax (although we may want to keep this one because of Obsidian compatibility).
  • Space Config? Not really necessary, we can have a Lua API for that.

The whole Objects story basically doesn’t change. All of this still makes perfect sense in a Lua world. You just query them using the Lua Integrated Query.

So a lot that needed to be custom built before, can now (often trivially) be replaced with Space Lua versions.

This is very exciting (to me).

However, there is a but…

The but

Parallel universes are expensive, especially if you’re a team of… well, me, with limited time and brain capacity. SilverBullet is a hobby. I have a “real” job and family as well. While I initially though I’d be fine with a slow and phased migration story from the old to the new, I’m now more concerned about the cost of doing this. I have already encountered a lot of parts of the code base where it’s very hard to keep things working in parallel. There’s a lot legacy, ugly code in there that I’d be eager to get rid of, but cannot because it would break existing functionality.

I also see bug reports and people running into issues where I think: well, that won’t be a problem in the Lua world, should I even bother to fix that bug? Actually, do I even have energy to fix bugs that I know are in parts of the system I don’t really want to keep supporting long term?

I really spend a lot of my (limited) mental energy right now thinking “how do I do this, without breaking that other feature?”

What I would love to do right now. And would have done absolutely, if I were SB’s only user, would be to rip the band aid and remove all the “legacy” features I just mentioned (as some other questionable features that I’m not sure anybody uses) in on fell swoop.

Gone.

Oh, that would be so nice.

However, the (happy) reality is that I do have active SilverBullet users and that’s great. I’m very happy you’re all here, but you do make my life a bit harder :wink:

Still I think something needs to happen, because ultimately there’s a slight risk of me burning out on the complexity of maintaining this system of ever increasing size and complexity, and that may not be good either.

So, as I’m writing this, I want to propose something and see what you all say.

SilverBullet 2.0

Versioning is weird, because there never was a SB 1.0. But let’s consider all the 0.* versions SilverBullet 1.0 for simplicity.

What I am considering is doing a last 0.x release, and then go in hardcore “rip everything we no longer need” out mode. A lot, if not all, features I mentioned at the top would disappear, plus perhaps some more.

I don’t want to break everybody’s happy space (hah) right now, so what everybody can (and perhaps should) do is pin their version to that last 0.x release. It will be there, you can use that until the end of time. If people get upset about me removing their favorite feature in this version, or just performing this rug pull, I’d be happy to support whoever is interested to maintain a branch for people who don’t want to migrate to 2.0. There can be a “SilverBullet classic”. There always could be of course, this is an open source project, people can always fork.

Then, the brave ones (I hope all of you) can do the migration of their space to what ultimately will become a 2.0 release, whenever they’re ready.

How to migrate

What would be involved in such a migration for you?

  • If you’re a casual SilverBullet user, e.g. using just the standard stuff like pages, basic markup, page links, stuff like that: nothing will really happen. The fundamentals don’t change.
  • If you’re using templates or queries: those blocks will simply not be rendered anymore and just appear as fenced code blocks. You’d have to manually translate your queries to Lua queries. This shouldn’t be hard (the syntax is 90% the same), and I’m happy to help in this process. You’d also have to manually translate templates in the same way (this may be slightly more involved, but ultimately more powerful as you learn the Lua syntax).
  • If you’re using space script, all of that will need to be converted to Lua. I think I’d also remove space script (although this one I may be talked out of). The scripts would simply stop being activated. Since space script allows for really random JavaScript to be run, there may be cases where you cannot translate this to Lua. However, I’d be very interested in those cases to see if we can somehow support with Lua after all.
  • Many (not included, third-party) plugs would likely break or need adjustment.

For a little while, some features may simply disappear. However, as you know I’m open to feedback. If you miss something, I’m happy to rethinking and add it back (but perhaps in a slightly tweaked more Lua-focused way).

If SilverBullet were a company with a full development team backing it, I wouldn’t propose this approach. But it’s not. It’s primarily me, and I have to balance where I spend my energy and keep my motivation high. This is why I’m proposing this more radical strategy.

Let me know what you think.

25 Likes
2

Really like the idea, two things tho:

  • There should be good docs, like really good docs for all the lua stuff then. Probably an extensive migration guide.
  • I know you proposed this before and I don’t know what exactly is on your list of things to remove, but is both sync mode and online mode really needed? I myself mostly use online mode, but only because it feels like the easier simpler thing mentally. There really isn’t a reason why somebody using online mode, couldn’t just use sync mode. And having worked on the codebase a little bit, it would properly cut a big, big chunk especially on the server.
6 Likes
3

Now you got me even more excited. Yes let me think about this a bit more, but oh my that would simplify things even more. Also making it more feasible to in time perhaps migrate (back) to Node.js, because I’ve become less convinced the Deno thing is worth it. It would make the backend very “dumb.” Which would be amazing.

3 Likes
4

Agreed. Actually I’d rewrite most of the docs largely from the ground up.

1 Like
5

I agree you should break with the current version and focus on the next one. I believe that the hacker mindset that made people use space scripts means they’ll be capable of handle the migration. I’m excited to start testing a lua version of Silverbullet!

5 Likes
6

Sounds good; I find myself slowly using Silverbullet and some of the advanced features more and more every day, but am definitely willing to put in the work to convert stuff for V2;

Especially since it sounds like this will make things a lot easier for you to develop and more consistent for us to use.

Sounds like this is a good way forward, looking forward to it!

2 Likes
7

I’m supportive of this. I’m happy to ditch the older JavaScript and switch to lua and legacy backwards compatibility introduces so much extra unnecessary work

1 Like
8

+1 for going all in on Lua. It would make everything so much more consistent. Rip things out!

I also echo the sentiment that we only need one client mode. I solely use online mode; I want all the heavy lifting to be done server-side. Helps out when viewing notes on a low-spec mobile device.

1 Like
9

Hello Zef

I agree that your health is more important to you than an overflowing silver bullet. As already mentioned from pjamar, I also think that most users can do the migration.
Take care of yourself and your family so that Silverbullet has a long future :wink:.

1 Like
10

Zef, your work on 1.0 has been amazing! Let’s do the move! I’m looking forward to 2.0! To be honest, I’m sort of waiting for it myself. I now have mixed environment and already started rewriting most of my space script functions to Lua about two weeks ago. I would also suggest to do some overall structure and CSS cleanups, because one can really struggle to change some things. There are CSS gurus in this community who can certainly give hand with this. SilverBullet 2.0 will certainly be even better, easier to maintain and with much more comprehensible internals.

6 Likes
11

I love the 2.0 idea as it removes a lot of uncertainty.
In my personal opinion, even a hacker’s mindset becomes vulnerable when it comes to an ever changing “API”. The lua thing lurking in the shadows, posed this threat.

Why would you build anything if it’s about become obsolete in the near future? I would not adopt it.

This “uncertainty period” would extend a lot as it requires major resources from Zef, just for the sake of compatibility not functionality. Scaring away newcomers.

For me, the 2.0 version is a way to shorten the transition period, by focusing only on the stuff that matters. It helps the project in the long run.
:crossed_fingers:

1 Like
12

I’ll vote for your 2.0 suggestion, too!

Short time after I fell in love with Silverbullet itself, the LUA idea came up and immediately I fell in love with it, too :slight_smile:
For me it feels a lot easier to learn (and I’m still in the beginning, anyway).

Only disadvantage would be, that it will take some time to convert all the collected PLUGS and snippets to LUA. But if we as community all work together, it could be easy manageable!

So, yes please cut all the “old stuff” and let`s start over with a cleaner system :slight_smile:
And, just like you said, anyone who wants/or needs to stay on the old code could easily fork…

However, I’m not sure, yet, about the sync-mode/online mode discussion, as I currently use both on different platforms. Basically I could live with only sync-mode, but I also like the online mode for PCs I’m not working very often or on my work-pc, where I do not really like my full set of data to be stored and synced every time…
But let’s see… :slight_smile:

2 Likes
13

Once space-lua hit the stable release, I started working with it and for me there’s no reason at all to still use space-script. I didn’t convert old commands/scripts yet, but it’s easy to port. The community is active enough to help each other out and small enough that there isn’t a huge amount of shared libraries already.

In short: all in favour of focusing on Lua.

Plus, this is your project. I appreciate you asking the community and maybe the collective mind has insights you didn’t think of yet, but if it feels right, it’s probably right. :slight_smile:

4 Likes
14

SilverBullet 2.0 for me also.

1 Like
15

yep, me too. Go the full Lua (I haven’t had time to do much space script customisation and will be glad to convert them

1 Like
16

I don’t know if we have bias to ask this question on the forum since people are active here are to some extent more keen on tinkering. But besides this concern, myself is pretty positive on the large migration. I myself also find it redundant to maintain two parts of interfaces, I understand that the costly maintaining would eventually slowdown how silverbullet will be improved overtime.

So yes, let’s have a large refactor and reintroduce everything in lua! I’ll be happy to migrate to the new silverbullet.

For migration, we should just have a dedicated section in the documentation explaining the migration should be okey.

2 Likes
17

I’m predominately more a sync mode user, but I want to voice something against it, that is it takes longer to load page on a new device when using sync mode because the full synchronization takes a bit of time.

On my personal case I just have a couple of fonts/images in my silverbullet space, my space is around 11M and roughly 300 pages but that already takes me everytime 1-3 mins for a new client to be toggled to sync mode. This given the backend log is because it is sequentially requesting each of the file from the server, not in parallel.

So one thing I would really want to make sure is if we can speed up the synchronization especially for setting up a new client. Otherwise default to sync mode would mean that you can’t have a quick edit on a new client until you wait for your whole space to be downloaded to the new device.

Another thing to take into consideration is if someones space contains more images or other stuff that it exceeds the storage limit of the browser. I’m not a web developer so correct me if I’m wrong to concern about this, but my information tells me that it is actually possible?

I think given my personal experience with using Silverbullet on daily basis, we can remove online mode once we can ensure the speed of synchronization and storage won’t be exceeded.

18

I support the scream migration to 2.0. I’ve already converted a most of space-scripts to LUA, still a couple to go, but they are waiting for the ability to call LUA during page creation mainly for creating dynamic paths (maybe I missed something).

Also adding my voice to the removal of online function. While my use of online is becoming rarer, I still do rely on it occasionally. Could I live without it? Yeah, I guess…but would be great if it stuck around.

19

I’ve never been sure what the difference is between sync and online mode. My main concern is that my data doesn’t end up on somebodies server somewhere. I prefer to keep my data under my control. How I deal with backups and moving it is my business. I’ve seen too many instances where people’s data is hacked from supposedly secure, well maintained servers. To say that my trust in someone else to secure my data is non-existent would be an understatement. So, would my data be on someone else’s computer/server?

20

I’m in favor of the move especially if the documentation is well done. I have limited my use of a few queries which should be easily translated to Lua. Besides, I’m learning Lua since it is not only integral to Silverbullet but also to Neovim configuration.