Space Lua: Making Space Script safer, simpler and unifying all the things

So over the last few weeks I’ve been tinkering with PICO-8, which is a nice game development environment to play with. Recommended if you’re looking into introducing your kids to programming (as I am). As a programming language it uses Lua, which is a programming language I was somewhat familiar with, but not deeply.

I dug into Lua a little more, and I like it quite a bit. First of all, because it’s a pretty simple, yet powerful language. The syntax is quite limited, so it’s easy to learn, and probably not all that hard to implement.

And then I started to think… could Lua be a solution to a design problem I’ve been thinking about for a few months now…

Here’s the issues I see that Lua may solve:

Security

Space script is a pretty cool feature, but also unsafe. The way it works is that it effectively just evals whatever JavaScript you throw at it on both the client and server. If you write this code yourself, this should be all good (unless you really do scary things), but the moment you start to copy & paste random scripts without understanding what they do, we get into scary territory. It’s pretty hard to sandbox running random JavaScript code in a browser environment (ironically). Some other tools do this by Wasm compiling a JavaScript interpreter and then using that, which is pretty heavy weight.

There are some ways of running/using Lua in the browser:

  1. You can WASM compile the official Lua interpreter and run it in the browser, for instance: GitHub - ceifa/wasmoon: A real lua 5.4 VM with JS bindings made with webassembly
  2. There is a JavaScript interpreter available: GitHub - fengari-lua/fengari: 🌙 φεγγάρι - The Lua VM written in JS ES6 for Node and the browser
  3. There is a Lua grammar available for the Lezer parser (which SB uses for all its editor parsing) that seems pretty complete: GitHub - R167/lezer-lua: Lezer-based parser for Lua this doesn’t run Lua, but it parses it. And… building an interpreter on top of this seems feasible if you know what you’re doing (famous last words).

So with some options on the table, it seems feasible to offer space-lua as an alternative to JavaScript and make it safe to run.

Accessibility

While JavaScript is a very popular language, it is also a complex one with lots of weird “features”. I’ve noticed that a fair number of SB users are writing space script, but don’t always really know the language. They mostly use ChatGPT to help them.

Lua can be a solution here too, because it’s an easier language to learn with fewer weird edge cases.

Unification of template language, expression language and space script

But here we get to the the thing that gets me even more excited. It is possible that we can replace SB’s expression language, and potentially tweak the template language to use Lua expressions instead, and to use it for queries as well. This would mean that SB would no longer have a fully custom expression language, but that those expressions would just support arbitrary Lua expressions, which would make them even more powerful.

Already, syntactically it’s quite surprisingly close and I’m thinking if this can be done in such a way that it is largely or entirely backwards compatible.

The Plan

There’s a big caveat with the Lua implementations that exist today, and that is that to allow Lua code to call asynchronous APIs (e.g. any syscall), you’d have to write Lua code with callbacks, which would become quite ugly. I already have a solution to work around this for SB’s expression language (where you can mix synchronous and async calls transparently), and I think I can expand that to a full Lua implementation.

This would mean that I’d end up writing a custom Lua interpreter for SilverBullet. I think this is feasible, and can be done in reasonable time. I happen to (literally) have a PhD in compiler construction and programming language design, so this is fact not the first time I’d do something like this. But you know… famous last words.

What this would mean for SilverBullet

I wouldn’t remove space script as it is today, perhaps ever, but we’d add a second option, let’s call it space-lua. This option would be enabled on all SilverBullet instances, because it would be perfectly secure. This would also mean that many built-in functions can be reimplemented in Space Lua and distributed via Libraries, which is something I’m not 100% comfortable doing right now with space script.

Space Lua would be the recommended new way of extending SB with custom commands etc. basically the use cases of Space Script today. Just safer and with a simpler language.

It would mean that SB’s Expression Language would become Lua Expressions. I still have to see if I’d tweak Lua expressions slightly to also support e.g. the != operator (which we use today) in addition to Lua’s version /= so that we can maintain backwards compatibility, but beyond that it should largely make the expression language more powerful than it is today.

I still have to see what this would look like, but it’s also conceivable that we’d expand/replace the template language with a Lua version. Conceptually:

{{for page in all(query("page"))}}
* {{page.name}}
{{end}}

Or something along those lines, or even expand the Lua syntax with SB queries natively.

Anyway, I’m thinking out loud here, but I’m really warming up to this plan.

Let me know what you think.

7 Likes

I agree that space-script is unsafe, and we need to find a way to fix it.

Regarding ‘Accessibility,’ I think the reason some users rely on AI to write space-script isn’t because JavaScript is hard to learn, but because they either don’t know how to program or aren’t familiar with any programming languages. I believe that any programmer can learn the basic of JavaScript, enough to write space-script, in a short time. The real obstacle is the time it takes to understand the terms and the API to interact with the underlying SB system. I don’t think Lua can help with this issue.

3 Likes

Unification of template language, expression language and space script

This seems like it’d really neat. There are times I wished the expression/template language was just a little closer to a full language. Even more bonus points if it can be done in a backwards compatible way.

One of the nice things about space-script currently is that it is “just” javascript, so there isn’t really much you can’t do with it. The javascript ecosystem is also (I assume) larger than lua’s, so users newer to programming in general would probably be able to find more libraries or examples of what they want to do in javascript.

Is there anything obvious that we wouldn’t be able to do with lua, like could a lua script still access and manipulate the DOM?

Maybe lua would eventually become the new space-script, and anything that can’t be done using lua would have to (or recommended to) become a js plug that exposes functions for lua to use?

1 Like

This would mean that I’d end up writing a custom Lua interpreter for SilverBullet.

This kinda worries me a little bit. Not that you won’t be able to write a interpreter, but that over a longer term ends up being a bigger time sink. Also as justyns mentioned, I too feel like accessing the DOM ends up being harder when necessary.

Javascript also has the added advantage of being something more people just know and this is a useful trait especially since SB is a web app. This is even useful when using llms as they probably had more training data for js.

I also wanted to ask about your thoughts on using lua for plugs in SB. The way I see it now, once I have a big enough space script I can convert it to a plug with some effort. If they use different languages, this becomes harder.

And finally on debugging. As of now, I can put in a bunch of console log or even a debugger statement and debug space-scripts. I’m guessing this would get much harder or we will have to put more work into making it more easily debuggable.

PS: At the end of the day, its your project and these are just my thoughts. Also, if lua gets added, it is likely that I’ll use it more as I prefer lua over js except for how it does regex.

4 Likes

I happen to (literally) have a PhD

Does this mean we should start calling you Dr Zef?

3 Likes

Blockquote I wouldn’t remove space script as it is today, perhaps ever

I would probably argue to improve security to move space script to something like this once it is (hopefully) implemented in most browsers:

You are actually right on the accessibility argument. It’s not a very strong one. I do see that the barrier to learn Lua is lower than many other languages, but it’s probably not a make or break factor. And generally, JS is more well known and in fact perhaps a selling point if you already know it.

The original design objective of the plug-based plugin system was to limit what you can do to a safe subset. This is more or less in place. It’s not very well contained, but you don’t have access to everything, specifically the DOM. Which is one of those areas where users can easily break things and do dangerous things.

In Space Script, all bets are off. You have access to absolutely everything, no constraints (beside whatever the JS runtime in the browser and Deno on the server put on you).

What Lua would bring is get is closer to the original Plug secure sandbox concept. You only get access to specific APIs (specifically: all syscalls), but not random access to the DOM or random JavaScript APIs. This would indeed be less flexible, you cannot hack things to degree that you can do today, but that flexibility comes at the cost of security. Not to scare people, but I could write a space script that implements a key logger right now and live streams everything you type to my server. The only thing stopping from this actually happening is that (hopefully) people don’t blindly copy & paste space script into their space without understanding what it does. But this is a big assumption.

As I mentioned, this would bring the development closer to what you can do in plugs today: it would be a more sandboxed environment where many things are blocked, except for whatever syscalls offer. So no, you wouldn’t have access to the DOM, unless that would be explicitly be exposed.

Yes, this is an option.

1 Like

I’m offended every time somebody doesn’t call me that, to be honest. But I’ve learned to live with it.

1 Like

The risk of becoming a time sink is absolutely there. My estimate is that the effort to build a Lua interpreter is doable (and I would do it even just for the fun of it), but I can really underestimate this. Lua is a small language, it doesn’t evolve much. But again: famous last words.

True.

This is also a good point. I don’t have a good solution to this. Other than to say “LLMs solve everything, maybe just ask it to translate.” It could actually work :laughing:

Can you put debug statements in space scripts? That’s cool, I didn’t know. My assumption would be that space scripts are generally small enough to debug in primitive ways (this is what I do anyway), that is: println debugging, just console.log or the Lua equivalent (print). If you are going to do bigger things, it’s probably better to develop it as a plug using TypeScript.

I love pico8 and lua, but not a big fan of introducing another language. Why not just sandbox JavaScript evaluation? I moved from logseq to SB mostly because clojurescript was terribly slow, had a limited (public) api and was a pain to use.

1 Like

How would you do that?

Have you considered executing code on an js engine compiled to WASM, such as QuickJS? The first googled article Execute JavaScript in a WebAssembly QuickJS Sandbox - DEV Community. Or maybe build deno_core to WASM Roll your own JavaScript runtime.

Yes, I mentioned this option in the original post. It’s an option, but pretty heavy weight to load and run an additional entire JavaScript engine to run a little bit of space script code.

Maybe this polyfill for ShadowRealm could help?

Hi Dr. @zef (since you want it), I think a it would be nice if we can unify space-script, expression language and template. Yet using lua might not converge with this ambition, it is still to be considered if lua should be the solution to this. I’ve never had experience of developing such large scale software like Silverbullet, but I do think it would be nice if you start to consider what you can achieve and how beneficial it is if you have a perfect silverbullet lua complier.

As a long-term neovim user, I do think the simplicity of lua language doesn’t mean lua is simple to work with, especially as lua is used underneath some app (like neovim or future lua-Silverbullet) without a proper debugger.

But still, it is such a inspiring thing to think about if I can write lua in silverbullet :grinning:.

TL;DR: I agree, but…

Choice of Lua

I’ll start with a disclaimer that I am a Lua fan. By chance, I got back to PICO-8 recently, and made a small game on a game jam.

Lua is embeddable

Lua is a fast language engine with small footprint that you can embed easily into your application. — Lua: about

It is really pleasant and well-documented for this use. I tried integrating it in a small game of mine built in Zig, and it went without any issues, even though I was clueless about the build system when starting.

In fact, it’s so embeddable that we used to write mission logic for flying drones using ArduPilot Lua scripting, running on tiny boards.

The team I was working with was split between engineers who loved it for the scripting possibilities it gave us, and ones that hated it for its “weirdness”, that it’s not the C-style if (cond) { statement }, or that the arrays are just tables and are traditionally indexed from 1 (not zero). In my experience the latter group was mostly prejudiced and came to appreciate having a scripting language where any other would be too heavy.

Having said all the positive things about Lua the language, the Lua plugin hell in Neovim made me quit it for Helix.

Custom interpreter

I’m afraid that with custom interpreter the first 80% will be a lot of fun, but the last 20% of compliance will be 80% of the work and none of the fun. At least that’s the typical comment under “look, I made a Scheme interpreter”. But don’t let a bunch of users stop you from doing the fun stuff :smiley:

I wonder if our need for async/await could be somehow be handled with Lua’s coroutines, the last section in that chapter even shows an approach to non-blocking HTTP.

Current ecosystem

Maybe it would be informative for this to get a summary and classify the space scripts currently shared here and on GitHub into internal data processing, and external APIs. The first would be the plethora of date/time manipulation utilities, and the other scripts that access DOM events (someone did scroll?), location etc.

I would really like that. I am tempted to say that anything that communicates externally should be a Plug, but in practice I think we’ve seen much more ideas being implemented and shared with the Space Script’s low barrier to entry with immediate results.

I’m split on HTTP requests, aka fetch API. It feels somewhere in-between, as I think it enables a lot of use cases, with single exit point, that could be reasonably monitored. Maybe some whitelist of domains? Or a prompt:

script is trying to access http://example.com/api/data/42 :
allow once / always allow example.com / deny

I think VS Code, Adobe Reader, and also pop ups in browsers do that, and it feels a good compromise between safety and convenience

Replacing custom languages

PICO-8 has a slightly modified Lua, the main issue I ran into was their custom operators not being recognised by any tooling. This will need to be documented, and I believe that with any such deviation you would lose the “it’s just Lua now” advantage. This means extra documentation and extra confusion especially with LLM-guided beginners.

I am like a broken record insisting on making if fully Lua, but I am working MBDyn at the university, and a custom language for a software with which you only interact through text is a central issue, both for new users learning curve, every day experience, and for maintainers trying to test or modernize it.

I think that there’s a bit of an uncomfortable choice between:

  • backward compatibility with SilverBullet
  • full compatibility with Lua
  • partial compatibility with both, so neither fully
More on backwards compatibility

I remember recently listening to Chris Lattner (probably in this interview) saying that they did breaking changes in Swift, but they provided tools to automatically translate old code to new to make it acceptable tradeoff. Just dropping it here, I understand there’s a difference between Developer Experience and User Experience when it comes to using tools.

Accessibility

Again, coming back to the recent PICO-8 experience, we almost immediately split the code into separate file with import statement to be able to edit it in an external text editor with full language support.

What if .lua files in the Space could be opened and edited in the client, and they were parsed the same way as if they were a page with one space-lua block? Then any code editor could be used (likely over SSH), and provide completion hints for more discoverability. ArduPilot mentioned before is a bit similar environment, where almost everything scripts do is just calling C from Lua. After configuring where the Lua definitions are the experience and discoverability of the API became much better.

On the other hand, SilverBullet is displayed in a browser, which comes with a JavaScript REPL built-in…

5 Likes

I’ve been thinking about this topic on and off since I saw it a few weeks ago. I was thinking that it would align well with the hacker philosophy of SB to not define what language you HAVE to use for space-script. Instead define the interface/spec and allow users to decide what language they WANT to use for space-script, that way the community can extend the available languages as they desire. This would allow people to continue to use the current js implementation if they want, and at the same time allowing ideas like this the ability to come to life.

Still early days, but making good progress. I’m sure there’s still a lot of bugs, but there is now a pretty complete Lua interpreter.

Preview of current state (I have to play with this more to figure out what a nice Lua API would look like):

1 Like

This would be nice in principle. In theory it would be possible to define some common interface that can be implemented e.g. through WebAssembly. The whole async programming aspect will still be an issue though. For instance, you could imagine cross-compiling Python to WebAssembly and run it in SilverBullet, but then any syscall you make would require passing a callback function, which would get quite ugly.

The “unique” thing I’m handling in the current (custom) Lua implementation I’m doing is that sync and async calls are handled completely transparently, which leads to much nicer code to write.