Markdown customizations and external Lua programs

Lua
1

Hi again,

TL;DR:

Authors using SB are asking for a short syntax to include metadata in the page or a custom Markdown syntax, which are requirements that I and others have found many times before while authoring in Markdown. And given that I have already approached such problems previously in Lua (and I don't know TS to go into the current markdown_parser) I wonder:

How can I reuse Lua Parser Expression Grammars (LPEG) and pure Lua libraries like Lustache and Lunamark to reuse Markdown syntax customizations, like page metadata tranclusions, Pandoc/Djot's div syntax for admonitions and others, without reimplementing them? In general, How can Space Lua reuse Lua scripts and libraries that are used in Markdown/document processing elsewhere so SB can integrate better to existing rich documentation workflows and ecosystems?

TIA

(The long version)

This inquiry concerns how to to integrate SB with Cardumem, a Lua based Static Site Generator (SSG) (still in alpha), that I have made to suit our documentation/publishing workflows in grassroots communities (like HackBo, our local Hackerspace) and at the university department, in the classes, workshops and research/prototyping projects across academic and non academic spaces/contexts. I'll explain how our documentation workflow looks like and where I would like to integrate SB. I will try to be pretty detailed as this particular concern could document how SB can be integrated into rich pre-existing documentation workflows and ecosystems.

Currently our publications workflow looks like this:

image
image1018×560 31.2 KB

So, we write Markdown content in our workshops/classes using the real time collaborative editor HedgeDoc, Cardumem downloads/renames that information to the hard disk and parses (Markdown) content, (YAML) metadatada and Mustache (HTML + CSS) logicless templates to produce a HTML output that we store in a Fossil repository, kind of a GitHub/Gitea/Forgejo in a box (3 Mb), providing us collaboration, versioning, web server, ticketing/issues and forum. We use also Codium, the non-panoptic and telemetry disabled variant of VSCode, for wiki refactoring (more on that and the role of SB for our audiences later).

As many other SSGs, we need to parse, separate and process YAML metadata from Markdown content, while dealing with the popularity and insufficiency of the Markdown (sub)"standard" (despite of important efforts like CommonMark) by providing custom syntax for everything that plain Markdown doesn't allow, including transclusions, an important hypertext feature about how to make content modular and "assemblable", that has been historically overshadowed by linking...

Anyway, part of my thinking has been related with how to reuse the libraries and functionality we already have for Markdown processing to deal with this format's (many) limitations, making its syntax extensible, while offering a gentle curve to the authors, so they can traverse from content to functionality creation and viceversa, as they need.

For example, given that I already have Lua's Lustache to process logicless templates, Cardumem provides page metadata transclusions using the same Mustache syntax. So, given this page metadata:

---
# This is part of this page YAML metadata.
example: "**I'm just a brief text**."
mutabit: >
  ===== <br>
  [mutabiT](https://mutabit.com/) is the small family enterprise that funds and sustains most of this long research and agile prototyping efforts, since 2003.
musketeers: ["Athos", "Aramis", "Porthos", "D'Artagnan"]
---

I can call those fields in Cardumem's Markdown with {{example}} , {{mutabit}} or {{#musketeers}} {{.}} {{/musketeers}}, to get them properly rendered inside the page.

Syntax extension reuse YAML + Mustache processing, so authors/collectives can create their own customized portable macros by defining them in a simple macros.yaml file, with nested keys containing the name of the macro, its input and how it is used in its output, again as a Mustache template. Similarly, global transclusion keys are defined in a transclusions.lua file, which allows, for example, to call {{_date}} in any page and get the current date when such page is exported as HTML.

Another example is the usage of Lunamark to process not only standard Markdown, but also its customizations, using Lua PEG, like in our admonitions, that follow Djot's div syntax, which is also used by Pandoc and HedgeDoc, as my policy regarding Markdown syntax extension is: "when in doubt, look at Pandoc/Djot as it's the variant that deals with most export/import scenarios.

Given the static/dynamic nature of Cardumem/SilverBullet, respectively, and that both have a Lua extension story, I would like to make each one the complement of the other: SB can be used for live coding and wiki refactoring of our documentation, instead of Codium, which suits better our general non-developers audience and Cardumem can work on the static site generation.

Finally, what I would like is to make SB to reuse Cardumem's syntax and hopefully some of its scripts. And make the bridge between both using Lua, not TypeScript. That's why I asked the short question at the beginning of this post.

Cheers,

Ps: While writing this, I remembered the Pascal quote: I made this one [letter] longer only because I have not had the leisure to make it shorter.

2

This answer is AI-generated.

I can't give a definitive answer, but here is what the Space Lua docs say and where I'd expect the limits to be:

  1. Space Lua is SilverBullet's own Lua implementation. Its scripts live in space-lua blocks in your pages, are loaded when the client boots (or on "System: Reload"), and their load order is controlled with a -- priority: <number> comment. I couldn't find documentation saying Space Lua can require arbitrary external Lua files or libraries from disk, so I'd treat that as unsupported unless the maintainers say otherwise.
  2. LPeg is a C library, so I wouldn't expect it to run inside Space Lua. Pure-Lua libraries such as Lustache might work if you paste their source into a space-lua block, but that depends on which standard-library functions they use. I'm not sure which ones Space Lua implements, so test it on a copy of your space and check the browser console, which is where the docs say load-time errors show up.
  3. For metadata transclusion, the docs describe the ${lua expression} syntax, which renders the result of a Lua expression in Live Preview. That might cover your {{field}} use case without a separate template engine.
  4. Since your site generator already works on the Markdown files on disk, another option is to leave the custom syntax in the files and let your generator handle it at build time, using SilverBullet only for editing. It's probably worth asking the maintainers whether a custom syntax hook is planned.
3

Thanks. But answers like this are precisely why I try to avoid AI (or Apparent Intelligence as I like to call it) as much as I can and used it "surgically" pointing to code bases I know or over questions with few lines of code as an answer that I can verify quickly. I would wait for a human answer made in context and less generic.

Cheers,

4

Fair point — the last reply was too generic.

On the specific question (reuse LPeg / Lustache / Lunamark from Space Lua):

  1. Space Lua is SilverBullet’s own Lua implementation (not Lua/LuaJIT/WASM). C libraries like LPeg are not available inside it.
  2. Definitions live in space-lua fenced blocks in pages and load on boot / System: Reload. I still don’t see a documented way to require arbitrary Lua files from disk the way Cardumem does.
  3. Pure-Lua code can be pasted into a space-lua block and tested, but anything that depends on C bindings or a full standard library will fail — check the browser console after Reload.
  4. Practical split that matches your setup: keep Cardumem’s custom Markdown/Mustache/Lunamark pipeline on the files for static export; use SilverBullet for editing/live refactoring; use ${…} only for live preview inside SB. For a real syntax bridge (custom admonitions, Mustache-style macros in SB itself), that needs a maintainer answer — I’m not going to invent one.

If that still isn’t useful, ignore this and wait for a human who works on SB day to day.

5

Yep, I'll wait for the human. Thanks again.

6

This is a lot of information to process, and I don't have the capacity to dig into each of these links and projects. Let me just give a few pointers that may or may not answer your question:

  1. Space Lua is a reimplementation of Lua and it's "hard wired" in, so you can not just swap out Lua dialects.
  2. There's a very simple extension point to extend the syntax right now: syntax
  3. Space Lua is deeply wired into SilverBullet itself, so cannot easily be pulled out and mixed and matched with other tools.
7

Thanks Zef for pointing me to the syntax extension point. I'll look at it.

And yes I tend to overwrite... it is some academician increased decease and also my way to raise awareness of local/Spanish projects that would be little to not known except for our participation in forums like this. Instead, to share the way SB will be part of our "ecosystems of memory", I'm planing to write some blog post and sharing some workshops that I will give to colleagues and undergrad /post grad students, using SB inside such ecosystems.

Coming back to my question, my idea is not to replace/swap Lua dialects. Instead what I want to do it to use the fact that Space Lua is a Lua to use external Lua programs I have used before to deal with inherited Markdown customizations, like those coming from HedgeDoc and Pandoc, and also the ones we have devised our selves to attend particular writing needs.

For example, I use TinyYAML combined with Lustache to support Mustache templating syntax exemplified in this very thread when calling page metadata inside Markdown. This allow us to take advantage of the logicless nature of Mustache, so authors can make document/template customizations and call metadata without even knowing any programming and instead using the Mustache concepts: presence, absence, plain, nested, unique, multiple, partial and complement. Given that both, TinyYAML and Lustache, are pure Lua packages, how can I use them in Space Lua?

Ultimately, given that Growing with you implies also Meeting where you are, I wonder how SB and Space Lua can benefit of the huge amount of pure Lua packages and/or those provided by Luarocks, to create a better synergy with the wider Lua community and its use cases around the globe. I believe this may increase SB awareness and utility, given the use Lua has in places like academic writing (with Pandoc), programming education for different ages/platforms, and so on.