I think data records should have their own index. I use data records extensively. But I had to write extremely ugly methods in Space Lua to get pure data records contents out of current index. You must drop all the poss, refs, itagss etc. columns to get pure original data record which is fairly sub-optimal. Either native core methods should be provided for this (because if anything change, e.g., new column is added, it will break my Space Lua API) or I would beg for separate index for data records for the reasons given. For example:
local p = query[[
from
r = index.data 'debt'
where
r.parent.page = '/some/page' and
r.parent.itags.includes('foo')
select
r.content['Person']
]]
My API for handling data currently supports data blocks and table cells, because I use these two for tracking several things. For example:
person
amount
remarks
Joe
100.0
#debt
Jane
300.0
#debt
```#debt
person: Tom
amount: 50.0
```
```#debt
person: Jim
amount: 1000.0
```
I also think about using list items with annotations:
The API queries for the tag #debt and the it must then do various cleanups for each kind of objects to get only the person and amount record fields back. It’s doable but I still think that to have data in separate index would be far more ergonomic and benefical. Perhaps I missed some better way. Be there any, please share with me.
Except the data blocks there is also no way to say “this tag represents data record”. There may therefor be some config option telling what tag is supposed to represent some data record, or it can be registered as data record on first index.data 'tag' call. The data records index will then start to function and indexing cleaned up (table cells, annotated list items - or any other tagged object with annotations) records (unioning them) for easy use in LIQ and Lua.
Please, share your opinion on how you would like to work with data records in Silverbullet.
Interesting. It wouldn’t be hard to add a convenience method like index.data that cleans things up. The question quickly becomes how much cleaning you really want. Generally the idea is that every object has an identifier in ref, so would you really like that removed?
Also, wouldn’t we have to differentiate between every single type of object in such a function, because one object e.g. has a parent field which is internal, but another object doesn’t. And if that other object has a parent field it’s part of the data and we shouldn’t delete it.
My programming mind would just index all the data under it’s own property, unsure what issues this would cause tho. For something like attributes on tasks it would probably be a BIG backwards compatibility problem.
We could (1) have separate indices for different kinds of objects and the current index could be constructed on the fly (and cached), maintaining the current functionality and backwards compatibility quite easily. Or we can (2) have it the other way round and have index.data be constructed on the fly (and cached), with cleaned up data records (which can be done, reliably and ensuring forward/future compatibility, only via some native API provided by Silverbullet). The latter one seems much harder to maintain to me (not speaking of ambiquities that may arise), while the former one is harder to implement (much more code to be written but improving flexibility to the future too as a benefit, IMHO).
I’m not expert in Silverbullet internals but as a user I must say that to for me the possibility to have “live” data in my pages and to mix spreadsheet- or SQL-like functions with documents, all in single page, is the expression of the power of Silverbullet. It’s doable now, but not without great deal of messy code in Lua and without any promises to the future.
