# Ideas on data records and indexing
**URL:** https://community.silverbullet.md/t/ideas-on-data-records-and-indexing/3032
**Category:** Ideas
**Created:** [August 15, 2025, 7:24am UTC](https://community.silverbullet.md/t/ideas-on-data-records-and-indexing/3032 "2025-08-15T07:24:17Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@mjf](https://community.silverbullet.md/u/mjf)
#### Post date: [August 15, 2025, 7:24am UTC](https://community.silverbullet.md/t/ideas-on-data-records-and-indexing/3032/1 "2025-08-15T07:24:17Z")
I think data records should have their own index. I use data records extensively. But I had to write extremely ugly methods in Space Lua to get pure data records contents out of current index. You must drop all the `pos`s, `ref`s, `itags`s etc. columns to get pure original data record which is fairly sub-optimal. Either native core methods should be provided for this (because if anything change, e.g., new column is added, it will break my Space Lua API) or I would beg for separate index for data records for the reasons given. For example:
```lua
local p = query[[
from
r = index.data 'debt'
where
r.parent.page = '/some/page' and
r.parent.itags.includes('foo')
select
r.content['Person']
]]
```
My API for handling data currently supports data blocks and table cells, because I use these two for tracking several things. For example:
| person | amount | remarks |
| --- | --- | --- |
| Joe | 100.0 | #debt |
| Jane | 300.0 | #debt |
````yaml
```#debt
person: Tom
amount: 50.0
```
```#debt
person: Jim
amount: 1000.0
```
````
I also think about using list items with annotations:
```markdown
- #debt [person: Eve] [amount: 100.0]
- #debt [person: Ron] [amount: 30.0]
```
The API queries for the tag `#debt` and the it must then do various cleanups for each kind of objects to get only the `person` and `amount` record fields back. It’s doable but I still think that to have data in separate index would be far more ergonomic and benefical. Perhaps I missed some better way. Be there any, please share with me.
Except the data blocks there is also no way to say “this tag represents data record”. There may therefor be some config option telling what tag is supposed to represent some data record, or it can be registered as data record on first `index.data 'tag'` call. The data records index will then start to function and indexing cleaned up (table cells, annotated list items - or any other tagged object with annotations) records (unioning them) for easy use in LIQ and Lua. 🙂
Please, share your opinion on how you would like to work with data records in Silverbullet.
---
### Author:  [@zef](https://community.silverbullet.md/u/zef)
#### Post date: [August 19, 2025, 8:11am UTC](https://community.silverbullet.md/t/ideas-on-data-records-and-indexing/3032/2 "2025-08-19T08:11:34Z")
Interesting. It wouldn’t be hard to add a convenience method like `index.data` that cleans things up. The question quickly becomes how much cleaning you really want. Generally the idea is that every object has an identifier in `ref`, so would you really like that removed?
---
### Author:  [@MrMugame](https://community.silverbullet.md/u/MrMugame)
#### Post date: [August 19, 2025, 9:42am UTC](https://community.silverbullet.md/t/ideas-on-data-records-and-indexing/3032/3 "2025-08-19T09:42:54Z")
Also, wouldn’t we have to differentiate between every single type of object in such a function, because one object e.g. has a `parent` field which is internal, but another object doesn’t. And if that other object has a `parent` field it’s part of the data and we shouldn’t delete it.
My programming mind would just index all the data under it’s own property, unsure what issues this would cause tho. For something like attributes on tasks it would probably be a BIG backwards compatibility problem.
---
### Author:  [@mjf](https://community.silverbullet.md/u/mjf)
#### Post date: [August 20, 2025, 11:08am UTC](https://community.silverbullet.md/t/ideas-on-data-records-and-indexing/3032/4 "2025-08-20T11:08:14Z")
We could (1) have separate indices for different kinds of objects and the current `index` could be constructed on the fly (and cached), maintaining the current functionality and backwards compatibility quite easily. Or we can (2) have it the other way round and have `index.data` be constructed on the fly (and cached), with cleaned up data records (which can be done, reliably and ensuring forward/future compatibility, only via some native API provided by Silverbullet). The latter one seems much harder to maintain to me (not speaking of ambiquities that may arise), while the former one is harder to implement (much more code to be written but improving flexibility to the future too as a benefit, IMHO).
I’m not expert in Silverbullet internals but as a user I must say that to for me the possibility to have “live” data in my pages and to mix spreadsheet- or SQL-like functions with documents, all in single page, is the expression of the power of Silverbullet. It’s doable now, but not without great deal of messy code in Lua and without any promises to the future.
---
### Author:  [@zef](https://community.silverbullet.md/u/zef)
#### Post date: [August 31, 2025, 5:31pm UTC](https://community.silverbullet.md/t/ideas-on-data-records-and-indexing/3032/5 "2025-08-31T17:31:24Z")
Possibly related or can be combined with: [Data schemas and "real" SQL](https://community.silverbullet.md/t/data-schemas-and-real-sql/3110)