Log flooded with "indexing attachment: silverbullet.db-shm" and db-wal

sanzoghenzo · December 28, 2024, 3:53pm

Hi there!

I’m hosting silverbullet on my NixOS server; I might have done something wrong with the setup, but while the web/PWA app runs OK, I constantly get these entries in the log:

Detected file change silverbullet.db-shm 1735400821454 1735400854531
Outside file change: reindexing silverbullet.db-shm
Indexing attachment silverbullet.db-shm
Outside file change: reindexing silverbullet.db-shm
Indexing attachment silverbullet.db-shm
Detected file change silverbullet.db-wal 1735400851471 1735400881445
Outside file change: reindexing silverbullet.db-wal
Indexing attachment silverbullet.db-wal
Outside file change: reindexing silverbullet.db-wal
Indexing attachment silverbullet.db-wal

notes and the database are all in the /var/lib/silverbullet folder, is this the issue? should the index db be in a separate folder than the notes?

sanzoghenzo · December 31, 2024, 10:56am

Just for a sanity check: can anyone else see these messages or it’s only me?

I now have added the *.db* rule to the spaceIgnore section in my settings (and in.gitignore since I’m using the git plug) so everything is OK, but I thought that this would be something that silverbullet does automatically.

I must add that I had manually set up the envvar SB_KV_DB=/var/lib/silverbullet/silverbullet.db (no dot at the start of the filename), could it be that the outside file change detector ignores hidden files, so it is not a problem with the default options?

After some searching through the code base, I only found this, but I don’t know if it’s relevant (and it certainly doesn’t do anything even with the default database name)

github.com

silverbulletmd/silverbullet/blob/df9edc5d64711c970687e3547f610864f8578aa9/common/spaces/disk_space_primitives.ts#L15


      
          import type { FileMeta } from "../../plug-api/types.ts";
          
          function lookupContentType(path: string): string {
            return mime.getType(path) || "application/octet-stream";
          }
          
          function normalizeForwardSlashPath(path: string) {
            return path.replaceAll("\\", "/");
          }
          
          const excludedFiles = ["data.db", "data.db-journal", "sync.json"];
          
          export class DiskSpacePrimitives implements SpacePrimitives {
            rootPath: string;
            fileListCache: FileMeta[] = [];
            fileListCacheTime = 0;
            fileListCacheUpdating: AbortController | null = null;
          
            constructor(rootPath: string) {
              this.rootPath = Deno.realPathSync(rootPath);
            }

justyns · January 5, 2025, 3:47pm

I’m not 100% sure, but I think this is what excludes files that begin with .:

github.com

silverbulletmd/silverbullet/blob/f801fd263bd2eecdea1c6ef3da300c5acaf31a54/lib/plugos/syscalls/fs.deno.ts#L86


      
          ): Promise<FileMeta[]> => {
            dirPath = resolvedPath(dirPath);
            const allFiles: FileMeta[] = [];
            for await (
              const file of walk(dirPath, {
                includeDirs: false,
                // Exclude hidden files
                skip: [
                  // Dynamically builds a regexp that matches hidden directories INSIDE the rootPath
                  // (but if the rootPath is hidden, it stil lists files inside of it, fixing #130 and #518)
                  new RegExp(`^${escapeRegExp(root)}.*\\/\\..+$`),
                ],
                maxDepth: recursive ? Infinity : 1,
              })
            ) {
              const fullPath = file.path;
              const s = await Deno.stat(fullPath);
              allFiles.push({
                name: fullPath.substring(dirPath.length + 1),
                created: s.birthtime?.getTime() || s.mtime?.getTime() || 0,
                lastModified: s.mtime?.getTime() || 0,

And this is what triggers the attachment indexing:

github.com

silverbulletmd/silverbullet/blob/f801fd263bd2eecdea1c6ef3da300c5acaf31a54/common/spaces/evented_space_primitives.ts#L36-L40


      
            if (!name.endsWith(".md") && !name.startsWith(plugPrefix)) {
              // Not a page nor plug, so an attachment!
              await this.dispatchEvent("attachment:index", name);
            }
          },

I think if you change SB_KV_DB to /var/lib/silverbullet/.silverbullet.db, you’ll probably get rid of the issue. But the code probably should automatically ignore the db file. If changing SB_KV_DB works for you, it’d be a good idea to open up a github issue

sanzoghenzo · January 5, 2025, 10:08pm

Thanks for the pointers, I saw the indexing code, but missed the fs module!

As I wrote in my previous post, I solved by adding the files in the spaceIgnore in my SETTINGS file, but If I have to start over I’ll make sure to make it an hidden file.

I agree with you that silverbullet should automatically add the SB_KV_DB path (and the related wal and shm files) to the ignored paths.

zef · January 8, 2025, 7:32pm

Ok this just clicked. The reason you’re seeing this is indeed because your .db file does not start with a . SilverBullet excludes all files and folders that start with . and therefore would never attempt to sync it. Simply renaming your DB file to start with a dot should fix this.

Edit: sorry I missed @justyns message, indeed — what he said.