Indexing and System Readiness Issues

miro · June 18, 2025, 7:47am

Hi! I have been ‘battle testing’ Silverbullet for a while (2.000 page files, 100kb each), and I keep running into an indexing issue.

What can I do to provide more details?

My observation:

Indexing takes a very long time (~10min, though given the amount of pages this might be OK). I’m unsure whether I have to wait for indexing to complete fully before making any changes
Browser CPU usage during indexing is very high (in the browser, not the NAS server running Silverbullet as most activity happens client-side. Running on MacBook M1 Pro)
I keep seeing top_bottom_panels.ts:12 System not yet ready, not rendering panel widgets. messages and the app is non responsive (e.g. opening a page does nothing) for a long time. After some time the page might load, but then the browser tab has frozen
Even after indexing completed, I see error messages and e.g. page navigation is broken:

sync_service.ts:298 Syncing file copy202.md
09:46:51.707 service_worker.ts:150 [Service worker] Fetch failed: TypeError: Failed to fetch
    at service_worker.ts:108:16
    at service_worker.ts:149:10
(anonymous) @ service_worker.ts:150
Promise.catch
(anonymous) @ service_worker.ts:150Understand this warning
09:46:51.707 http_space_primitives.ts:41 
            
            
           GET https://myserver.org/20copy%20202.md 503 (Service Unavailable)
authenticatedFetch @ http_space_primitives.ts:41
getFileMeta @ http_space_primitives.ts:203
getFileMeta @ plug_space_primitives.ts:102
syncFile @ sync_service.ts:307
await in syncFile
(anonymous) @ client.ts:280Understand this error
09:46:51.708 sync_service.ts:352 Sync error Error: Offline
    at ps.authenticatedFetch (http_space_primitives.ts:43:15)
    at async ps.getFileMeta (http_space_primitives.ts:203:17)
    at async ds.syncFile (sync_service.ts:307:23)
syncFile @ sync_service.ts:352
await in syncFile
(anonymous) @ client.ts:280Understand this error

System setup:

Docker on NAS exposed as website
MacBook M1 Pro to open website

Thank you.

miro · June 19, 2025, 8:46pm

Here’s AI’s attempt of investigating this issue…

SilverBullet Indexing/Sync “Offline” Errors Analysis

During initial indexing (app startup), SilverBullet experiences frequent “Sync error Error: Offline” messages every few seconds. This analysis identifies the root cause as timing conflicts between multiple concurrent sync intervals combined with aggressive fallback behavior when local storage is empty.

Key Finding: The errors are not genuine connectivity issues but rather a “perfect storm” of competing sync mechanisms overwhelming the network layer during the critical initial sync period.

Root Cause Analysis

The Problem Pattern

client.js?v=cache-1750156356381:13 Sync error Error: Offline
    at ps.authenticatedFetch (client.js?v=cache-1750156356381:13:52392)
    at async ps.getFileMeta (client.js?v=cache-1750156356381:13:54672)
    at async ds.syncFile (client.js?v=cache-1750156356381:13:49656)
    at async ds.scheduleFileSync (client.js?v=cache-1750156356381:13:48385)

These errors occur at predictable intervals during initial indexing due to mathematical collision of multiple timer-based sync operations.

Core Architecture Components

1. Space Primitives Stack

EventedSpacePrimitives(
  FallbackSpacePrimitives(
    DataStoreSpacePrimitives,    // Local IndexedDB storage
    HttpSpacePrimitives          // Remote server requests (source of "Offline" errors)
  )
)

2. Competing Sync Intervals

Component	Interval	File	Line	Purpose
Space Sync	17s	`web/sync_service.ts`	30	Full space synchronization
Sync Check	8.5s	`web/sync_service.ts`	252	Check if sync needed (spaceSyncInterval/2)
Page Sync	6s	`web/client.ts`	280	Sync currently open file
File Watch	5s	`web/space.ts`	9	Monitor file changes
Cron Events	1s	`web/client.ts`	254	Background operations

Detailed Error Flow Analysis

Timeline of Events

T+0ms: App Boot (web/boot.ts)

Fetch client config
Setup service worker
Initialize client

T+100ms: Client Initialization (web/client.ts:176)

async init() {
  // Setup IndexedDB
  const kvPrimitives = new IndexedDBKvPrimitives(this.dbPrefix);

  // Create space primitives stack
  const localSpacePrimitives = new EventedSpacePrimitives(
    new FallbackSpacePrimitives(
      new DataStoreSpacePrimitives(this.ds),
      this.plugSpaceRemotePrimitives  // HTTP layer
    )
  );

  // CRITICAL: Initial sync hasn't completed yet
  if (!await this.hasInitialSyncCompleted()) {
    this.space.spacePrimitives.enablePageEvents = false;
  }
}

T+200ms: Multiple Intervals Start (web/client.ts:272-280)

// 1. Page sync every 6 seconds
setInterval(() => {
  try {
    this.syncService.syncFile(this.currentPath(true)).catch((e: any) => {
      console.error("Interval sync error", e); // "Offline" errors appear here
    });
  } catch (e: any) {
    console.error("Interval sync error", e);
  }
}, pageSyncInterval); // 6000ms

T+235ms: Sync Service Starts (web/sync_service.ts:234-252)

start() {
  this.syncSpace().catch(console.error); // Immediate sync attempt

  // Check every 8.5 seconds for full sync
  setInterval(async () => {
    if (!await this.isSyncing()) {
      const lastFullCycle = await this.ds.get(syncLastFullCycleKey) || 0;
      if (Date.now() - lastFullCycle > spaceSyncInterval) {
        await this.syncSpace(); // Another sync attempt
      }
    }
  }, spaceSyncInterval / 2); // 8500ms
}

T+250ms: File Watching Starts (web/space.ts:164-173)

watch() {
  this.watchInterval = setInterval(() => {
    safeRun(async () => {
      if (this.saving) return;
      for (const fileName of this.watchedFiles) {
        await this.spacePrimitives.getFileMeta(fileName); // More HTTP requests
      }
    });
  }, pageWatchInterval); // 5000ms
}

Error Generation Mechanism

Step 1: HTTP Request Fails (lib/spaces/http_space_primitives.ts:97-118)

} catch (e: any) {
  // Network errors detected by substring matching
  const errorMessage = e.message.toLowerCase();
  if (errorMessage.includes("fetch") || errorMessage.includes("load failed")) {
    console.error("Got error fetching, throwing offline", url, e);
    throw new Error("Offline"); // Error thrown here
  }
  throw e;
}

Step 2: Fallback Mechanism Amplifies (lib/spaces/fallback_space_primitives.ts:48-72)

async getFileMeta(name: string): Promise<FileMeta> {
  try {
    return await this.primary.getFileMeta(name); // Local storage (often empty)
  } catch (e: any) {
    try {
      const meta = await this.fallback.getFileMeta(name); // HTTP request - "Offline"
      return { ...meta, noSync: true };
    } catch (fallbackError: any) {
      console.error("Error during getFileMeta fallback", fallbackError.message);
      throw e; // Original error propagated
    }
  }
}

Step 3: Sync Service Catches and Reports (web/sync_service.ts:348-352)

} catch (e: any) {
  this.eventHook.dispatchEvent("sync:error", e.message).catch(console.error);
  console.error("Sync error", e); // "Sync error Error: Offline"
}

Mathematical Collision Analysis

The “every few seconds” pattern is caused by Least Common Multiple (LCM) collision of intervals:

5000ms (file watching) and 6000ms (page sync) collide every 30000ms
Individual conflicts occur much more frequently:
- T+5000ms: File watching
- T+6000ms: Page sync (collision potential)
- T+8500ms: Sync service check
- T+10000ms: File watching
- T+12000ms: Page sync (collision potential)
- T+15000ms: File watching + halfway to sync check
- T+17000ms: Sync service check

Why Errors Stop After Initial Sync

Once hasInitialSyncCompleted() returns true (web/sync_service.ts:154-157):

async hasInitialSyncCompleted(): Promise<boolean> {
  return !!(await this.ds.get(syncInitialFullSyncCompletedKey));
}

Page events re-enabled: enablePageEvents = true in web/client.ts:1389
Local storage populated: Fallback to HTTP requests reduces dramatically
Better coordination: Sync conflicts decrease
Network load normalizes: Fewer concurrent requests

Network Layer Analysis

Timeout Configuration

// lib/spaces/http_space_primitives.ts:9
const defaultFetchTimeout = 30000; // 30 seconds

// web/boot.ts:14
signal: AbortSignal.timeout(1000), // 1 second for config fetch

Browser Connection Limits

Modern browsers limit concurrent connections per domain (typically 6-8). During initial indexing:

Multiple sync intervals fire simultaneously
Each attempts HTTP requests
Connection pool exhaustion triggers timeouts
Timeouts interpreted as “offline” state

Service Worker Interference

Service worker registration during boot can interfere:

// web/boot.ts:52-86
if (navigator.serviceWorker) {
  navigator.serviceWorker.register(workerURL, { type: "module" })
}

Specific Improvements Needed

1. Implement Exponential Backoff

Current: Immediate retry on failure
Proposed:

// lib/spaces/http_space_primitives.ts
class HttpSpacePrimitives {
  private retryDelays = [1000, 2000, 4000, 8000]; // Progressive delays

  async authenticatedFetchWithBackoff(url: string, options: RequestInit, attempt = 0): Promise<Response> {
    try {
      return await this.authenticatedFetch(url, options);
    } catch (e: any) {
      if (attempt < this.retryDelays.length && this.isRetryableError(e)) {
        await sleep(this.retryDelays[attempt]);
        return this.authenticatedFetchWithBackoff(url, options, attempt + 1);
      }
      throw e;
    }
  }
}

2. Add Circuit Breaker Pattern

class SyncCircuitBreaker {
  private failures = 0;
  private lastFailure = 0;
  private readonly maxFailures = 5;
  private readonly resetTimeout = 30000;

  async execute<T>(operation: () => Promise<T>): Promise<T> {
    if (this.isOpen()) {
      throw new Error("Circuit breaker open");
    }

    try {
      const result = await operation();
      this.onSuccess();
      return result;
    } catch (e) {
      this.onFailure();
      throw e;
    }
  }
}

3. Coordinate Sync Operations

Current: Independent intervals with no coordination
Proposed: Centralized sync coordinator

// web/sync_coordinator.ts
class SyncCoordinator {
  private pendingOperations = new Map<string, Promise<any>>();

  async scheduleOperation(key: string, operation: () => Promise<any>): Promise<any> {
    if (this.pendingOperations.has(key)) {
      return this.pendingOperations.get(key);
    }

    const promise = operation();
    this.pendingOperations.set(key, promise);

    try {
      return await promise;
    } finally {
      this.pendingOperations.delete(key);
    }
  }
}

4. Improve Error Differentiation

Current: All network errors become “Offline”
Proposed: Distinguish error types

// lib/spaces/http_space_primitives.ts
class NetworkError extends Error {
  constructor(message: string, public readonly type: 'timeout' | 'connection' | 'server' | 'offline') {
    super(message);
  }
}

// Better error classification
if (e.name === 'TimeoutError') {
  throw new NetworkError('Request timeout', 'timeout');
} else if (e.message.includes('fetch')) {
  throw new NetworkError('Connection failed', 'connection');
}

5. Add Jitter to Intervals

Current: Fixed intervals cause synchronized load
Proposed: Add randomization

// Add 10% jitter to prevent synchronized requests
const jitteredInterval = baseInterval * (0.9 + Math.random() * 0.2);
setInterval(operation, jitteredInterval);

6. Configurable Sync Behavior

Current: Hardcoded intervals
Proposed: Configuration options

interface SyncConfig {
  spaceSyncInterval: number;      // Default: 17000
  pageSyncInterval: number;       // Default: 6000
  fileWatchInterval: number;      // Default: 5000
  maxRetries: number;             // Default: 3
  retryBackoffBase: number;       // Default: 1000
  enableAggressiveSync: boolean;  // Default: false during initial sync
}

Implementation Priority

Phase 1: Immediate Fixes (Low Risk)

Add jitter to intervals - Prevents synchronized collisions
Increase timeout during initial sync - Reduces false “offline” states
Better error logging - Distinguish between error types

Phase 2: Coordination Improvements (Medium Risk)

Implement sync coordinator - Prevents duplicate operations
Add exponential backoff - Reduces server load during failures
Circuit breaker for sync operations - Prevents cascading failures

Phase 3: Architectural Changes (High Risk)

Redesign sync intervals - Use a single coordinator with smart scheduling
Implement adaptive timeouts - Adjust based on network conditions
Add sync operation priorities - Critical operations first

Testing Strategy

Unit Tests

Test sync interval collision scenarios
Verify exponential backoff behavior
Test circuit breaker state transitions

Integration Tests

Simulate network failures during initial sync
Test with slow server response times
Verify fallback behavior under load

Performance Tests

Measure sync performance with/without improvements
Test concurrent user scenarios
Validate memory usage during long sync operations

Monitoring and Metrics

Key Metrics to Track

Sync error rate during initial indexing
Time to initial sync completion
Network request concurrency levels
Circuit breaker activation frequency
User-perceived performance during startup

Alerting Thresholds

Sync error rate > 10% during initial sync
Initial sync time > 2 minutes
Concurrent requests > browser connection limit

Conclusion

The “offline” errors during SilverBullet indexing are caused by a well-understood architectural pattern where multiple independent sync mechanisms create network congestion during the critical initial sync period. The solution requires coordinated sync operations, better error handling, and adaptive timeout strategies.

The proposed improvements will:

Eliminate false “offline” errors during normal operation
Improve initial sync performance by reducing conflicts
Provide better user feedback with accurate error reporting
Scale better under high load conditions

Implementation Bug Analysis

Following the architectural analysis, a detailed code review revealed 10 critical implementation bugs that directly contribute to the sync/indexing “offline” errors. These bugs confirm the architectural findings and provide specific targets for fixes.

Critical Race Conditions & State Management

Bug #1: Race Condition in Sync State Detection

File: web/sync_service.ts:131-152
Issue: isSyncing() method has race conditions and potential null dereference

async isSyncing(): Promise<boolean> {
  // ... code ...
  const startTime = await this.ds.get(syncStartTimeKey); // RACE CONDITION
  if (!startTime) {
    return false;
  }
  const lastActivity = await this.ds.get(syncLastActivityKey)!; // NULL DEREFERENCE
  // ... more race conditions in cleanup logic
}

Impact: Corrupt sync state can cause infinite loops and false “offline” detection

Bug #2: Resource Leak in Sync Scheduling

File: web/sync_service.ts:217-227
Issue: Exception in syncFile() leaves files permanently in scheduled set

async scheduleFileSync(path: string): Promise<void> {
  this.filesScheduledForSync.add(path);
  await this.noOngoingSync(7000);
  await this.syncFile(path); // IF THIS THROWS, path never removed
  this.filesScheduledForSync.delete(path); // Never reached on exception
}

Impact: Growing scheduled set prevents future sync operations

Bug #3: Infinite Loop in Sync Wait

File: web/sync_service.ts:205-215
Issue: noOngoingSync() can loop infinitely if sync state is corrupted

while (await this.isSyncing()) { // INFINITE LOOP POTENTIAL
  await sleep(321); // Fixed interval, no jitter
  // Timeout check, but isSyncing() may always return true
}

Impact: Hangs sync operations indefinitely

Error Handling & Propagation Issues

Bug #4: Swallowed Sync Errors

File: web/client.ts:272-280
Issue: Page sync interval catches and discards all errors

setInterval(() => {
  try {
    this.syncService.syncFile(this.currentPath(true)).catch((e: any) => {
      console.error("Interval sync error", e); // SWALLOWED ERROR
    });
  } catch (e: any) {
    console.error("Interval sync error", e);
  }
}, pageSyncInterval);

Impact: Real network errors masked, no circuit breaker or backoff

Bug #5: Fragile Error Classification

File: lib/spaces/http_space_primitives.ts:106-116
Issue: String matching for error detection loses context

const errorMessage = e.message.toLowerCase();
if (errorMessage.includes("fetch") || errorMessage.includes("load failed")) {
  throw new Error("Offline"); // LOSES ORIGINAL ERROR CONTEXT
}

Impact: All network errors become generic “Offline” messages

Bug #6: Cascading Fallback Retry Loops

File: lib/spaces/fallback_space_primitives.ts:48-72
Issue: No exponential backoff creates retry storms

try {
  const meta = await this.fallback.getFileMeta(name);
  return { ...meta, noSync: true };
} catch (fallbackError: any) {
  console.error("Error during getFileMeta fallback", fallbackError.message);
  throw e; // POTENTIAL INFINITE RETRY LOOP
}

Impact: Failed fallbacks trigger immediate retries, overwhelming network

Resource Leaks & Memory Issues

Bug #7: AbortSignal Timeout Leak

File: lib/spaces/http_space_primitives.ts:39
Issue: Timeout timers accumulate without cleanup

options.signal = AbortSignal.timeout(fetchTimeout); // RESOURCE LEAK
const result = await fetch(url, options);
// No cleanup of timeout timer in catch blocks

Impact: Memory leak from accumulated timeout timers

Bug #8: Unbounded File Set Growth

File: web/client.ts:378-396
Issue: allKnownFiles Set grows without bounds

this.eventHook.addLocalListener("file:changed", (fileName: string) => {
  this.clientSystem.allKnownFiles.add(fileName); // MEMORY LEAK
});
// No size limits or cleanup mechanisms

Impact: Memory consumption grows indefinitely

File Watching & Iteration Issues

Bug #9: File Watch Error Cascading

File: web/space.ts:164-174
Issue: Single file error fails entire watch cycle

for (const fileName of this.watchedFiles) {
  await this.spacePrimitives.getFileMeta(fileName); // NO ERROR HANDLING
}

Impact: One bad file stops all file watching

Bug #10: Set Modification During Iteration

File: web/space.ts:169-171
Issue: watchedFiles Set may be modified during iteration

for (const fileName of this.watchedFiles) {
  await this.spacePrimitives.getFileMeta(fileName); // May trigger events
}

Impact: Undefined behavior if Set is modified during iteration

Bug Impact Analysis

How These Bugs Create the “Perfect Storm”

Race Conditions (Bugs #1, #3) corrupt sync state during high concurrency
Resource Leaks (Bugs #2, #7, #8) degrade performance over time
Poor Error Handling (Bugs #4, #5, #6) masks real issues as “offline”
Cascading Failures (Bugs #6, #9) amplify individual problems

Timing Correlation with Sync Intervals

The bugs directly correlate with the interval collision analysis:

5-6 second intervals trigger file watching bugs (#9, #10)
8.5-17 second intervals hit race conditions (#1, #2, #3)
Continuous retry loops from error handling bugs (#4, #5, #6)
Resource accumulation compounds over time (#7, #8)

Immediate Fix Requirements

Phase 0: Critical Bug Fixes (Immediate - Zero Risk)

Add try/finally to sync scheduling (Bug #2)
Fix null dereference in isSyncing() (Bug #1)
Add individual error handling in file watching (Bug #9)
Copy Set before iteration (Bug #10)

Phase 1: Error Handling Improvements (Low Risk)

Implement proper error propagation (Bug #4)
Preserve original error context (Bug #5)
Add exponential backoff to fallbacks (Bug #6)
Add Set size limits (Bug #8)

Phase 2: Resource Management (Medium Risk)

Implement AbortController cleanup (Bug #7)
Add sync state locking (Bug #1)
Implement sync operation timeout (Bug #3)

Code Quality Impact

These implementation bugs explain why the architectural improvements outlined earlier are critical:

Circuit breaker pattern → Prevents bugs #4, #5, #6 from cascading
Sync coordinator → Eliminates race conditions from bugs #1, #2, #3
Resource cleanup → Addresses memory leaks in bugs #7, #8
Better error handling → Fixes root causes in bugs #4, #5, #6