Help with widgets that relies on web calls

General
1

Howdy! I've been working on a little dashboard that helps me use RSS in what I believe is a healthier way. It treats incoming items as "ephemeral", that can be either read, saved, or automatically dismissed if ignored for a couple days. So the dashboard is less like a todo list and more a "I've got some time and want something to read, what's available right now?" where nothing shown is something like stale/outdated news. That's why it also has support for showing things like active livestreams. This is what it looks like atm:

image
image1094×1097 68.9 KB

The issue for me is that the rss feeds (the forums, videos, and articles rows plus a few that aren't shown) are all a single widget that makes a request to my miniflux db and then does some filtering, random sorting, and finally rendering the buttons themselves. It can take a couple seconds, which can become annoying when I, say, click a link to a short piece of text and quickly return to the page and have to wait for it to load again.

I'm thinking perhaps I can cache the response and have it render based on the old data immediately, and update the info on page load (using an event hook) or when the refresh button is pressed. The info update process would fetch, do the post-processing, store it in the cache or w/e, and finally tell the editor to reload the widgets.

For the cache, I'd probably do it via frontmatter or something, although that feels like a lot of data to store that way and I don't know how to get around the issue with patchFrontmatter not working with arrays well (or at least it didn't when I was working on my journaling library).

Or maybe to avoid the page updating right when I'm about to click on something, I could somehow show a banner or something that the cache has changed, with a button to reload the widgets? Perhaps with some way to determine if the data actually changed. And I suspect there's no way to, like, do this sync as a cron job so it would have to be via page load or manual refresh.

Is that the right way to go about this? Does this make sense?

2

I'm afraid I'm not able to help you but I'm interested to learn more on how you read rss feeds and create these cards - could you share?

3

What you could consider is writing a (Lua) script that fetches all the data you need from Miniflux or some other "slow" data source, and writes it to a page with a data block — simply rewrites the entire page every time, for instance:

```#feed-item
id: 1345
url: https://something.com
title: bladiedah
---
id: 13452
url: https://something2.com
title: bladiedah
---
id: 134542
url: https://something.3com
title: bladiedah
```

SilverBullet will index this "locally" and allow you to query this via for instance index.tag "feed-item" fairly quickly (no remote fetches). This page then serves as a cache.

You could run this script either via a cron job (e.g. by listening to the cron:secondPassed event, or indeed by pressing a button on demand.

Would something like that make sense?

4

Oh, I totally forgot about the secondPassed event. Yeah I think that's feasible, tysm.

I wouldn't want to have the cron job refresh the widgets directly though, in case I'm about to click something. Do you think it's feasible to add or unhide an element saying "Items updated. Click here to refresh"? Obviously without that element itself depending on rebuildEditorState being called.

5

I'll make a post/library about it once it's a bit more polished and generalized, but right now I have a couple utils for making the button and the rows and shuffling items and stuff.

The button code is currently this:

function campfire.create_button(href, onclick, children, style)
  return dom.a {
    href = href,
    target = "_blank",
    onclick = onclick,
    style = table.concat({
      "display: flex",
      "flex-direction: column",
      "gap: 4px",
      "padding: 10px !important",
      "margin-right: 10px",
      "min-width: 220px",
      "text-decoration: none",
      "color: inherit",
      "border: 1px solid #ddd",
      "border-radius: 8px",
      "background: var(--subtle-background-color)",
      table.unpack(style or {})
    }, ";"),
    table.unpack(children)
  }
end

The alert buttons pass in the red border via the style parameter.

The code for each source of data is currently a bit different, but for the livestreams it looks like this:

function campfire.get_livestreams()
  local streamsRes = twitch.get_streams({
    -- (ommitted)
  })
  if streamsRes.data == nil then
    -- We need to get a new oauth token
    local children = {
      dom.div {
        style = table.concat({
          "font-size: 12px",
          "font-weight: 500"
        }, ";"),
        "Get new oauth token for twitch"
      }
    }
    local href = twitch.get_oauth_url()
    local button = campfire.create_button(href, nil, children, {
      "border-color: red"
    })
    return widget.html(dom.div {
      style = "margin: -1em 0",
      button
    })
  end
  local items = streamsRes.data
  if #items == 0 then
    return widget.html(dom.i {
      "No livestreams active!"
    })
  end
  local row = {}
  local streams = { }
  for k, v in pairs(items) do streams[k] = v end
  streams = shuffle(streams)
  for _, stream in ipairs(streams) do
    local children = {}
    local href = "https://www.twitch.tv/" .. stream.user_login

    local icon = stream.thumbnail_url
    icon = icon:gsub("{width}", "14")
    icon = icon:gsub("{height}", "14")
    local topRow = {
      dom.img {
        src = icon,
        style = "width: 14px; height: 14px;"
      },
      stream.user_name
    }
    table.insert(children, dom.div {
      style = table.concat({
        "display: flex",
        "align-items: center",
        "gap: 6px",
        "font-size: 12px",
        "opacity: 0.7"
      }, ";"),
      table.unpack(topRow)
    })

    table.insert(children, dom.div {
      style = table.concat({
        "font-size: 12px",
        "font-weight: 500"
      }, ";"),
      stream.title
    })

    table.insert(children, dom.div {
      style = table.concat({
        "font-size: 12px",
        "opacity: 0.7;"
      }, ";"),
      stream.game_name
    })

    local button = campfire.create_button(href, nil, children)
    table.insert(row, button)
  end
  return widget.html(dom.div {
    style = table.concat({
      "display: flex",
      "gap: .5em",
      "margin-top: -1em",
      "margin-bottom: -1em",
      "overflow-x: auto"
    }, ";"),
    table.unpack(row)
  })
end

RSS is a bit more complicated because I merge items from the same forum thread together and have it open the thread directly instead of miniflux (and use the onclick function to mark each post in miniflux as read).

For actually talking to twitch, miniflux, etc. I write files like Library/Infrastructure/Miniflux that look something like this (with the vars defined in a separate file):

function filter(arr, func)
    local result = {}
    for i, v in ipairs(arr) do
        if func(v, i) then
            table.insert(result, v)
        end
    end
    return result
end

function miniflux.get_unread()
  local feeds = miniflux.fetch("/v1/feeds/counters").body.unreads
  local unread = 0
  for i, v in pairs(feeds) do
    unread = unread + v
  end
  return unread
end

function miniflux.mark_read(ids)
  miniflux.fetch("/v1/entries", {
    entry_ids = ids,
    status = "read"
  }, "PUT")
end

function miniflux.get_recent_entries(days)
  local timestamp = os.time()
  timestamp = timestamp - days * 24 * 60 * 60
  return miniflux.fetch("/v1/entries?status=unread&after=" .. timestamp).body.entries
end

function miniflux.get_icon(id)
  return miniflux.fetch("/v1/icons/" .. id).body.data
end

function miniflux.get_failing_feeds()
  local feeds =  miniflux.fetch("/v1/feeds").body
  return filter(feeds, function(e)
    return e.parsing_error_count > 0
  end)
end

function miniflux.fetch(url, body, method)
  local baseUrl = config.get("miniflux.baseUrl")
  if not baseUrl then
    error("miniflux.baseUrl config not set")
  end
  local token = config.get("miniflux.token")
  if not token then
    error("miniflux.token config not set")
  end
  return net.proxyFetch(baseUrl .. url, {
    method = method or "GET",
    headers = {
      ["X-Auth-Token"] = token
    },
    body = body
  })
end