LanguageTool Grammar Checking

Plugs and Libraries
1

I've been working on a (very much WIP) LanguageTool based grammar checker library. So far I've pointed it to my self-hosted LanguageTool server and it works pretty well. I hope you guys like it!

Also, any suggestions on how to render the feedback? Right now, I just insert the feedback as markdown at the user's cursor position, which isn't ideal. Is there a way I can render lines under the text similar to grammarly?

Code (click to open) 



config.define("grammarcheck", {
  description = "Configure Grammar Checker Library",
  type = "object",
  properties = {
    url = {
      type = "string",
      default = "http://127.0.0.1:8081/v2/check",
      description = "API Endpoint for languagetool",
      ui = { category = "Grammar Checker", label = "API Endpoint" }
    },
    lang = {
      type = "string",
      default = "en-US",
      description = "Language code to use for LanguageTool. (auto will autodetect language, your mileage may vary)",
      ui = { category = "Grammar Checker", label = "Language" }
    },
    apiKey = {
      type = "string",
      description = "API Key for premium LanguageTool features",
      ui = { category = "Grammar Checker", label = "API Key" }
    }
  },
  additionalProperties = false
})

local function urlencode(str)
   if str then
      str = str:gsub("\n", "\r\n")
      str = str:gsub("([^%w %-%_%.%~])", function(c)
         return string.format("%%%02X", string.byte(c))
      end)
      str = str:gsub(" ", "+") -- Standard for application/x-www-form-urlencoded
   end
   return str
end


slashCommand.define {
  name = "grammar",
  description = "Check all grammar on this page, inserting feedback at cursor",
  run = function()
    local languagetool_url = config.get("grammarcheck.url", "http://127.0.0.1:8081/v2/check")
    local lang = config.get("grammarcheck.lang", "en-US")

    local text = urlencode(editor.getText())

    local reqbody = string.format("language=%s&text=%s", lang, text)
    if config.get("grammarcheck.apiKey", nil) then
      body = body .. "&apiKey=" .. config.get("grammarcheck.apiKey", nil)
    end
    
    local resp = net.proxyFetch(languagetool_url, {
      method = "POST",
      headers = {
        ["accept"] = "application/json",
        ["Content-type"] = "application/x-www-form-urlencoded"
      },
      body = reqbody
    })

    if resp.ok and resp.status ~= 400 then
      local body = resp.body
      -- TODO: add highlighting to grammar errors in place
      -- TODO: add ability to go through errors and approve changes.
      local confirm = editor.confirm("Do you want to insert feedback?")
      if confirm then
        editor.insertAtCursor(string.format("### %d grammar errors\n\n", #body.matches))
        for _, match in ipairs(body.matches) do
          local blockquote = match.context.text
          
          -- Insert the highlight
          blockquote = blockquote:sub(1, match.context.offset) .. "==" .. blockquote:sub(match.context.offset + 1, match.context.offset + match.context.length) .. "==" .. blockquote:sub(match.context.offset + match.context.length + 1)

          -- Reformat to block quote
          blockquote = blockquote:gsub("\r", "")
          blockquote = blockquote:gsub("\n", "\n>  ")
          local msg = match.message:gsub("\r", "")
          msg = msg:gsub("\n", " ")
          local feedback = "> [!WARNING] " .. msg ..  "\n>  " .. blockquote .. "\n\n"
          editor.insertAtCursor(feedback)
        end
      end
    else
      editor.alert(string.format("Language tool call to url %s failed with code %d", languagetool_url, resp.status))
    end
  end
}
2

Nice! For those unfamiliar with LanguageTool (like me), this is it right? https://dev.languagetool.org/

3

Yep, that’s it! You can either self host the server or use their software as a service to provide the grammar check API

4

Does this make the connection server-side? So if I'm hosting it on my home server, but access SilverBullet from abroad, it'll resolve the localhost uri correctly?

Also if you self host are you still limited on character limit like the saas free tier?

5

I used net.proxyFetch to do the API call, so I believe that fetches it server side. There shouldn't be any character limits if you self host, LanguageTool is an open source project.

6

I've rewritten the plugin to take advantage of the linter to show underline messages. Keep in mind this is a little experimental, and some Unicode characters will break the annotations.

Code 

config.defineCategory {
  name = "Grammar Checker",
  description = "Settings for Grammar Checker Library"
}

config.define("grammarcheck", {
  description = "Configure Grammar Checker Library",
  type = "object",
  properties = {
    url = {
      type = "string",
      default = "http://127.0.0.1:8081/v2/check",
      description = "API Endpoint for languagetool",
      ui = { category = "Grammar Checker", label = "API Endpoint" }
    },
    lang = {
      type = "string",
      default = "en-US",
      description = "Language code to use for LanguageTool. (auto will autodetect language, your mileage may vary)",
      ui = { category = "Grammar Checker", label = "Language" }
    },
    apiKey = {
      type = "string",
      description = "API Key for premium LanguageTool features",
      ui = { category = "Grammar Checker", label = "API Key" }
    },
    enabled = {
      type = "boolean",
      default = true,
      description = "Enable the spell checker",
      ui = { category = "Grammar Checker", label = "Enabled" }
    },
    tag = {
      type = "boolean",
      default = true,
      description = "Only run on pages with grammarcheck: true in meta (recommended)",
      ui = { category = "Grammar Checker", label = "Only run on tagged pages" }
    }
  },
  additionalProperties = false
})

local function urlencode(str)
   if str then
      str = str:gsub("([^%w %-%_%.%~])", function(c)
         return string.format("%%%02X", string.byte(c))
      end)
      str = str:gsub(" ", "+") -- Standard for application/x-www-form-urlencoded
   end
   return str
end

local function getMetaKey(key)
  local meta = editor.getCurrentPageMeta()
  if meta ~= nil then
    return meta[key]
  end
  return nil
end

if config.get("grammarcheck.enabled", false) then
event.listen {
  name = "editor:lint",
  run = function()
    if config.get("tag", true) == false or getMetaKey("grammarcheck") == true then
    local languagetool_url = config.get("grammarcheck.url", "http://127.0.0.1:8081/v2/check")
    local lang = config.get("grammarcheck.lang", "en-US")

    -- Also check page override
    local lang_override = getMetaKey("grammarcheck-lang")
    if lang_override ~= nil then
      lang = lang_override
    end

    local text = editor.getText()
    -- Replace various unicode symbols
    -- TODO: find more characters that need replacing
    -- TODO: ignore feedback from frontmatter or in code blocks
    
    -- Single quotes
    text = string.gsub(text, "[‘’]", "'")
    -- Double quotes
    text = string.gsub(text, "[”“]", "\"")
    -- Dashes
    text = string.gsub(text, "[—–]", "-")
    local textEncoded = urlencode(text)

    local reqbody = string.format("language=%s&text=%s", lang, textEncoded)
    if config.get("grammarcheck.apiKey", nil) then
      body = body .. "&apiKey=" .. config.get("grammarcheck.apiKey", nil)
    end
    
    local resp = net.proxyFetch(languagetool_url, {
      method = "POST",
      headers = {
        ["accept"] = "application/json",
        ["Content-type"] = "application/x-www-form-urlencoded"
      },
      body = reqbody
    })

    if resp.ok and resp.status ~= 400 then
      local body = resp.body
      local result = {}

      -- TODO: add ability to go through errors and approve changes.
      for _, match in ipairs(body.matches) do
        local msg = match.message
        local start = match.offset
        local len = match.length
        
        table.insert(result, {
          from = start,
          to = start + len,
          message = msg,
          severity = "info"
        })
      end

      return result
    else
      editor.flashNotification(string.format("Language tool call to url %s failed with code %d", languagetool_url, resp.status))
    end
    end
  end
}
end
7

What causes the editor:lint event? I don't see it in the docs here and nothing in this code sends it.

Also, if one has to manually call the lint event, I don't see why have config settings to disable the script globally nor on untagged pages.

Maybe it should be tied to editor:pageModified, with some debouncing? Or maybe on one of the saving related events, since those have debouncing already?

8

It's under the interactions section of the page. It seems to be called every time the page is updated. Here's a screenshot of it working (I changed the CSS a bit to make the underlines look more similar to grammarly):

image
image2870×1402 288 KB

9

Oh, thanks! I must've missed it under interactions somehow. It hadn't been working for me, and since I hadn't seen that event under editor, it felt plausible that the event being missing was why it wasn't working...

Turned out the actual issue for me was that I'd set grammarcheck.tag = false in CONFIG, but the code looks for tag instead (not under grammarcheck).

10

Now I just wish there was a way to add spellcheck="false" to all the inputs, so the browser doesn't add it's own squigglies!

11

That's my bad about the tag config option. I'm going to clean the code up and push it to a git repo in the next few days

12

You have a TODO in your code about stripping code blocks and frontmatter before passing the text to languagetool. The code to do that is:

    -- Code blocks
    text = string.gsub(text, "```.-```", "")
    text = string.gsub(text, "`[^`]-`", "")
    -- Frontmatter
    local fm = index.extractFrontmatter(text, {
      removeFrontMatterSection = true,
      removeTags = false
    })
    text = fm.text

edit: Ah, this makes the character ranges inaccurate. I see why this wasn't already implemented haha

13

Exactly. In a perfect world this would work. I think the method might be to create a table of all ranges that contain stuff we don't want to check (frontmatter, code blocks, latex blocks, etc) and then discard feedback that falls within these ranges. This might cause other issues though.

14

I have pages with a lot of code that made languagetool use a lot of cpu. Id rather strip it and write a function to map languagetool character indices to full text indices