Dynamically changing vim config?

Usually I like j and k to do the usual motion, but sometimes I like to map them to gj and gk. I figured this should work (it does if I run it on startup and not as a function), but the action seems to do nothing, even if I reload the vim config. What am I missing?

function vim_display_mode()
  config.set {
    vim = {
      map = {
        {
          map = "k",
          to = "gk",
        },
        {
          map = "<Up>",
          to = "gk",
        },
        {
          map = "<Down>",
          to = "j",
        },
        {
          map = "<Down>",
          to = "gj",
        }
      }
    }
  }
end

command.define {
  name = "Editor: Vim display-wise mode",
  run = function()
    vim_display_mode()
  end
}