Panics with 'index out of bounds' at crates/ide_db/src/line_index.rs:106 in neovim

[dimbleby]

rust-analyzer version: (eg. output of "Rust Analyzer: Show RA Version" command)

rust-analyzer 0add6e9 2021-12-20 stable

rustc version: (eg. output of rustc -V)

rustc 1.57.0

---

I frequently have to restart rust-analyzer because it has panics like this:

[ERROR][2021-12-15 22:59:08] .../vim/lsp/rpc.lua:417>   "rpc">  "rust-analyzer">"stderr">   "Panic context:\n> \nversion: 7d6fcbc0b 2021-12-06 stable\nnotification: textDocument/didChange\n\nthread 'main' panicked at 'index out of bounds: the len is 146 but the index is 146', crates/ide_db/src/line_index.rs:106:9\nstack backtrace:\n"
[ERROR][2021-12-15 22:59:08] .../vim/lsp/rpc.lua:417>   "rpc">  "rust-analyzer">"stderr">   "   0: rust_begin_unwind\n             at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/panicking.rs:517:5\n   1: core::panicking::panic_fmt\n             at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/core/src/panicking.rs:100:14\n  "
[ERROR][2021-12-15 22:59:08] .../vim/lsp/rpc.lua:417>   "rpc">  "rust-analyzer">"stderr">   " 2: core::panicking::panic_bounds_check\n             at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/core/src/panicking.rs:76:5\n   3: rust_analyzer::from_proto::text_range\n   4: rust_analyzer::lsp_utils::apply_document_changes\n   5: core::ops::function::FnOnce::call_once\n   6: rust_analyzer::dispatch::NotificationDispatcher::on\n   7: rust_analyzer::main_loop::<impl rust_analyzer::global_state::GlobalState>::handle_event\n   8: rust_analyzer::main_loop::<impl rust_analyzer::global_state::GlobalState>::run\n   9: rust_analyzer::main_loop::main_loop\n  10: rust_analyzer::try_main\n  11: rust_analyzer::main\nnote: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.\nthread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: \"SendError(..)\"', /home/runner/.cargo/registry/src/github.com.-1ecc6299db9ec823/lsp-server-0.5.2/src/stdio.rs:29:37\nstack backtrace:\n   0: rust_begin_unwind\n             at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/panicking.rs:517:5\n   1: core::panicking::panic_fmt\n             at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/core/src/panicking.rs:100:14\n   2: core::result::unwrap_failed\n             at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/core/src/result.rs:1616:5\nnote: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.\n"

I can reliably provoke this in neovim as follows:

• create a file with a mistake in it, eg like this (note the extra closing bracket)

  fn hello() {
     println!("Hello"));
  }

• run a bit of vimscript that tries to format this, spots the failure, and immediately undoes that format. Here's a simplified version of my formatting code that is rust-specific and sufficient to reproduce the problem:

function! format#FormatFile() abort
  let tmpfile = tempname()
  let shellredir_save = &shellredir
  let &shellredir = '>%s 2>'.tmpfile
  silent execute '%!rustfmt --edition 2018 --color never --emit stdout'
  let &shellredir = shellredir_save
  if v:shell_error != 0
    silent undo
  endif
  call delete(tmpfile)
endfunction

Similar to #10138, but reporting separately because (i) it looks like a different stack and (ii) I have a clear repro.

[Veykril]

Panic workaround implemented in #11182

[matklad]

Let's reopen this, as it sounds like something serious which we should fix!

To give a broad context here, in general rust-analyzer is carefully architectured such that panics do not crash the process or corrupt state. Panics happen fairly often, but they should just display an annoying message, and not require restarting of the process.

So, what we are seeing here, a panic bringing down the whole process, is a very serious issue. The problem isn't the panic per se, but rather the fact that the panic compromizes user experience.

According to the log, the error happens in notification: textDocument/didChange. This is a "write" path where we change the internal state (to account for user's typing), so, indeed, panic here is rather catastrophic -- if we recover here, we'll get state divergence between the client and the server, and that's bad.

I have two hypothesis here:

• There's a bug in VIM LSP client, and it sends us invalid change notifications
• We have some stupid off-by-one error at the boundary (the len is 146 but the index is 146')

@dimbleby Am I correct that you are using neovim's built-in client?

[dimbleby]

Am I correct that you are using neovim's built-in client?

yes

[matklad]

Got the LSP log:

[DEBUG][2022-03-04 16:26:08] .../vim/lsp/rpc.lua:347	"rpc.send"	{  jsonrpc = "2.0",  method = "textDocument/didChange",  params = {    contentChanges = { {        range = {          end = {            character = 0,            line = 3          },          start = {            character = 0,            line = 0          }        },        rangeLength = 46,        text = ""      } },    textDocument = {      uri = "file:///home/matklad/tmp/r/src/main.rs",      version = 4    }  }}
[DEBUG][2022-03-04 16:26:08] .../vim/lsp/rpc.lua:347	"rpc.send"	{  jsonrpc = "2.0",  method = "textDocument/didChange",  params = {    contentChanges = { {        range = {          end = {            character = 0,            line = 0          },          start = {            character = 0,            line = 0          }        },        rangeLength = 0,        text = 'fn main() {\n    println!("Hello, world!"));\n}'      } },    textDocument = {      uri = "file:///home/matklad/tmp/r/src/main.rs",      version = 5    }  }}
[DEBUG][2022-03-04 16:26:08] .../vim/lsp/rpc.lua:347	"rpc.send"	{  jsonrpc = "2.0",  method = "textDocument/didChange",  params = {    contentChanges = { {        range = {          end = {            character = 0,            line = 3          },          start = {            character = 0,            line = 3          }        },        rangeLength = 0,        text = ""      } },    textDocument = {      uri = "file:///home/matklad/tmp/r/src/main.rs",      version = 6    }  }}
[ERROR][2022-03-04 16:26:08] .../vim/lsp/rpc.lua:420	"rpc"	"rust-analyzer"	"stderr"	"Panic context:\n> \nversion: f8329ba98 2022-03-04 dev\nnotification: textDocument/didChange\n\nthread 'main' panicked at 'Invalid line: 3, total lines: 3', crates/ide_db/src/line_index.rs:109:32\nstack backtrace:\n"
[ERROR][2022-03-04 16:26:08] .../vim/lsp/rpc.lua:420	"rpc"	"rust-analyzer"	"stderr"	"   0: rust_begin_unwind\n             at /rustc/0a4f984a87c7ba6c74ec3e78442fec955a419e32/library/std/src/panicking.rs:584:5\n   1: core::panicking::"
[ERROR][2022-03-04 16:26:08] .../vim/lsp/rpc.lua:420	"rpc"	"rust-analyzer"	"stderr"	"panic_fmt\n             at /rustc/0a4f984a87c7ba6c74ec3e78442fec955a419e32/library/core/src/panicking.rs:143:14\n   2: ide_db::line_index::LineIndex::offset\n   3: rust_analyzer::from_proto::text_range\n   4: rust_analyzer::lsp_utils::apply_document_changes\n   5: core::ops::function::FnOnce::call_once\n"
[ERROR][2022-03-04 16:26:08] .../vim/lsp/rpc.lua:420	"rpc"	"rust-analyzer"	"stderr"	"   6: rust_analyzer::dispatch::NotificationDispatcher::on\n   7: rust_analyzer::main_loop::<impl rust_analyzer::global_state::GlobalState>::handle_event\n   8: rust_analyzer::main_loop::<impl rust_analyzer"
[ERROR][2022-03-04 16:26:08] .../vim/lsp/rpc.lua:420	"rpc"	"rust-analyzer"	"stderr"	"::global_state::GlobalState>::run\n   9: rust_analyzer::main_loop::main_loop\n  10: rust_analyzer::run_server\n "
[ERROR][2022-03-04 16:26:08] .../vim/lsp/rpc.lua:420	"rpc"	"rust-analyzer"	"stderr"	" 11: rust_analyzer::main\nnote: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.\n"

I think it's pretty clear what the problem, these two messages are to blame (reformatted and lightly edited for clarity):

{ method = "textDocument/didChange",  params = {
  contentChanges = {{
    range = { end = { character = 0, line = 0 }, start = { character = 0, line = 0 } },
    rangeLength = 0,
    text = 'fn main() {\n    println!("Hello, world!"));\n}'
  }}, textDocument = { uri = "main.rs", version = 5 }}}

{ method = "textDocument/didChange",  params = {
  contentChanges = {{
    range = { end = { character = 0, line = 3 }, start = { character = 0, line = 3 } },
    rangeLength = 0,
    text = ""
  }}, textDocument = { uri = "main.rs", version = 6 }}}

Note how the second message refers to the third line, but, given the text in the previous message, there are only two lines. That is, this seems to be neovim bug. I have a vague feeling that I've seen this already somewhere (specifically, the problem with trailing newline), but I am can't find that now.

[jonas-schievink]

Given that the panic has an implemented workaround, and this is not our bug, removing the Broken Window label

[matklad]

SGTM! Though, we should revert #11182 once the client fix propagates. We shouldn’t be masking client errors. We probably shouldn’t panic on invalid input either, but we at least need to send an LSP error message

…

On Friday, 22 April 2022, Jonas Schievink ***@***.***> wrote: Given that the panic has an implemented workaround, and this is not our bug, removing the *Broken Window* label — Reply to this email directly, view it on GitHub <#11081 (comment)>, or unsubscribe <https://github.com./notifications/unsubscribe-auth/AANB3M7I4FENUQKRF5JY74DVGKP3RANCNFSM5KP3ZG6Q> . You are receiving this because you modified the open/close state.Message ID: ***@***.***>

[federico-terzi]

Hey @matklad ,
First of all, rust-analyzer is beyond amazing and I highly appreciate your work!

Are there any updates on this issue? Unfortunately, this problem makes rust-analyzer unusable for me on the recent VSCode extension versions...

Edit: just noticed the VSCode extension has migrated to another page (it wasn't showing further updates). Removing the previous one and re-installing the new one works as expected, sorry for bothering you :)

[Veykril]

We seem to already be sending an error message back in these cases now (Bad Offset), so I think we can close this