DRIVERS-2477: use w:'majority' in consecutive resume change stream tests

[baileympearson]

https://jira.mongodb.org/browse/DRIVERS-2477

POC in Node: mongodb/node-mongodb-native#3451

The change stream consecutive resume test (here) fails intermittently in Node because the test inserts documents with write the default write concern instead of majority.

Change events are not reported by the server until they are majority committed. The test inserts multiple documents and attempts to iterate the change stream, expecting change events. If the documents haven't been committed by the time the test attempts to iterate the stream, no change will be reported and the test will fail.

This only impacts the consecutive resume test and does not impact users because the change event will eventually be reported. This test specifically sets up fail commands on getMores, so that the getMore fail and the subsequent aggregate call succeeds and returns a change event in the firstBatch. The race condition is present only because we expect the aggregate to immediately return documents in the firstBatch.

Please complete the following before merging:

• Update changelog.
• Make sure there are generated JSON files from the YAML test files.
• Test changes in at least one language driver.
• Test these changes against all server versions and topologies (including standalone, replica set, sharded clusters, and serverless).

[benjirewis]

Your explanation makes sense, but I'm surprised other tests are not failing in Node (sounds like this is somehow not a problem in other drivers). I may have missed something in your description, but do you know why the lack of propagation to multiple nodes' oplogs is only a problem in this test and not in the others for Node?

[benjirewis]

Can we leave a comment above this writeConcern to explain that this is needed to make sure the insert propagates and a change stream event is present on all oplogs?

[baileympearson]

Yup, will do!

[benjirewis]

Same thing here.

[baileympearson]

Your explanation makes sense, but I'm surprised other tests are not failing in Node (sounds like this is somehow not a problem in other drivers). I may have missed something in your description, but do you know why the lack of propagation to multiple nodes' oplogs is only a problem in this test and not in the others for Node?

@benjirewis I do know why this test fails actually. It's because of the failCommand on getMore set to 2 times (> 1 is what matters).

Here's what we expect to happen when we iterateUntilDocumentOrError (and what happens with w: majority):

• we try a get more and it fails
• we attempt to resume
• the aggregate call on resume comes back with a single document (batch size 1 set on change stream)
• the change stream yields the document

Here's what actually happens

• we try a get more and it fails
• we attempt to resume
• the aggregate call completes before the writes have been committed, so no change is returned in the aggregate call. but the change stream does resume successfully
• we try a get more again but it fails again
• change streams only attempt to resume once, so the error is returned and the test fails

You can also get the test to pass by setting the fail command to only fail once, but then we're not testing consecutive resumes anymore 🙂

I misread your comment! I don't know why this only fails in Node but I can look closely at it

[baileympearson]

@benjirewis

The root cause of the issue was a bug Node introduced last spring into our change streams (this test did continue to pass for a long while though, it only started failing recently). You can see the fix PR here - mongodb/node-mongodb-native#3453.

Should we close this PR and the associated drivers ticket?

[benjirewis]

Yeah if it was just a bug in Node's change streams code, then closing this PR and the associated drivers ticket SGTM.