Fix Flake in TestReplicationNodeDiverges

[samliok]

The problem was a race between the lagging node's persistAndBroadcastNotarization and the test. That function writes the notarization to the WAL and then broadcasts it, so laggingNode.WAL.AssertNotarization(0) can return before the broadcast happens.

The test then disconnects the lagging node and calls SetAllNodesMessageFilter, which replaces every node's filter, including the lagging node's, with one that only blocks round 0 votes. TestComm.Broadcast checks whether the sender is disconnected once up front, then reads the filter per recipient. So if the lagging
node was in Broadcast before the call to Disconnect and reached the loop after the filter swap, its notarization passed the new filter (the old sendVotesToOneNode filter dropped notarizations) and reached the other nodes. AdvanceWithoutLeader(0, ...) then failed because those nodes notarized round 0 instead of timing out.

laggingNode.WAL.AssertNotarization(0)
for _, n := range net.Instances {
    if n.E.ID.Equals(laggingNode.E.ID) {
        continue
    }
    require.Equal(t, false, n.WAL.ContainsNotarization(0))
}

// we disconnect lagging node first so that it doesn't send the notarized block to any other nodes
net.Disconnect(laggingNode.E.ID)
net.SetAllNodesMessageFilter(
    // block sending votes from round 0 to ensure all nodes will timeout
    func(msg *common.Message, _, to common.NodeID) bool {
        return msg.VoteMessage == nil || msg.VoteMessage.Vote.Round != 0
    },
)

// This function call ensures all nodes will timeout, and
// receive an empty notarization for round 0(except for lagging).
net.AdvanceWithoutLeader(0, laggingNode.E.ID)

The fix is to set the new filter only on the other nodes, so the lagging node keeps its original filter while disconnected. It gets the new filter right before net.Connect.

Closes #466

[yacovm]

I'm not sure I understand why skipping the lagging node has any difference.

Isn't the bug here that the lagging node sends its vote too fast before we disconnect it?