Logged Conversation (all times UTC)
[22:26] <bibbit> graydon: What is the technical story on the network split?
[23:24] <graydon> bibbit: as far as I can tell it was pretty much behaving-as-designed; the existing protocol seems willing to continue closing ledgers even in the presence of a drop in quorum size (which can happen any time they get out of sync), with the assumption that it'll switch forks (and abandon the ledgers it closed in the meantime) when it returns to synchronization.
[23:26] <bibbit> How many non-SDF validators were in the SDF UNL?
[23:26] <graydon> bibbit: small, transient divergence in consensus seems to happen quite often. why this one lasted so long, we don't know. there are a lot of time and load-based factors that seem to govern its behavior, but fundamentally I think the blog post is correct: that the algorithm-as-coded favours availability over consistency.
[23:27] <graydon> at this point, the SDF validators were only trying to maintain consensus among themselves (no non-SDF validators on their UNL).
[23:29] <bibbit> Isn't the minority side of the split supposed to detect that it has fragmented and halt operations until it rejoins?
[23:30] <bibbit> If small frequent divergences are happening, could it be more a matter of the detection not working than a more fundamental flaw?
[23:36] <graydon> perhaps, but from what we can tell of the algorithm -- it's quite complex -- it has several different criteria for deciding to close a ledger and attempt to make progress anyways
[23:39] <graydon> (some of which are actually feedback cycles, so that if it makes a bad decision once due to rule X, it'll keep making that bad decision repeatedly due to rule Y observing what it did most recently, etc.)
[23:41] <bibbit> Did observation of any of the previous small divergences show any commonality in departure points, or were they mostly unique conditions?
[23:43] <graydon> they usually seem to get started because of timing. someone times out waiting for some other peer that got slow due to transient load.
[23:44] <graydon> we've put them on quite fast machines with high I/O and are continuing to investigate upping the hardware, but it's already well past the point of reasonable provision for the (honestly, modest) level of transaction traffic and theoretical working set.
[23:45] <sl01> graydon: now that this is known do you expect it will open up Ripple's network to a targeted attack using this as a vector?
[23:45] <bibbit> Did moving the nodestore into SQL not help with the high IOPs from the way that RocksDB/LevelDB are architected?
[23:46] <graydon> we have not moved the entire nodestore to sql. we added a some extra sql tables to the side to answer specific queries without hammering the nodestore, but we're still writing everything to the nodestore, and the consensus system is still based on reconciling shamaps (which draw on the nodestore)
[23:48] <graydon> sl01: I don't know, it seems like it'd be difficult to engineer any particular behavior out of the system, there are so many nondeterministic pieces. I guess it's possible. I'm not expert at that kind of thing.
[23:49] <graydon> it's also possible RL knows operational tricks we don't to keep it better behaved, or has found-and-fixed some of the worst misbehaviors on their lineage. we don't know, we have looked over their changes and nothing stands out but it's possible.
[23:50] <graydon> we've talked to them a bit about it, and they might suggest something. as far as we know it's still quite similar though. there hasn't been a ton of substantial algorithmic divergence between the two lineages, more just a lot of code movement on their side.
[23:56] <bibbit> Aside from load, have you made any observations with regard to latency. For example, starting up nodes in South Africa, Australia or New Zealand (or just using a network tool to inject latency) to observe the impact of latency effects vs load?
[23:59] <graydon> we haven't done latency simulation. we're going to be doing that on the new codebase, which is being designed more explicitly for testing. the existing system is not especially testable aside from running a bunch of systems in (say) VMs or containers with virtual networks between them and watching them run (in real time, i.e. slowly). we haven't done that since we've been too busy fighting fires due to failures we _didn't_ synthesize :(
About StellarVerse IRC Logger
StellarValue IRC Logger
is part of