Logged Conversation (all times UTC)
[03:51] <luvcrypto> love the stellar consensus paper
[03:52] <stellar-slack> @luvcrypto glad you're enjoying it :simple_smile:
[03:53] <luvcrypto> is it possible to store STR in a cold storage?
[03:54] <stellar-slack> luvcrypto: yessir
[03:55] <luvcrypto> so I'm unsure about the protocol. I see STR sold on exchanges
[03:55] <luvcrypto> is that the same thing?
[03:55] <luvcrypto> i.e. where can one play with STR?
[03:56] <stellar-slack> the exchanges are just off-ledger
[03:57] <stellar-slack> no, if you're using an internet connection to trade/send stellars, that'd be considered a hot wallet
[03:57] <luvcrypto> I guess my confusion is - is there a wallet or daemon I can run?
[04:19] <stellar-slack> @graydon: I figured out why we're seeing unreliable / mismatches of hashes. The history upload is running in parallel with the main thread (that's fine & expected), but it uses the app.db.session (soci) to fetch tx and ledger headers from the database... while the same object is being used in the main thread. this makes the soci calls behave kinda randomly. so history may contain garbage or ledger
[04:19] <stellar-slack> off for the night, but unless I somehow triggered this with my local changes, this is what we've been looking for (well, one of the things :simple_smile: ). I am not sure which other objects are being shared like this... Maybe we should create special interfaces that contain the thread safe methods for objects so that it becomes obvious/won't even build to spot this kind of stuff: something like LedgerManagerThreadS
[05:37] <stellar-slack> luvcrypto: http://launch.stellar.org|launch.stellar.org
[10:16] <stellar-slack> any reason to change the address generating mechanism? I found the new address/account_id was not from a hash but public key directly.
[10:24] <stellar-slack> is it secure to exposing public key?
[10:30] <stellar-slack> and the switchover will be a big mess by lack of explicit version field in keypair.
[13:38] <stellar-slack> oh, that's a bit f'd up
[13:50] <stellar-slack> so tx's are basically going to be 24 bytes longer
[13:50] <stellar-slack> fuck
[14:04] <stellar-slack> lab: in the old system the public key is exposed after the first tx
[14:05] <stellar-slack> dzham: txs are smaller since you don't have to include the accountID & the public key now
[14:33] <stellar-slack> alright, I'll have to check it out more thoroughly
[14:34] <stellar-slack> my use case for this was rather specific though
[14:36] <stellar-slack> but I'm still going to need two addresses and a transaction signature
[14:37] <stellar-slack> what's the use case?
[14:38] <stellar-slack> If you need to obfuscate the source of a public key, I believe you would be safe using one ed25519 public key (32 bytes) as the seed for a new ed25519 keypair
[15:16] <Ladang[Galau]> hii
[15:23] <stellar-slack> welcome!
[15:24] <Ladang[Galau]> hii, stellar going moon now?
[15:41] <stellar-slack> @donovan: https://github.com/stellar/go-stellar-base I’d love any feedback you have (other than “add docs”, that’s a known issue :)). I’m a complete go n00b, to put it kindly
[15:41] <stellar-slack> No support for xdr yet, but it has signing, hashing, and base58 support
[15:42] <Ladang[Galau]> ok :)
[15:43] <Ladang[Galau]> https://stellarflip.com/?lang=id
[15:43] <Ladang[Galau]> you are dev game?
[15:43] <Ladang[Galau]> i need script
[15:46] <stellar-slack> Why are you reporting yourself as a bot?
[15:47] <stellar-slack> Ah! that’s how the irc bridge works
[15:48] <stellar-slack> What sort of script do you need?
[15:50] <Ladang[Galau]> stellarflip because I want to make a web like it
[15:54] <stellar-slack> I don’t know of any open source implementations of something like stellar flip. You’ll probably have to write something on your own.
[15:55] <Ladang[Galau]> I've been able to but I was confused for a win / defeat
[15:55] <Ladang[Galau]> okay thanks for the information
[16:32] <stellar-slack> @scott: cool, looks like a good start :simple_smile: Have got a few idiomatic Go comments, do you have any plans to use a code review tool at all (gerrit is well used by the Go team https://go-review.googlesource.com but it's a big commitment...)?
[16:32] <stellar-slack> In some ways it would be nice to get the Stellar types into Go `struct` form as early as possible to get a good idea of how the different types (Account,Balance,etc.) will work well together in practice. Could you perhaps give a quick summary of the pros and cons of doing codegen in Ruby vs Go runtime (de)serialization itself, i.e.
[16:32] <stellar-slack> https://github.com/stellar/xdrgen
[16:32] <stellar-slack> vs
[16:32] <stellar-slack> https://godoc.org/github.com/davecgh/go-xdr/xdr2
[16:32] <stellar-slack> Would help understand the thinking or issues found :simple_smile:
[16:32] <stellar-slack> Also spent a bit of tim going through the TLA+ hyperbook. It's certainly a powerful framework and has opened my eyes up and interesting field of design testing!
[16:34] <stellar-slack> @donovan: I think for now I’d prefer to use github pull requests for code review. I’ll certainly start submitting my own modifications as PRs when the library is fleshed out and another person is interested in reviewing the code (I’m the only person doing any go at the SDF at the moment)
[16:35] <stellar-slack> And yeah, the goal is to finish up the xdrgen changes to get the stellar data structures rendered as go code today.
[16:37] <stellar-slack> Just spotted that you are using go-xdr/xdr2 in the codegen template :simple_smile:
[16:37] <stellar-slack> https://github.com/stellar/xdrgen/blob/master/lib/xdrgen/generators/go/xdr_common.go.erb
[16:38] <stellar-slack> fmt.Errorf is a shortcut to this:
[16:38] <stellar-slack> https://github.com/stellar/xdrgen/blob/master/lib/xdrgen/generators/go/xdr_common.go.erb#L117
[16:38] <stellar-slack> https://golang.org/pkg/fmt/#Errorf
[16:42] <stellar-slack> ah, cool!
[16:43] <stellar-slack> and yeah, I’m using go-xdr in the common file to do the primitive decodes/encodes
[16:48] <stellar-slack> The big change I’m working through right now in regards to xdr is to move away from passing a pointer into the decode methods and instead simply return a value
[16:51] <stellar-slack> Not intended as criticism, but this does seem like a lot of code to write:
[16:51] <stellar-slack> https://github.com/stellar/xdrgen/blob/master/lib/xdrgen/generators/go.rb
[16:51] <stellar-slack> When the xdr library already has a lot of the mappings that get determined at runtime using the reflect package:
[16:51] <stellar-slack> https://godoc.org/github.com/davecgh/go-xdr/xdr2#Marshal
[16:51] <stellar-slack> You mentioned some issues over enums previously, could you expand on what they where?
[16:54] <stellar-slack> It wouldn't be too hard to write a:
[16:54] <stellar-slack> ```func StellarMarshal(w io.Writer, v interface{}) (int, error)```
[16:54] <stellar-slack> and
[16:54] <stellar-slack> ```func StellarUnmarshal(r io.Reader, v interface{}) (int, error)```
[16:54] <stellar-slack> pair of functions that intercept the difficult cases before passing them down into the xdr package functions
[16:58] <stellar-slack> no worries regarding criticism, I’m happy to have it as long as it’s constructive :simple_smile: The big one for me regarding enums (and this may not be an issue to be concerned with) is type safety. DecodeEnum in go-xdr just returns an int32, and I personally would prefer the safety of having a TransactionResultType != LedgerEntryType, for example
[17:00] <stellar-slack> Initially my reluctance for reflection was a) ignorance of go’s reflection capabilities and b) performance concerns.
[17:02] <stellar-slack> monsieurnicolas: interesting to hear; the whole point of the snapshot isolation enforcement in the database is to prevent that from happening...
[17:03] <stellar-slack> backend is fine - this is soci
[17:04] <stellar-slack> on my dev box I ended up with (at best crashes), at worst hash mismatch
[17:05] <stellar-slack> we have java xdr parsing support in xdrgen now. https://github.com/stellar/xdrgen
[17:05] <stellar-slack> Yep, I agree it's lovely to have strongly typed enums :simple_smile: The reflect package has the Kind type for finding out exactly what the "base" type is: http://golang.org/pkg/reflect/#Kind
[17:05] <stellar-slack> I don't think reflection performance is a huge concern in Go and the xdr library is already using it heavily anyway :simple_smile: One of Go's main benchmarks is JSON (de)serialization, so they spend a lot of time making it fast. I guess it really depends on the message volume...
[17:05] <stellar-slack> My experience from Ripple protobuf message handling is that you end up having to declare your own types anyway and copy from the codegen-ed message types, because they are rarely what you want and you get lots of extra methods on the type that confuse the end library user.
[17:05] <stellar-slack> I can have a play with some simple reflection which avoids codegen wholly if you'd like me to put something together for a discussion.
[17:07] <stellar-slack> monsieurnicolas: hm, can you explain a little about what you mean by "this is soci"?
[17:07] <stellar-slack> yeah, totally! I’m happy for any help I can get (one of the best ways for me to learn)
[17:07] <stellar-slack> soci::session is not thread safe
[17:07] <stellar-slack> monsieurnicolas: we're not (supposedly) using it from multiple threads
[17:07] <stellar-slack> well we are :simple_smile:
[17:07] <stellar-slack> monsieurnicolas: we acquire separate sessions for each thread via the pool interface
[17:08] <stellar-slack> guess what happens when you pass down "db"
[17:08] <stellar-slack> I mean I'm not trying to be a jerk about this; just ... there's explicit machinery in db to handle this, so I'm trying to figure out which part's being used wrong / using soci wrong
[17:09] <stellar-slack> monsieurnicolas: which call path using db?
[17:10] <stellar-slack> Has consideration yet been given on how to version the core types?
[17:10] <stellar-slack> https://github.com/stellar/stellar-core/blob/master/src/xdr/Stellar-ledger-entries.x
[17:10] <stellar-slack> Say an extra field is later needed in AccountEntry, how would a client know that they are decoding a type that is the older version without the new field?
[17:10] <stellar-slack> ( https://github.com/stellar/stellar-core/blob/master/src/database/Database.cpp#L141 etc.)
[17:10] <stellar-slack> writeHistoryBlocks
[17:10] <stellar-slack> calls things like copyLedgerHeadersToStream using db
[17:10] <stellar-slack> donovan: we've discussed it in terms of leaving an open-ended enum on the end of the main structures, so that they can be evolved at least backward-compatibly
[17:11] <stellar-slack> versioning: yes, we're going to add an enum everywhere. Didn't want to bloat things up before we settle on "v1"
[17:12] <stellar-slack> monsieurnicolas: copyLedgerHeadersToStream passes the session-to-use (which is a pool-session) along with the call
[17:12] <stellar-slack> @graydon: Does that have implications for the definition of what a canonical transaction is for signing purposes?
[17:13] <stellar-slack> monsieurnicolas: the only 'db.' call in that function is on a medida timer, the data access routines use the passed session
[17:13] <stellar-slack> Oh I see then the problem is just this:
[17:13] <stellar-slack> soci::session& sess(snapSess ? *snapSess : mApp.getDatabase().getSession());
[17:13] <stellar-slack> monsieurnicolas: is that .. not .. right?
[17:13] <stellar-slack> sess ends up being the main session
[17:13] <stellar-slack> if pooling is disabled
[17:14] <stellar-slack> I'm going to create a new session instead
[17:14] <stellar-slack> no wait
[17:14] <stellar-slack> canUsePool() is called in writeNextSnapshot()
[17:15] <stellar-slack> and we only pass this off to a thread in the first place if it's true
[17:15] <stellar-slack> IOW "query runs on a separate thread" === "can use pool"
[17:15] <stellar-slack> that's the design assumption / invariant that needs to be enforced
[17:15] <stellar-slack> if it's not being enforced, I agree it'll fail. but I _thought_ it was being enforced.
[17:15] <stellar-slack> if we can't use the pool we write the snapshot on the main thread
[17:16] <stellar-slack> then I don't know why it's on a different thread
[17:16] <stellar-slack> are you on postgres?
[17:17] <stellar-slack> this is a sqlite test
[17:17] <stellar-slack> hm
[17:17] <stellar-slack> I believe they both support pools so long as they're on-disk, not in-memory
[17:17] <stellar-slack> is your test in memory or on disk?
[17:18] <stellar-slack> disk based
[17:19] <stellar-slack> so canUsePool() will return true
[17:19] <stellar-slack> so you'll get multiple threads
[17:19] <stellar-slack> and sess should be snapSess, from a pool, not main session
[17:20] <stellar-slack> (you can at least double-check that it's in this area by setting canUsePool to always return false, see if your test starts working :simple_smile:
[17:21] <stellar-slack> alternatively we might accidentally make a db call through some other codepath that isn't using the provided session. that'd be bad.
[17:24] <stellar-slack> yeah it's using pooling
[17:24] <stellar-slack> aha
[17:24] <stellar-slack> transactions!
[17:24] <stellar-slack> savetransactionhelper
[17:24] <stellar-slack> in TransactionFrame
[17:24] <stellar-slack> takes db and calls on it directly, not the passed-through session
[17:24] <stellar-slack> (serves me right for not documenting this invariant strongly enough)
[17:24] <stellar-slack> but this is only used in catchup?
[17:24] <stellar-slack> nope
[17:24] <stellar-slack> used in publishing
[17:25] <stellar-slack> copyTransactionsToStream calls it
[17:25] <stellar-slack> OK, let me try that :simple_smile:
[17:25] <stellar-slack> I wonder how we can trap this sort of thing in the future...
[17:26] <stellar-slack> back when I added the prepared statement cache I thought about augmenting the soci interface with some additional wrappers up at stellar-core level, to mediate access a little more
[17:26] <stellar-slack> specifically thread-checking, prepared-statement caching and metric-counting
[17:26] <stellar-slack> maybe I should get on that
[17:29] <stellar-slack> yeah we should be passing around some equivalent of session instead of db
[17:33] <stellar-slack> that way we sweep the code and only use "Database" to grab sessions from the pool (or create a new one)
[17:36] <stellar-slack> we can at least assert to getSession() and getPreparedStatement() that we're on the main thread
[17:36] <stellar-slack> that'll trap the most immediate case
[17:48] <stellar-slack> Quick design question:
[17:48] <stellar-slack> https://github.com/stellar/stellar-core/blob/master/src/overlay/Peer.h#L80-L87
[17:48] <stellar-slack> Was consideration given to using ASIO strands to avoid concurrent access to the write buffer?
[17:48] <stellar-slack> http://think-async.com/Asio/asio-1.10.6/doc/asio/reference/io_service__strand.html
[17:55] <stellar-slack> Another question( :-)):
[17:55] <stellar-slack> https://github.com/stellar/stellar-core/blob/a152f5a6f6af68ba8551bc5bf245653a976ba069/src/overlay/Peer.cpp#L364-L392
[17:55] <stellar-slack> The peer hello message doesn't seem to be doing any handshaking to check the cryptographic identity of the connecting peer and also there is no SSL encryption of the sent messages. What's the reasoning for not using these MITM and identity-checking techniques?
[17:55] <stellar-slack> donovan: there's only ever one thread involved at that level.
[17:55] <stellar-slack> @graydon: That negates the need for strands :simple_smile:
[17:55] <stellar-slack> yep
[18:00] <stellar-slack> The SSL question is a key one though. What stops an attacker tampering with the messages sent between peers without any encryption enabled between them?
[18:07] <stellar-slack> at the moment, probably only the signatures on the messages involved. most of the messages are signed and the p2p level is mostly supposed to act as an untrusted broadcast medium; but I wouldn't be surprised if there are MITM risks in there, security engineering is pretty fussy.
[18:14] <stellar-slack> Yep :simple_smile: I guess the obvious attack is suppressing TX_SET, TRANSACTION, SCP_QUORUMSET,SCP_MESSAGE messages to stop consensus being reached for a particular peer. I'm no expert, but transmitting these messages in the clear does open them for alteration.
[18:24] <stellar-slack> sure, yeah. we may well flip it over to running over TLS.
[18:25] <stellar-slack> TLS SGTM :simple_smile:
[18:41] <stellar-slack> donovan: TLS type stuff is on the road map for later.
[21:33] <jroweboy> I've got a probably simple question, but I still don't understand how, with all the different quora, each of the servers will be able to maintain a unified ledgers. I would expect that ledgers will start to diverge if the servers are isolated?
[21:33] <jroweboy> What am i misunderstanding?
[21:37] <jroweboy> (also, is this the right place to ask this sort of question?)
[21:54] <stellar-slack> jroweboy: yes the quorums have to have some degree of transitive intersection or it wont work
[22:36] <stellar-slack> gmarceau: memory leaks in the faster-timers PR. you have / know how to use asan?
[22:36] <stellar-slack> gmarceau: https://travis-ci.org/stellar/stellar-core/jobs/58018777#L5555 leak reports here
[22:40] <stellar-slack> i've been picking it up. i usually on valgrind, asan isn't that different.
[22:42] <stellar-slack> yeah for leaks they tend to find the same stuff, asan just does it faster. I'm an old valgrind fan (and even contributor! :simple_smile:
[22:43] <stellar-slack> hey, lets talk about sequence numbers
[22:43] <stellar-slack> how do they work for an account? what's different?
[22:44] <stellar-slack> did you RTFM?
[22:44] <stellar-slack> heh
[22:44] <stellar-slack> haha not let, let me read it then. thanks
[22:44] <stellar-slack> :simple_smile:
[22:44] <stellar-slack> let me find the relevant part
[22:45] <stellar-slack> https://github.com/stellar/stellar-core/blob/master/src/transactions/readme.md
[22:45] <stellar-slack> https://github.com/stellar/stellar-core/tree/master/src/transactions#sequence-number
[22:46] <stellar-slack> yes that's it - there is nothing really there that special for seq number per say for stellar-core. the cool new stuff is when combined with multisig and nested operations
[22:46] <stellar-slack> cool, thanks
[22:47] <stellar-slack> I put some examples at the bottom of the file https://github.com/stellar/stellar-core/tree/master/src/transactions#examples
[22:47] <stellar-slack> I'm not sure they are good examples - I'd love to hear from people what they think
[22:49] <stellar-slack> all right after fixing the session problem in the database I have the rebased herder changes ready at this point https://github.com/stellar/stellar-core/pull/427 any up for r+ ?
[23:01] <stellar-slack> For multisig, do all the signatures have to be submitted in the same transaction?
[23:02] <stellar-slack> @graydon: do you have an compilation entry point somewhere that turns asan `on` locally? I'm only seeing `-fsanitize=address` in the travis script
[23:05] <stellar-slack> yes: a transaction needs to be submitted with all signatures. Basically people need to pass it around
[23:05] <stellar-slack> we didn't want to use the ledger as temp storage :
[23:05] <stellar-slack> )
[23:06] <stellar-slack> I guess that's a lot simpler than some asynchronous ticketing mechanism :simple_smile:
[23:06] <stellar-slack> that would be cool to build as a sidekick service though
[23:06] <stellar-slack> the haha moment was to use "SourceAccount" like you would use "sudo"
[23:07] <stellar-slack> Lol, such unix passwd file :simple_smile:
[23:07] <stellar-slack> (well it's really using the signature as a token to impersonate for a single transaction)
[23:08] <stellar-slack> Would be interesting to see an escrow transaction sequence documented as an example.
[23:09] <stellar-slack> so with a 3rd party involved?
[23:09] <stellar-slack> Well, three parties, the buyer, the seller and the arbitrator.
[23:11] <stellar-slack> ah yes, I think it's a variation of the "swap" using a dedicated arbitrator for the top level transaction
[23:13] <stellar-slack> source=ESCROW
[23:13] <stellar-slack> Operation 1
[23:13] <stellar-slack> source=A
[23:13] <stellar-slack> Payment send X -> B
[23:13] <stellar-slack> Operation 2
[23:13] <stellar-slack> source=B
[23:13] <stellar-slack> Payment send Y -> A
[23:13] <stellar-slack> One way of approaching is that you need two out of three signatures for one of the two possible outcomes to proceed.
[23:15] <stellar-slack> ah if you have more complicated flow, funds may have to be stored in escrow accounts.
[23:15] <stellar-slack> I figured that for the really more complicated stuff, people will just want to use contracts
[23:15] <stellar-slack> They probably would if a working contract engine existed :simple_smile:
[23:16] <stellar-slack> with those semantics it's already possible to "precompute" the transitions though, so people can get creative
[23:17] <stellar-slack> basically it's the difference between "fixed" flows and complex ones that have arbitrary loops etc
[23:17] <stellar-slack> I guess that's possible with a new account per escrow operation. The account sequence would break precomputation if an account was used for multiple escrows which might complete at unknown future points in time.
[23:19] <stellar-slack> well, the good news is that even if it requires GB of data to precompute those graphs, the network only needs to see the execution path
[23:20] <stellar-slack> True, I guess it's not like there could be millions of possible transaction possibilities :simple_smile:
[23:20] <stellar-slack> Unless you were eBay :simple_smile:
[23:21] <stellar-slack> the amounts need to be precomputed as well, so the state explosion is real
[23:24] <stellar-slack> anyways like I said, this thing is probably good enough for a good percentage of the scenarios. There are lots of people focused on making general purpose trustless compute to get actual apps running in there. It's just a really big fish :simple_smile:
About StellarVerse IRC Logger
StellarValue IRC Logger
is part of