Logged Conversation (all times UTC)
[01:16] <stellar-slack> after rebuild this morning
[01:16] <stellar-slack> `startup [default] WARN No config file stellar-core.cfg found`
[01:16] <stellar-slack> The original cfg is still present (in the binary's directory). Is core looking for a more specific location now?
[02:06] <stellar-slack> it looks for stellar-core.cfg in the cwd i believe
[02:24] <stellar-slack> andrew: thanks. Everything was cool yesterday so it could be that this particular machine (ubuntu) is just chucking a fit.
[14:46] <stellar-slack> @donovan: regarding `Last-Event-Id`, you are exactly right. And, since every other progression of data within the Stellar Network is quantized to the ledger’s beat, I have a simple mechanism that allows me to do the same for every paged endpoint in the api.
[14:46] <stellar-slack> You'll be able to do, for example: `var txs = new EventSource("http://live.stellar.org/account/12345/transactions|live.stellar.org/account/12345/transactions")` and get an streamed feed of every transaction in which account 12345 participated.
[14:46] <stellar-slack> I'm hoping to have an initial working system done today :simple_smile:
[15:06] <stellar-slack> @scott: Where `Last-Event-Id` is particularly handy is re-synchronisation for a local database after having suffered a disconnection. For example if I have all transactions up to ledger 12345 and then get disconnected, I can just do a call to the SSE endpoint, with the `Last-Event-Id` header set to 12345, and I should get all transactions for ledgers 12345 -> current with the latest transactions streaming
[15:06] <stellar-slack> difficult to orchestrate at the server end as you have to effectively join two streams into one (historical and the ever-progressing current). I’ve previously used SSE for sync-ing between servers for another project and it worked very well. Ended up with this:
[15:06] <stellar-slack> https://github.com/donovanhide/eventsource/
[15:20] <stellar-slack> yep, totally. I’ve done the same thing in the past and it worked out well. The central motivator behind the streaming and paging design in horizon is to avoid the “I was disconnected momentarily and don’t know if I missed an event” problem.
[15:25] <blibbyblob> hi
[15:25] <blibbyblob> echo..o...o
[15:25] <stellar-slack> hi
[15:26] <stellar-slack> It especially works great on mobile devices :simple_smile: The great thing is that the JS eventsource object automatically handles the reconnection with the `Last-Event-Id` header set correctly. It does mean that the event id has to be carefully chosen though. If transactions come in a set for each ledger, the ledger sequence makes sense, but if the transactions come as individual events, then a concatenat
[15:26] <stellar-slack> might make more sense. How are transactions uniquely identified in the new Stellar?
[15:27] <blibbyblob> liking the feel of stellar. was wandering if it was possible to have a currency built into it?
[15:28] <stellar-slack> If anyone feels like watching a running example:
[15:28] <stellar-slack> https://bitcoinity.org/ev/markets/markets_bitstamp_USD
[15:28] <stellar-slack> Bitcoinity doesn’t make use of sequential ids though, so it’s not the best example of resumption :simple_smile:
[15:30] <stellar-slack> donovan: For the purposes of ordering, I expose a total ordering to transactions, it’s the `(ledger-sequence, tx-order-of-application)` pair. It’s called `application_order` in the api. That tuple gets packed into an opaque paging token that will be suitable for an event-id.
[15:31] <stellar-slack> Paging? Does that make sense for SSE?
[15:31] <blibbyblob> sorry if im not explaining it properly. i am fairly newbish. I meant if it's possible to build a new coin into it like nxt has and ethereum.
[15:31] <stellar-slack> Paging for non-SSE requests
[15:31] <stellar-slack> the two systems are (paging, streaming) share some machinery, is all I’m trying to express
[15:33] <stellar-slack> blibbyblob: You can’t really build a new coin into stellar with different cryptographic properties or features, so you can’t really express something like ethereum in stellar… but you can define your own currency codes.
[15:33] <blibbyblob> you mean like tokens?
[15:33] <stellar-slack> yeah
[15:34] <blibbyblob> is there an option to include PoS? my thinking is no with tokens
[15:34] <stellar-slack> By PoS you mean proof of stake?
[15:34] <blibbyblob> i know PoS is flawed..yeah
[15:35] <blibbyblob> my main problem is because PoS is flawed then pow is only option. but to get the mining power behind it is costly
[15:36] <stellar-slack> Stellar uses consensus in place of PoS or PoW, and that’s fixed for the system. Not a lot of wiggle room there
[15:36] <stellar-slack> That’s what I meant by “different cryptographic properties or features” (sorry, that’s not a very good term).
[15:37] <stellar-slack> One slight risk though is that SSE becomes a DDOS target. If many clients all try and stream from the first ledger, it can quickly exhaust the SQL connection pool, as queries can be fairly long-running. Obviously chunking queries with `LIMIT` and a suitable `WHERE` predicate and `ORDER BY` can mitigate the connection pool exhaustion, but a client can also consume a ton of bandwidth. I guess an admin limit
[15:37] <stellar-slack> in history you can go might work.
[15:39] <stellar-slack> yep, that is a concern I’m taking into account
[15:40] <stellar-slack> My present plan: the system will always have a maximum “page” size that will be enforced by the server. After I transmit (at present) 100 records to the client I’ll kill the connection from the server side and they’ll need to reconnect. That will let us leverage the normal load balancing system, as well as the normal rate limiting system we.
[15:40] <blibbyblob> ah ok so by creating tokens does that mean they are limited to stellar or can they be used on exchanges like other coins?
[15:42] <stellar-slack> blibbyblob: have you by any chance read https://www.stellar.org/learn/explainers/#Gateways_trust_and_credit ?
[15:42] <stellar-slack> It explains the system of creating tokens (credits, we call them in Stellar terms)
[15:43] <blibbyblob> nope not yet. Just got here but gonna read up on this now. cheers :)
[15:43] <stellar-slack> np :simple_smile: happy to help
[15:45] <stellar-slack> @scott: Another approach is to simply limit the speed at which the historic events are dispatched for each individual HTTP `event/stream` request. Kind of thing Go is really good at with a goroutine per request :simple_smile: There’s a sketch of a token bucket implementation towards the bottom of this page:
[15:45] <stellar-slack> https://code.google.com/p/go-wiki/wiki/RateLimiting
[15:46] <stellar-slack> sweet. I’ll look into it. Thanks for being a sounding board :simple_smile: it’s much appreciated
[15:47] <stellar-slack> Thanks for implementing good ideas :simple_smile: SSE is cool!
[20:34] <stellar-slack> I'm restarting the testnet
[20:35] <stellar-slack> scott do I need to do anything to the horizon node?
[20:35] <stellar-slack> jed, are you clearing the ledger? If not, then horizon should continue to run just fine
[20:36] <stellar-slack> If you are clearing the ledger, we just have to clear the history db and then it will start importing the new ledger chain
[20:36] <stellar-slack> I’ve got a sql snippet I can run
[20:39] <stellar-slack> crap, `ActionController::Live` always creates a new thread per streaming request… I’m going to have to build something custom to replace it
[20:42] <stellar-slack> yeah ledger is being cleared
[20:43] <stellar-slack> jed: on the plus side the crash reporting seems to be working
[20:44] <stellar-slack> cool
[20:44] <stellar-slack> It looks like the actual assertion failure only ends up in stdout though
[20:45] <stellar-slack> so guessing I should probably send the upstart logs rather than the stellar-core logs
[20:45] <stellar-slack> or maybe both
[20:48] <stellar-slack> jed, just give me a heads up when you clear the ledger and I can run the snippet
[20:49] <stellar-slack> scott it was done
[20:49] <stellar-slack> word, fixing horizon now
About StellarVerse IRC Logger
StellarValue IRC Logger
is part of