1 2015-09-26 00:16:37 <sipa> maaku: muha
  2 2015-09-26 00:17:06 <maaku> sipa: why no yoda conditions?
  3 2015-09-26 00:17:14 <sipa> maaku: they aren't used anywhere
  4 2015-09-26 00:19:02 <maaku> is this coding conventions? i figured yoda checks would be safer
  5 2015-09-26 00:19:44 <sipa> maaku: my only reason for saying that is consistency - i don't care either way otherwise
  6 2015-09-26 00:20:03 <maaku> sipa: i don't care either, just want to know what the rules are for future prs
  7 2015-09-26 00:20:40 <sipa> generally try to mimick style of the code around - less deviations means less surprises
  8 2015-09-26 01:21:11 <Luke-Jr> maaku: yoda conditions are .. obsoleteish
  9 2015-09-26 01:21:28 <Luke-Jr> maaku: pretty much every compiler has had a warning for assignment in that context, for a while now
 10 2015-09-26 01:21:58 <cfields> sipa: ping
 11 2015-09-26 01:22:12 <sipa> cfields: pung
 12 2015-09-26 01:22:16 <cfields> sipa: trying to figure out message ordering wrt rejecting invalid blocks
 13 2015-09-26 01:22:27 <cfields> *ordering requirements
 14 2015-09-26 01:22:48 <cfields> any quick insights? or do should i keep typing? :)
 15 2015-09-26 01:23:06 <cfields> -do || -should
 16 2015-09-26 01:23:08 <sipa> cfields: you mean wrt the reject message?
 17 2015-09-26 01:24:06 <cfields> sipa: yea. if you send me 10 random-order blocks, and the 5th one is invalid, what are the requirements for the ordering of the response?
 18 2015-09-26 01:24:25 <sipa> cfields: i think that message fails to be useful, because not getting a reject doesn't mean it was accepted
 19 2015-09-26 01:25:00 <sipa> cfields: so i think nobody can rely on any ordering constraints whatsoever, except that not getting it immediately means the block was 'processed'
 20 2015-09-26 01:25:16 <cfields> sipa: agreed. I'm just trying to stick to the current assumptions as much as possible
 21 2015-09-26 01:25:33 <sipa> where processed may mean 'looks ok, but not going to validate now, because it's not in the best chain'
 22 2015-09-26 01:26:32 <sipa> cfields: right now they are sort-of sent asynchronously... whenever actual validation happens
 23 2015-09-26 01:26:35 <sipa> which may be later
 24 2015-09-26 01:26:44 <cfields> right
 25 2015-09-26 01:26:46 <midnightmagic> .. I like yoda conditionals ..
 26 2015-09-26 01:27:57 <midnightmagic> lack of readability. bah. how the hell is a yoda conditional illegible
 27 2015-09-26 01:28:26 <cfields> sipa: ok, to simplify a bit. Stupid and obvious overstatement of the issue: I recieve "block" + "ping". Ok to respond immediately with "pong" ?
 28 2015-09-26 01:28:28 <maaku> readable they can be, if right perspective you have
 29 2015-09-26 01:29:36 <sipa> cfields: if you want to stick to the current behaviour... not before processing the block
 30 2015-09-26 01:29:53 <sipa> cfields: but processing does not necessarily mean full validation
 31 2015-09-26 01:30:02 <cfields> sipa: but processing may only be dumb checks, no?
 32 2015-09-26 01:30:05 <cfields> right
 33 2015-09-26 01:30:48 <sipa> i think there is a lot of software out there (mostly test software) that assumes that after a block message is sent, and the pong is received, that the current chain is the best known one
 34 2015-09-26 01:31:11 <sipa> i'm not sure how maintainable that is in the future... more asynchronous processing is definitely better for performance
 35 2015-09-26 01:31:24 <cfields> sipa: ok thanks, that's exactly the answer i was looking for
 36 2015-09-26 01:31:44 <cfields> well, wait
 37 2015-09-26 01:32:36 <cfields> that seems pretty broken then :)
 38 2015-09-26 01:32:39 <sipa> ?
 39 2015-09-26 01:33:11 <sipa> right now it is the case that after processing a message (which happens in order), the best chain will be the best known chain
 40 2015-09-26 01:33:41 <sipa> one possibility is turning it the other way around: whenever processing a command (which you do in order) requires knowledge of the best chain, you stall processing for that peer until background block processing has reached a fixed point
 41 2015-09-26 01:34:13 <sipa> but processing of a block is just initial checks + pushing it into a queue for processing
 42 2015-09-26 01:36:22 <cfields> sipa: right, that's pretty much what i'm trying to determine. I'm mapping out each command, trying to determine when the current command processing can be "backgrounded" and the next one can start
 43 2015-09-26 01:36:50 <sipa> cfields: i think you should not see it as backgrounding the execution of a command
 44 2015-09-26 01:36:57 <sipa> a command is processed immediately, in order
 45 2015-09-26 01:37:09 <sipa> but it may push work into some background worker thread
 46 2015-09-26 01:37:27 <cfields> sipa: yes. i think we're in agreement here, i'm just explaining myself terribly as always
 47 2015-09-26 01:37:35 <sipa> but for example, the worker thread for script validation may be separate from that of block validation, and shared with transaction validation
 48 2015-09-26 01:37:39 <sipa> cfields: heh, stop that
 49 2015-09-26 01:38:19 <sipa> and shared with the script validation for transactions
 50 2015-09-26 01:38:21 <sipa> i mean
 51 2015-09-26 01:38:51 <sipa> but it's complicated... more asynchronisity (is that a word?) will certainly break assumptions
 52 2015-09-26 01:39:09 <sipa> cfields: i think we should for now try to stick to the existing model, if that's possibke
 53 2015-09-26 01:39:28 <sipa> run all commands, ordered per peer
 54 2015-09-26 01:39:29 <cfields> sipa: well my hope is to emulate the current behavior in every way, but in a fully async way. if that makes any sense
 55 2015-09-26 01:39:49 <sipa> cfields: my condolences
 56 2015-09-26 01:39:53 <cfields> so that current assumptions can be lifted later as/if possible
 57 2015-09-26 01:40:30 <sipa> that sounds awesome, but very hard to do and verify
 58 2015-09-26 01:40:47 <cfields> sipa: heh, ok. forget all that for now... let's just go back to the block example for a sec..
 59 2015-09-26 01:41:17 <sipa> cfields: you don't start processing anything received by the same peer until you have fully processed its block message
 60 2015-09-26 01:41:44 <sipa> where fully processed means: your best block is the best known block at that point in time
 61 2015-09-26 01:42:00 <sipa> that's as conservative as you get :)
 62 2015-09-26 01:42:59 <cfields> ok
 63 2015-09-26 01:43:01 <sdaftuar> sipa: are you suggesting maintaining that behavior only because of test software that relies on it?
 64 2015-09-26 01:43:35 <sipa> sdaftuar: there is more than test software that relies on it
 65 2015-09-26 01:43:36 <sdaftuar> because i think we may be able to come up with workarounds for testing, at least in bitcoin core
 66 2015-09-26 01:43:37 <cfields> sdaftuar: i agree with the idea of trying to maintain current behavior, if only to make initial review/testing earlier
 67 2015-09-26 01:43:52 <sdaftuar> ah, ok
 68 2015-09-26 01:43:59 <sipa> for example, bitcoin core currently assumes that if it sends a getheaders + getdata, the headers will arrive before the data
 69 2015-09-26 01:44:42 <sipa> if you want to make things more asynchronous, the headers may need to wait, because it relies on knowing the best block, while the getdata can be processed immediately
 70 2015-09-26 01:45:59 <sdaftuar> right.  seems like maybe if you know your peer is going to respond asynchronously, that you could also wait to process the block until the headers chain comes in...  but i guess we start getting into bigger logic changes, than just the networking rewrite cfields is working on
 71 2015-09-26 01:46:28 <cfields> sipa: if it eases your mind at all, i'm not suggesting that i'm trying to make core into a tangled web of callbacks. I'm just trying to design an initial "if i recieve X message, I must reply with Y after Z condition, or I may move on without responding"
 72 2015-09-26 01:46:46 <cfields> that map will make the next steps easier, even if it doesn't translate into any real-world changes
 73 2015-09-26 01:47:40 <sipa> sdaftuar: well in this specific case the getheaders sender doesn't actually care that the other side is done processing everything, because it's asking about information for a block it already talked about
 74 2015-09-26 01:47:55 <cfields> where "moving on" may be stupid/unsafe from a validation perspective, but reasonable from the networks's perspective
 75 2015-09-26 01:48:36 <sipa> sdaftuar: so maybe an explicit 'barrier' command is useful, that stops processing for that peer until all (preexisting) background work is done
 76 2015-09-26 01:48:43 <sipa> perhaps ping could be that barrier :)
 77 2015-09-26 01:48:47 <cfields> sipa: you mean ping? :)
 78 2015-09-26 01:48:48 <cfields> heh
 79 2015-09-26 01:48:48 <sdaftuar> hah
 80 2015-09-26 01:49:49 <cfields> sipa: please read the above if you missed it, i think i finally managed to convey my point
 81 2015-09-26 01:49:57 <cfields> (only took ~20min, too :)
 82 2015-09-26 02:11:02 <cfields> petertodd: lol, read the tail of your sig on your last response on the "sendheaders" mail. My window lined up perfectly so that it looked like part of the reponse. You're usually _slightly_ more subtle :p
 83 2015-09-26 02:16:06 <jgarzik> cfields, merged your univalue PR(s)
 84 2015-09-26 02:16:22 <cfields> jgarzik: that was quick! thanks :)
 85 2015-09-26 02:16:43 <jgarzik> cfields, I like the non-recursive makefile stuff...  the old way is how I've been doing autoconf/automake for 10+ years... you are teaching me good new stuff.
 86 2015-09-26 02:17:05 <cfields> heh, good to hear
 87 2015-09-26 02:17:11 <jgarzik> cfields, Now I just need to convince you to do the same conversion for https://github.com/jgarzik/posixutils ;p
 88 2015-09-26 02:17:31 <cfields> jgarzik: in every project i've worked on that's started out as recursive, we've _always_ regretted it later
 89 2015-09-26 02:17:47 <sipa> jgarzik: you've been doing autoconf for 10 years? i feel sad for you :(
 90 2015-09-26 02:17:49 <sipa> :p
 91 2015-09-26 02:18:01 <jgarzik> cfields, I like non-recursive... it's just not how all the docs are written
 92 2015-09-26 02:18:05 <jgarzik> sipa, hehe
 93 2015-09-26 02:18:24 <cfields> sipa: why, nothing's changed since 10 years ago :p
 94 2015-09-26 02:19:07 <jgarzik> cfields, with the bazillion tiny sub-directories in posixutils, I bet a conversion would speed up the build at least 2x
 95 2015-09-26 02:19:40 <cfields> jgarzik: yep, probably. Easier to deal with warnings/errors too
 96 2015-09-26 02:20:02 <cfields> jgarzik: can you point me to a particularly nasty one? The ones i've clicked on so far are pretty straightforward
 97 2015-09-26 02:20:30 <jgarzik> cfields, they're all trivial...  just a lot of them.  sheer numbers, not complexity
 98 2015-09-26 02:20:35 <jgarzik> cfields, a few have built sources
 99 2015-09-26 02:20:39 <jgarzik> like getconf
100 2015-09-26 02:22:01 <cfields> jgarzik: any reason not to handle those dependencies without BUILT_SOURCES? They look to be straightforward
101 2015-09-26 02:23:09 <cfields> (i tend to consider BUILT_SOURCES to be harmful as it usually ends up screwing up custom targets)
102 2015-09-26 02:23:39 <jgarzik> cfields, as long as the source is dynamically built as a step in compile, I don't care how it's done :)
103 2015-09-26 02:24:31 <cfields> jgarzik: heh. if i find myself really bored one day, i might start hacking away at it :)
104 2015-09-26 02:41:28 <jgarzik> cfields, it's a fun diversion and great way to learn
105 2015-09-26 02:41:55 <cfields> jgarzik: doing up a quick one now. perfect diversion indeed :)
106 2015-09-26 02:41:58 <cfields> damn you
107 2015-09-26 02:42:07 <jgarzik> hehe
108 2015-09-26 02:58:22 <cfields> jgarzik: https://github.com/theuni/posixutils/commit/e62c4ab86061ce800c84330f1a2c3be1c9dd6986
109 2015-09-26 02:58:26 <cfields> there's your freebie :)
110 2015-09-26 02:58:31 <cfields> date night time, nnite
111 2015-09-26 11:04:25 <cfields> jgarzik: https://github.com/theuni/posixutils/commits/non-recursive-make, in case you didn't see last night
112 2015-09-26 11:04:40 <cfields> i think that's a pretty good template for doing the others
113 2015-09-26 13:30:40 <jgarzik> cfields, nifty thanks :)
114 2015-09-26 23:11:43 <cfields> jgarzik: btw, seem to be reinventing busybox?
115 2015-09-26 23:17:18 <petertodd> cfields: ???
116 2015-09-26 23:17:58 <cfields> petertodd: https://github.com/jgarzik/posixutils
117 2015-09-26 23:18:28 <petertodd> cfields: I am now even more confused :)
118 2015-09-26 23:18:41 <cfields> petertodd: heh. what was the ??? about?
119 2015-09-26 23:18:53 <petertodd> cfields: "lol, read the tail of your sig on your last response on the "sendheaders" mail."
120 2015-09-26 23:19:14 <cfields> oh, heh
121 2015-09-26 23:19:47 <cfields> petertodd: your sig on that message:
122 2015-09-26 23:19:57 <cfields> ...JXZEWrMGTIXr2zkX9mtLzfjAzDc6ZULoEAHCV3sVa0M=
123 2015-09-26 23:19:57 <cfields> =SLUT
124 2015-09-26 23:20:02 <petertodd> hahahahaha
125 2015-09-26 23:20:51 <petertodd> perfect example of why people have invented vowel-less encodings...
126 2015-09-26 23:58:43 <jgarzik> cfields, no not busybox.   Hitting the sweet spot between "tiny and lacking most features" and "overly bloated with features, most of which nobody uses"
127 2015-09-26 23:58:53 <jgarzik> the former being busybox, the latter being GNU utils
128 2015-09-26 23:59:13 <cfields> fair enough