1 2015-11-15 00:39:11 <midnightmagic> michagogo: What is the environment you're using these days to build the osx gitian? Ubuntu? What version host? Has anyone moved up from precise yet?
 2 2015-11-15 02:11:41 <wizkid057> which hash does getbestblockhash return?  the hash of the latest fully verified block, or the best headers-only block?
 3 2015-11-15 02:12:20 <wizkid057> my guess is the latter, since sometimes polling getbestblockhash gives me a new block hash before blocknotify does
 4 2015-11-15 02:12:36 <wizkid057> and if it's the former, I'd wonder why blocknotify is delayed
 5 2015-11-15 02:13:53 <gmaxwell> If it's doing the latter then its buggy.
 6 2015-11-15 02:14:25 <wizkid057> gmaxwell: I've been testing with polling getbestblockhash, and it's returned the new hash up to 2 seconds before blocknotify
 7 2015-11-15 02:15:30 <wizkid057> which is what led me to the question in the first place since the code looks like it should return the latest verified block
 8 2015-11-15 02:15:44 <wizkid057> which would make blocknotify happen first always
 9 2015-11-15 02:16:18 <gmaxwell> yes, looking at the code it returns the active tip.
10 2015-11-15 02:16:31 <gmaxwell> are you comparing times on different hosts?
11 2015-11-15 02:16:41 <wizkid057> no
12 2015-11-15 02:17:06 <wizkid057> I have my GBT proxy polling getbestblockhash once per second, as well as listening for blocknotify from the same bitcoind
13 2015-11-15 02:17:49 <wizkid057> I just found it odd that getbestblockhash would return the newest hash before blocknotify
14 2015-11-15 02:18:47 <kanzure> wizkid057: my best-practice rule on my end has been to only treat blocknotify as a wake-up event, and look at all blocks each time blocknotify events are received
15 2015-11-15 02:19:16 <kanzure> wizkid057: but uh, if this behavior is buggy in the way you are describing, then perhaps it would be ideal to propose this be inserted into the documentation or switch blocknotify to be blockpingnotify or something..
16 2015-11-15 02:19:36 <gmaxwell> stop
17 2015-11-15 02:20:35 <gmaxwell> please don't be shitty engineers, it's embarassing. I am getting fed up with "hey something is weird, lets randomly replace database software" or whatever sorts of responses. :/
18 2015-11-15 02:21:04 <wizkid057> nah, blocknotify is super reliable
19 2015-11-15 02:21:26 <gmaxwell> in any case, what I think you're seeing is that the blocknotify call runs after inventory pushes of the block for all peers. And this can give enough time for a getbestblockhash to run ahead.
20 2015-11-15 02:21:49 <wizkid057> oh it does? interesting
21 2015-11-15 02:21:56 <gmaxwell> It would probably be usesful to add timing logs all through that function to see where the delays are.
22 2015-11-15 02:21:59 <wizkid057> that would explain a lot
23 2015-11-15 02:22:10 <gmaxwell> but thats my best first guess.
24 2015-11-15 02:22:29 <gmaxwell> wizkid057: well until recently (I guess) everyone used p2pool for block notify listening.
25 2015-11-15 02:22:32 <kanzure> the other reason why i look at all recent blocks when i receive a blocknotify event is because my blocknotify events are processed on a delay due to reasons unrelated to bitcoind.
26 2015-11-15 02:23:10 <gmaxwell> blocknotify is not the fastest of interfaces, I mean it spawns a new thread and then runs system in it.
27 2015-11-15 02:23:24 <wizkid057> well, if it's safe to build on getbestblockhash then I'm not all that concerned.  blocknotify can be delayed for hours or not happen at all as long as I have the latest valid hash to build on from somewhere
28 2015-11-15 02:24:00 <wizkid057> just trying to figure out the fastest method to get the latest validated block hash to build on
29 2015-11-15 02:24:39 <wizkid057> I just don't want to be building on unverified headers
30 2015-11-15 02:26:01 <gmaxwell> wizkid057: if you have a way to benchmark, please try moving uiInterface.NotifyBlockTip to the part right below the getblockhash in that block in main.cpp around lime 2400 in master.
31 2015-11-15 02:28:11 <gmaxwell> if you confirm that fixes it, I think it'll be prudent to just make the change upstream. because the notify spawns a thread it shouldn't create any significant delay, while the peer broadcast might contend for cs_vNodes and suffer other delays.
32 2015-11-15 02:29:25 <wizkid057> makes sense.  I'll test it out
33 2015-11-15 02:29:34 <wizkid057> thanks
34 2015-11-15 02:31:37 <wizkid057> looks like 9 out of the last 40 blocks my polling of getbestblockhash returned the latest hash before blocknotify
35 2015-11-15 03:09:50 <michagogo> midnightmagic: right now it's like this: I have my Windows computer. It runs Virtualbox, with a VM running Ubuntu 14.04.
36 2015-11-15 03:10:15 <michagogo> The builds all run in an LXC in that VM
37 2015-11-15 03:10:58 <michagogo> (i.e. the Trusty Virtualbox VM is what gbuild runs in)
38 2015-11-15 03:41:35 <gmaxwell> wizkid057: I'll bug you in 24 hours about performance numbers from that change.
39 2015-11-15 03:48:03 <wizkid057> roger
40 2015-11-15 04:53:49 <midnightmagic> I *think* I figured out the problem: I think 0.10.x doesn't automatically retrieve the signature.tar.gz : I assumed it would. I think that's my mistake.
41 2015-11-15 04:53:53 <midnightmagic> \o
42 2015-11-15 05:05:51 <wizkid057> gmaxwell: initial results are promising.  modified bitcoind's blocknotify is beating all of my other bitcoind's blocknotify 100% of the time so far, polling getbestblockhash hasn't beat it since the change
43 2015-11-15 13:47:03 <jtimon> cfields: I would be very happy if we could get #6625 out of the way of libconsensus encapsulation and globals removal asap
44 2015-11-15 13:48:27 <jtimon> it is still mostly your code
45 2015-11-15 13:53:05 <jtimon> you could also close #6526 if #6625 is merged
46 2015-11-15 13:54:31 <jtimon> and jgarzik could maintain bip102 much more easily
47 2015-11-15 16:04:27 <wizkid057> gmaxwell: so, that one line change improves blocknotify times for sure.  interestingly enough, my polling of getbestblockhash sometimes still gets the latest hash a few ms before blocknotify, but it seems like when I poll right as bitcoind is verifying a block getbestblockhash hangs for a short time, then returns the new block hash... which is fine, just unexpected
48 2015-11-15 16:07:19 <wizkid057> usually my polling returns almost instantly (<50ms), but when bitcoind is verifying a block and polling outputs the new hash it can hang from anywhere from 100ms to 1000ms before returning with the new block hash.  not sure if that's intended behavior, but I like it.
49 2015-11-15 16:07:55 <wizkid057> (that's unrelated to the blocknotify 1-line change)