1 2015-04-28 01:47:17 <phantomcircuit> it was reported yesterday that gcc-5.1 doesn't build bitcoin correctly on dragonfly
  2 2015-04-28 01:47:24 <andytoshi> is there a tool for c++ so i can see a call graph of all functions?
  3 2015-04-28 01:47:30 <andytoshi> even just static (file-scope) functions would be fine
  4 2015-04-28 01:47:34 <phantomcircuit> i installed gcc:5.1 on gentoo to test this and found that i cant build boost either
  5 2015-04-28 01:47:47 <phantomcircuit> so im thinking that isn't a bitcoin issue as much as it's a gcc:5.1 issue
  6 2015-04-28 01:47:58 <phantomcircuit> andytoshi, perf record -g; perf report -G
  7 2015-04-28 01:48:01 <andytoshi> thx phantomcircuit
  8 2015-04-28 01:48:12 <phantomcircuit> it's limited to stuff that actually runs though
  9 2015-04-28 01:48:13 <andytoshi> oh, that'll look at what's actually called..
 10 2015-04-28 01:48:15 <andytoshi> yeah
 11 2015-04-28 01:48:37 <andytoshi> i think what i mean to ask for is a "dependency graph"
 12 2015-04-28 01:48:58 <phantomcircuit> not sure you can build a complete dependency graph with static analysis
 13 2015-04-28 01:49:04 <andytoshi> this is fine, i'm looking at libsecp256k1 and i've already run kcov on it ... the ./tests binary hits like every path
 14 2015-04-28 01:49:08 <phantomcircuit> given function pointer shenanigans
 15 2015-04-28 01:49:18 <andytoshi> iirc s/like//
 16 2015-04-28 01:50:42 <phantomcircuit> neat
 17 2015-04-28 02:12:56 <andytoshi> sipa: why is a call to secp256k1_fe_normalize_weak() needed on group_impl.h:278 in libsecp256k1 #210
 18 2015-04-28 02:13:15 <andytoshi> in general when is normalization required and when is weak normalization sufficient?
 19 2015-04-28 03:54:42 <theymos> Is it useful to submit a pull request for my gitian sigs? Or are there already enough?
 20 2015-04-28 03:56:49 <phantomcircuit> theymos, dont see why not
 21 2015-04-28 03:58:33 <theymos> cool, thanks
 22 2015-04-28 04:41:24 <michagogo> theymos: yes, more sigs are never a bad thing
 23 2015-04-28 05:14:39 <hulkhogan_> question re: PR64 on secp256k1, is all thats required for merge a rebase of the JNI code (to include contexts)?
 24 2015-04-28 05:30:55 <hulkhogan_> Sorted it, nvm
 25 2015-04-28 10:11:46 <ThomasV> anyone from BitPay here?
 26 2015-04-28 10:22:58 <hearn> ThomasV: jgarzik is. some of them hang out in #bitcore as well, i think
 27 2015-04-28 10:23:29 <ThomasV> well, I get a certificate expired for bitpay.com, in a PR
 28 2015-04-28 10:23:48 <ThomasV> generated here: http://bitgivefoundation.org/donate-now/
 29 2015-04-28 10:24:21 <ThomasV> they renewed it on the website a few days ago
 30 2015-04-28 10:24:26 <hearn> ah
 31 2015-04-28 10:24:36 <hearn> they forgot to update their payment request server, that's bad :(
 32 2015-04-28 10:24:51 <hearn> i'll email steven
 33 2015-04-28 10:25:26 <ThomasV> ok
 34 2015-04-28 10:25:45 <ThomasV> I mailed Ryan X Chares, but then I realized he left the company :P
 35 2015-04-28 10:27:28 <hearn> is there a way to make bitcoind print ip addresses to debug.log again?
 36 2015-04-28 10:27:32 <hearn> instead of peer ids?
 37 2015-04-28 10:56:00 <jgarzik> ThomasV, hearn, the ideal is always support@bitpay.com to get a ticket
 38 2015-04-28 10:56:23 <jgarzik> it's not a dead letter, we check all that stuff daily, sometimes minute-to-minute
 39 2015-04-28 10:57:23 <ThomasV> jgarzik: ack!
 40 2015-04-28 10:58:17 <hearn> jgarzik: thanks. stephen replied right away so things are being fixedc
 41 2015-04-28 11:08:45 <jgarzik> hearn,
 42 2015-04-28 11:08:45 <jgarzik> init.cpp:    fLogIPs = GetBoolArg("-logips", false);
 43 2015-04-28 11:08:45 <jgarzik> init.cpp:    strUsage += HelpMessageOpt("-logips", strprintf(_("Include IP addresses in debug output (default: %u)"), 0));
 44 2015-04-28 11:08:53 <hearn> thanks
 45 2015-04-28 11:09:14 <hearn> i was getting spammed by some app that was trying to pretend to be Bitcoin Core (very badly)
 46 2015-04-28 11:09:22 <hearn> but it connected and disconnected too fast to find it in getpeerinfo
 47 2015-04-28 11:09:27 <hearn> it's gone now though.
 48 2015-04-28 11:09:39 <hearn> 
ACTION will remember there is a flag for this

 49 2015-04-28 11:09:46 <sipa> the gangnam style ones?
 50 2015-04-28 11:09:48 <hearn> no
 51 2015-04-28 11:10:00 <hearn> this one rather hilariously, was     /bitcoinj:0.12SNAPSHOT/Satoshi:0.2.0/
 52 2015-04-28 11:10:14 <hearn> they didn't realise that calling peerGroup.setUserAgent(name, version) *appends* to the subVer, not replaces it
 53 2015-04-28 11:10:20 <sipa> haha
 54 2015-04-28 11:10:33 <sipa> ... and 0.2.0? really? :)
 55 2015-04-28 11:10:48 <hearn> yeah, the amount of fail they packed into one line of code is quite impressive
 56 2015-04-28 11:11:21 <sipa> i once saw a website that had a meta generator tag "Microsoft Linux 2.0"
 57 2015-04-28 11:12:41 <hearn> haha
 58 2015-04-28 11:13:06 <sipa> this was a decade ago, at least
 59 2015-04-28 12:31:10 <jgarzik> jonasschnelli, sipa: where is the background for the xor'd dns seed stuff?   The bitcoind PR says "client part of $seeder commit" and the seeder commit says "client part coming soon"
 60 2015-04-28 12:31:42 <jonasschnelli> jgarzik: i need to remove "client part comming soon"
 61 2015-04-28 12:31:47 <jonasschnelli> because it's there now.
 62 2015-04-28 12:31:56 <sipa> jgarzik: the background is that some ISPs block hostnames that resolve to evil stuff
 63 2015-04-28 12:32:13 <jonasschnelli> i think it was on the mailing list
 64 2015-04-28 12:32:19 <jgarzik> jonasschnelli, You need to replace it with a real commit description :)
 65 2015-04-28 12:32:22 <jonasschnelli> IIRC it came from thebluematt
 66 2015-04-28 12:32:35 <sipa> i wonder whether the xor value should not be time depedent or something
 67 2015-04-28 12:32:47 <sipa> or at least have some configurability
 68 2015-04-28 12:32:59 <jonasschnelli> what do you mean with time dependent?
 69 2015-04-28 12:33:11 <jgarzik> rotate without inter-node communication, I presume
 70 2015-04-28 12:33:24 <sipa> you risk blocking a particular ip out if it accidentally matches something isps do not like
 71 2015-04-28 12:34:00 <sipa> one way would be to have 26 xor values, and have the client pick one by resolving X.seed.bitcoin.bla
 72 2015-04-28 12:34:50 <jonasschnelli> To prevent/complicate the xoring from unxoring through ISPs?
 73 2015-04-28 12:35:06 <sipa> no
 74 2015-04-28 12:35:18 <sipa> from accidentally blacking someone out forever
 75 2015-04-28 12:35:28 <sipa> isps can do what they want
 76 2015-04-28 12:35:50 <sipa> if their xorred ip matches a blacklist
 77 2015-04-28 12:36:54 <jonasschnelli> do you think if a xored ip matches a blacklisted ip, the DNS relay throught a ISP will be stopped?
 78 2015-04-28 12:37:08 <sipa> yes?
 79 2015-04-28 12:37:18 <jonasschnelli> could be.
 80 2015-04-28 12:37:18 <sipa> what else would you expect to happen?
 81 2015-04-28 12:37:27 <jonasschnelli> i don't know. :)
 82 2015-04-28 12:37:28 <sipa> the isp does not know it was xorred...
 83 2015-04-28 12:37:37 <jonasschnelli> maybe removing only the blacklisted ip from the response? probably no.
 84 2015-04-28 12:37:51 <sipa> no no, the block out the hostname
 85 2015-04-28 12:38:03 <sipa> so dns requests for that host just fail
 86 2015-04-28 12:38:14 <hearn> why not just move to the cartographer protocol? it's not very complicated
 87 2015-04-28 12:38:18 <hearn> solves that and a whole lot of other problems
 88 2015-04-28 12:38:30 <sipa> what is the cartographer protocol?
 89 2015-04-28 12:38:45 <jonasschnelli> hearn: DNS is distributed widely already.
 90 2015-04-28 12:38:49 <hearn> the one i designed for bitcoin xt. seeds serve IP data in a signed protocol buffer served via http.
 91 2015-04-28 12:38:57 <hearn> so it cannot be MITMd
 92 2015-04-28 12:39:11 <jonasschnelli> hearn but how many "cartographer" are running now?
 93 2015-04-28 12:39:17 <hearn> + if it misbehaves there is an audit trail, and ISPs can't interfere with it
 94 2015-04-28 12:39:46 <hearn> just one, i think. yes, incremental hacks onto the existing dns seed code are obviously going to be less work. or people can just run the seed i wrote ...
 95 2015-04-28 12:40:08 <sipa> dns has the advantage of scaling incredibly easily
 96 2015-04-28 12:40:19 <sipa> it has flaws too, obviously
 97 2015-04-28 12:40:39 <jonasschnelli> I think if we sort out the XORin risks it might be a save way of bootstraiping some node ips
 98 2015-04-28 12:40:53 <sipa> not sure what you mean by that
 99 2015-04-28 12:41:15 <jonasschnelli> sipa: how would the 26 xor values lower the risk of accitendialy xor to a blacklisted ip?
100 2015-04-28 12:41:52 <sipa> jonasschnelli: if one gets blacklisted, 25 others remain
101 2015-04-28 12:42:26 <sipa> which means the effect of blocking would only apply all the time
102 2015-04-28 12:42:39 <jonasschnelli> ah. Now i see. This makes sense.
103 2015-04-28 12:43:48 <jonasschnelli> Or could the xor key be dynamic and shared between client and server? systime is probably to inaccurate.
104 2015-04-28 12:44:49 <sipa> yes, that is also sn option
105 2015-04-28 12:46:45 <jonasschnelli> sipa: your solution would be to allow dns queries for x<0-27>.seed.sipa.bl where the x<val> would select the xor value for response?
106 2015-04-28 12:46:59 <sipa> or a. b. c. d. ...
107 2015-04-28 12:51:01 <jonasschnelli> Right. This should be implemented. As well as the detection of local ip ranges after rfc6890.
108 2015-04-28 12:51:32 <sipa> not sure that is necessary
109 2015-04-28 12:52:37 <jonasschnelli> This opendns features made me think that it's probably neccessary: https://labs.opendns.com/2013/09/03/what-are-the-suspicious-responses-and-why-you-should-block-them/
110 2015-04-28 12:52:50 <timothy> just don't use opendns
111 2015-04-28 12:53:27 <jonasschnelli> Agreed. But it's a indication of a wish to block suspicious DNS responses.
112 2015-04-28 12:53:52 <jonasschnelli> suspicious ~= dns response with ips in reserved ranges
113 2015-04-28 12:56:12 <jonasschnelli> sipa: or could we use the server IP in the response as a xoring value base: ;; SERVER: 10.0.1.1#53(10.0.1.1)? Or the WHEN line?
114 2015-04-28 12:58:06 <jonasschnelli> or define that the last ip in the response is the xor key.
115 2015-04-28 12:58:19 <sipa> that means using a raw dns library
116 2015-04-28 12:58:31 <sipa> typically you don't get access to the raw results
117 2015-04-28 12:58:57 <sipa> and if you do that, we could just use txt records with an encrypted packet inside
118 2015-04-28 12:59:13 <jonasschnelli> rewrite DNSSec. :)
119 2015-04-28 13:12:05 <jonasschnelli> what if the seeder would create a sig of all ips in the response and shorten the sigs down to 4 resp. 16 bytes? Place dns pubkeys in the client to verify.
120 2015-04-28 13:12:22 <jonasschnelli> the 4 resp. 16 bytes can be placed as ip within the response.
121 2015-04-28 13:12:31 <jonasschnelli> first ip = xor value
122 2015-04-28 13:12:36 <jonasschnelli> second ip = sig
123 2015-04-28 13:12:45 <hearn> jonasschnelli: you're just reinventing the Cartographer protocol badly, at this point. DNS is not suited for this kind of work. for instance, at some point, we may wish to start giving nodes identity keys and including them in responses, so clients that can benefit from it can know they are connecting to the real network and not a simulation
124 2015-04-28 13:12:49 <hearn> tor style
125 2015-04-28 13:12:53 <hearn> http://main.seed.vinumeris.com/peers.html
126 2015-04-28 13:13:10 <hearn> with a protobuf based protocol adding data is easy
127 2015-04-28 13:13:21 <hearn> with a DNS based protocol you end up with more and more bizarre hacks until the whole thing just falls apart
128 2015-04-28 13:14:10 <jonasschnelli> looking at https://github.com/mikehearn/httpseed
129 2015-04-28 13:14:20 <timothy> 
ACTION wants irc seed back :P

130 2015-04-28 13:14:55 <jonasschnelli> hearn: "You will need a Java 8 runtime"?!
131 2015-04-28 13:17:01 <hearn> jonasschnelli: yes, cartographer is written in kotlin and runs on the jvm
132 2015-04-28 13:17:39 <hearn> no memory leaks, buffer overflows, etc .... code can libraries like bitcoinj/mapdb and is small/simple as a result.
133 2015-04-28 13:17:51 <hearn> the protocol is just protobuf+http thought
134 2015-04-28 13:18:01 <hearn> so, it can be implemented by anything that can serve http   (no ssl required)
135 2015-04-28 13:18:59 <jonasschnelli> hearn: i see that bitcoinj adds no http seed by default. So it's up to the apps to add a seed?
136 2015-04-28 13:19:04 <timothy> why using java?
137 2015-04-28 13:19:18 <sipa> no language war here please
138 2015-04-28 13:19:24 <timothy> it's not a way
139 2015-04-28 13:19:28 <hearn> currently, yes. i might make bitcoinj mix in cartographer seeds by default in a future version. for now it's only used by Lighthouse, to locate nodes that support getutxo
140 2015-04-28 13:19:32 <timothy> bitcoin-core is written in C++
141 2015-04-28 13:19:38 <timothy> war*
142 2015-04-28 13:19:50 <timothy> and on many web server you DON'T have java
143 2015-04-28 13:19:56 <sipa> hearn: how is the signing done?
144 2015-04-28 13:19:59 <hearn> timothy: cartographer is a separate program, it's not bitcoin core.
145 2015-04-28 13:20:15 <hearn> sipa: secp256k1 over a serialized protobuf. pubkey is hard coded into the client.
146 2015-04-28 13:20:52 <hearn> jonasschnelli: that said there *is* client support for the protocol. it can enabled with one line of code, if you want to use it. so bitcoinj has support but it's not on by default.
147 2015-04-28 13:21:15 <jonasschnelli> hearn: right. i saw https://github.com/bitcoinj/bitcoinj/blob/db86185972faf068f04183e93199f728f45520d4/wallettemplate/src/main/java/wallettemplate/Main.java#L132
148 2015-04-28 13:21:24 <hearn> jonasschnelli: at some point, once we've done a bit more refactoring, i will probably switch to cartographer by default when Tor is in use, as otherwise if you pick a bad exit to locate peers, you're out of luck.
149 2015-04-28 13:21:51 <hearn> timothy: it's something you can just unzip. i can make a tarball that includes a bundled jre for linux users if people want that, it's not very hard to do.
150 2015-04-28 13:22:12 <jonasschnelli> hearn: but what if a ISP/net authority thinks "i'd like to block bitcoin". He just can block all IPs running a cartographer service?
151 2015-04-28 13:22:14 <hearn> timothy: the code itself is not written in java. it's just using the jvm as a runtime to provide things like networking, garbage collection, compilation etc.
152 2015-04-28 13:22:21 <hearn> jonasschnelli: he can do that with dns too
153 2015-04-28 13:22:37 <jonasschnelli> hearn: you can switch DNS servers?
154 2015-04-28 13:23:02 <hearn> jonasschnelli: pretty useless if the ISP has even basic DPI. but you can proxy HTTP requests too - not sure why this is a big deal.
155 2015-04-28 13:23:08 <sipa> hearn: yeah, at some point we would want something that can relay more information than just IP addresses, but bitcoin core already does have a way for that, namely just connecting to a bitcoin peer and doing a getaddr
156 2015-04-28 13:23:09 <hearn> e.g. you could proxy the lookup via tor
157 2015-04-28 13:23:22 <hearn> yes
158 2015-04-28 13:23:31 <hearn> a future version of the p2p protocol could replace much of the functionality of the cartographer protocol
159 2015-04-28 13:24:15 <hearn> it would need authentication (+optional crypto) for that. but it could be done. you would lose the transparent http caching the cartographer protocol gets but that's perhaps no big deal.
160 2015-04-28 13:24:43 <sipa> i'm not exactly keen on a protobuf dependency in the core, but if that is really an issue, we protobuf is supersimple to decode anyway
161 2015-04-28 13:25:03 <sipa> yeah, i would like crypto in the p2p protocol
162 2015-04-28 13:26:31 <jonasschnelli> i just try to ponder if it's worth to add a secp256k1 sig and masquerade as IPs in the response to secure that it would allow to detect manipulating of the DNS response. But probably over the top...
163 2015-04-28 13:26:44 <hearn> i'm not wedded to protobufs. it just happens to be convenient, well tested and extensible. satoshis format is simple but hard to extend.
164 2015-04-28 14:30:52 <StephenM347> There's no way to know if a signature (not your own) was calculated with the RFC6979 method, right?
165 2015-04-28 14:36:36 <sipa> StephenM347: indeed
166 2015-04-28 14:37:05 <sipa> StephenM347: and proving that it was signed with rfc6979 means revealing the nonce, through which anyone can calculate the private key that was used to sign
167 2015-04-28 14:37:24 <sipa> i answered oleganza on the ML, but forgot to reply to all
168 2015-04-28 14:37:51 <StephenM347> sipa: gotcha, I won't bother then
169 2015-04-28 14:49:06 <jonasschnelli> To save disk space while testing: can i share (ln -s) the block files between a datadirs between two fullnodes (one with txindex, one without)?
170 2015-04-28 14:50:45 <jonasschnelli> s/between a datadirs/between datadirs
171 2015-04-28 14:51:06 <sipa> yes, but it's a bit tricky
172 2015-04-28 14:51:18 <sipa> only link the files that aren't being touched anymore
173 2015-04-28 14:52:27 <jonasschnelli> if i don't run the nodes in parallel, i can probably just swap the blocks/index dir to enable/disable -txindex?
174 2015-04-28 14:52:52 <sipa> should be, yes
175 2015-04-28 14:53:45 <Adlai> I lay down in my bed and prepare for the come up, this being only my second time with plugging anything I wasnt exactly sure when that would be. Laying there I remember feeling first alerts within 5 minutes, a faint psychedelic mindset could be noticed. By 10 minutes my heart ratee had picked up to 90bpm and I was starting to feel slightly dizzy, I closed my eyes and focused on my breathing.
176 2015-04-28 14:53:45 <jonasschnelli> okay. Thanks.
177 2015-04-28 14:53:53 <Adlai> blah sorry about that
178 2015-04-28 14:54:02 <jonasschnelli> :-)
179 2015-04-28 15:13:35 <andytoshi> sipa: in libsec256k1 am i reading right that you need to (weakly at lesat) normalize field elements basically before any call to secp256k1_fe_mul_int, or else Bad Things (overflow in the individual words of the fe) will occur?
180 2015-04-28 15:13:56 <andytoshi> also the range of "small integer" that can be passed to secp256k1_fe_mul_int(), even after normalization, is different on different arches?
181 2015-04-28 15:14:06 <sipa> andytoshi: yup
182 2015-04-28 15:14:12 <andytoshi> kk
183 2015-04-28 15:14:14 <sipa> there is an issue about that; i should still respond
184 2015-04-28 15:14:24 <andytoshi> cool, i'm just sanity checking my own reads..
185 2015-04-28 15:14:28 <sipa> it's intentional - basically to allow per-architecture optimizations
186 2015-04-28 15:15:22 <andytoshi> i've checked all the table-generation code in #210 so far, it's really well-written ... about 60% through the first patch, so complaints except a couple incomplete comments i just noted
187 2015-04-28 15:15:40 <andytoshi> ah, yes, i did see that issue
188 2015-04-28 15:16:01 <sipa> andytoshi: thanks for the review so far; i'll try to address the comments
189 2015-04-28 15:16:16 <sipa> yeah, it has gone through several stages of rebasing and refactoring :)
190 2015-04-28 15:17:07 <andytoshi> ah, yes :) thx for not pinging me until the refined one
191 2015-04-28 18:49:13 <hearn> bahaha. trezor based code signing for lighthouse is alive
192 2015-04-28 18:50:48 <Giszmo> hearn: sounds awesome!
193 2015-04-28 18:52:01 <hearn> next step - multisig+trezor together!
194 2015-04-28 18:58:24 <jgarzik> hearn, how is over-subscription and under-subscription handled?
195 2015-04-28 18:58:43 <jgarzik> I always presumed a design where tranches were funded
196 2015-04-28 18:59:07 <hearn> in lighthouse?
197 2015-04-28 18:59:32 <hearn> if you raise too little money none can be claimed. the software won't let you claim too much money either. you have to hit the target exactly. the GUI won't let the user create a pledge that would push the project over the target
198 2015-04-28 18:59:51 <hearn> if you want to fundraise in tranches you can just create multiple projects. they're just small files so the gui makes it fairly easy.
199 2015-04-28 19:04:30 <jonasschnelli> hearn: is there a BIP for the hardware-wallet<->software-wallet interface?
200 2015-04-28 19:05:36 <hearn> no. AFAIK most hardware wallets are a bit too different to really standardise
201 2015-04-28 19:09:53 <Luke-Jr> well, Trezor's interface has been cloned by Chinese competitors at least
202 2015-04-28 19:10:00 <Luke-Jr> so it might become a de facto standard
203 2015-04-28 19:11:35 <hearn> heh
204 2015-04-28 19:11:43 <hearn> well, technically they cloned the trezor, not just the interface
205 2015-04-28 19:11:48 <Adlai> bitcoin-qt: main.cpp:3491: void ProcessGetData(CNode*): Assertion `!"cannot load block from disk"' failed.
206 2015-04-28 19:12:08 <Adlai> this is the second time I've gotten this error. last time, I restarted and everything worked fine for a few days... and it happened again.
207 2015-04-28 19:13:02 <sipa> andytoshi: version?
208 2015-04-28 19:13:06 <sipa> eh, Adlai
209 2015-04-28 19:13:21 <Luke-Jr> hearn: did they? I thought they renegged on that
210 2015-04-28 19:13:25 <Luke-Jr> oh
211 2015-04-28 19:13:27 <Luke-Jr> misread, nm
212 2015-04-28 19:13:28 <sipa> Adlai: there's a known bug in 0.10.0 that can cause this, which is fixed in 0.10.1
213 2015-04-28 19:13:43 <Adlai> sipa: v0.10.99.0-c8a1350
214 2015-04-28 19:14:22 <Adlai> also, just before the death: ERROR: CheckProofOfWork(): nBits below minimum work // ERROR: ReadBlockFromDisk: Errors in block header at CBlockDiskPos(nFile=0, nPos=184855)
215 2015-04-28 19:14:29 <Luke-Jr> Adlai: git code from master is not considered stable fyi
216 2015-04-28 19:14:37 <sipa> Adlai: ok, so that's just corrupted disk data
217 2015-04-28 19:14:39 <Adlai> Luke-Jr: I know, this is running without money
218 2015-04-28 19:15:02 <Luke-Jr> yeah, sounds like filesystem/disk error; or bad shutdown
219 2015-04-28 19:15:16 <Luke-Jr> sadly, 0.10 has not been very good about unclean shutdowns :/
220 2015-04-28 19:15:47 <Adlai> it crashed with the exact same error a few days ago, I restarted it, and it ran for a few days... any recommended way to restart without nuking the whole on-disk blockchain?
221 2015-04-28 19:15:54 <michagogo> 17:52:26 <jonasschnelli> if i don't run the nodes in parallel, i can probably just swap the blocks/index dir to enable/disable -txindex? <-- Not 100% sure, but you may want to swap out blocks/../chainstate too
222 2015-04-28 19:16:02 <Adlai> and/or find the actual bug
223 2015-04-28 19:16:04 <sipa> Adlai: run with -reindex
224 2015-04-28 19:16:17 <Luke-Jr> Adlai: the bug is in the past: the data on disk is corrupt
225 2015-04-28 19:16:17 <sipa> Adlai: it will reuse the data you already have
226 2015-04-28 19:16:45 <Adlai> oh boy. here goes...
227 2015-04-28 19:17:05 <Luke-Jr> expect a few hours/days wait for reindex
228 2015-04-28 19:17:09 <michagogo> 22:15:15 <Luke-Jr> sadly, 0.10 has not been very good about unclean shutdowns :/ <-- Hm? I thought someone did something to make writes happen in some order such that an unclean shutdown can't break the data irrecoverably
229 2015-04-28 19:17:24 <Adlai> "6 years and 14 weeks behind" ;_;
230 2015-04-28 19:17:30 <sipa> michagogo: indeed
231 2015-04-28 19:17:38 <Luke-Jr> michagogo: well, it never recovers when I have unclean shutdown
232 2015-04-28 19:17:40 <sipa> Adlai: this does not mean it will take 6 years :)
233 2015-04-28 19:17:49 <phantomcircuit> Adlai, you'll be entirely cpu limited doing a reindex
234 2015-04-28 19:17:56 <Luke-Jr> phantomcircuit: CPU and I/O
235 2015-04-28 19:18:05 <sipa> phantomcircuit: I/O initially, CPU after last checkpoint
236 2015-04-28 19:18:17 <phantomcircuit> Luke-Jr, unless he's running from a floppy disk it should be cpu limited almost immediately now
237 2015-04-28 19:18:41 <Adlai> 
ACTION fades back into the lurkground, will report if reindexing fails

238 2015-04-28 19:19:02 <Luke-Jr> phantomcircuit: not so sure. my i7 takes days to reinex
239 2015-04-28 19:19:04 <Luke-Jr> reindex*
240 2015-04-28 19:19:09 <Luke-Jr> I can only assume it's I/O-related
241 2015-04-28 19:19:40 <sipa> Luke-Jr: is your blockchain stored on an sshfs across the globe? :)
242 2015-04-28 19:19:46 <Luke-Jr> sipa: nope, local hard drive
243 2015-04-28 19:20:14 <Luke-Jr> SAMSUNG HD753LJ
244 2015-04-28 19:21:10 <phantomcircuit> Luke-Jr, might want to check SMART ...
245 2015-04-28 19:21:22 <Luke-Jr> phantomcircuit: I had to do that to get its model #
246 2015-04-28 19:21:53 <sdaftuar> sipa: will -reindex be able to use blocks on disk after the first bad block encountered?
247 2015-04-28 19:22:03 <sipa> sdaftuar: i guess not
248 2015-04-28 19:22:12 <sdaftuar> adlai's error was in nFile=0...
249 2015-04-28 19:22:17 <sdaftuar> that seems unfortunate no?
250 2015-04-28 19:22:22 <michagogo> Hm, no seeds for the 0.10.1 torrent...
251 2015-04-28 19:22:22 <sipa> oh!
252 2015-04-28 19:22:29 <michagogo> (at least not in DHT)
253 2015-04-28 19:22:40 <sipa> Adlai: you may want to just throw things out then
254 2015-04-28 19:22:50 <Adlai> 
ACTION appears to be reindexing the whole blockchain

255 2015-04-28 19:22:59 <michagogo> And 3 of the trackers had connection timeouts, followed by 2 with request timeouts.
256 2015-04-28 19:22:59 <sipa> yes, it will always reindex the whole thing
257 2015-04-28 19:23:04 <michagogo> Adlai: that's what -reindex is
258 2015-04-28 19:23:06 <Luke-Jr> michagogo: http://luke.dashjr.org/programs/bitcoin/files/bitcoind/0.10.1/
259 2015-04-28 19:23:19 <sipa> Adlai: the point is that it will fail, if one of your first blocks is already corrupted
260 2015-04-28 19:23:25 <sipa> Adlai: it won't be able to reuse anything
261 2015-04-28 19:23:39 <michagogo> Luke-Jr: thanks, webseed added
262 2015-04-28 19:23:53 <michagogo> erm, it showed up in the list and then went away immediately
263 2015-04-28 19:23:56 <Luke-Jr> eh, I don't know if my directory structure matches
264 2015-04-28 19:24:10 <Adlai> sipa: alrighty, I'll trash the whole thing. thanks sdaftuar
265 2015-04-28 19:24:25 <michagogo> Yeah, it just tried again and failed
266 2015-04-28 19:24:27 <michagogo> I guess it doesn't
267 2015-04-28 19:24:34 <phantomcircuit> Luke-Jr, to satisfy my curiosity im going to reindex from a hdd on an i3 after drop_caches
268 2015-04-28 19:24:40 <sdaftuar> Adlai: good luck
269 2015-04-28 19:24:49 <phantomcircuit> michagogo, magnet?
270 2015-04-28 19:25:06 <michagogo> 2F
271 2015-04-28 19:25:06 <michagogo> magnet:?xt=urn:btih:b6f8da60aaf2007cd6db631637951ae673e31044&dn=bitcoin-core-0.10.1&tr=udp%3A%2F%2Ftracker.openbittorrent.com%3A80%2Fannounce&tr=udp%3A%2F%2Ftracker.publicbt.com%3A80%2Fannounce&tr=udp%3A%2F%2Ftracker.ccc.de%3A80%2Fannounce&tr=udp%3A%2F%2Ftracker.coppersurfer.tk%3A6969&tr=udp%3A%2F%2Fopen.demonii.com%3A1337&ws=https%3A%2F%2Fbitcoin.org%2Fbin%
272 2015-04-28 19:25:06 <michagogo> phantomcircuit: well, it's
273 2015-04-28 19:25:07 <sdaftuar> it's kind of an unfortunate failure mode, since you have all the headers and most of the blocks, there should be a way to recover better...
274 2015-04-28 19:25:26 <michagogo> But I don't know if that'll work if it's not in DHT
275 2015-04-28 19:25:34 <Adlai> my kingdom for a write-only journaling blockchain! too bad I have no kingdom
276 2015-04-28 19:25:36 <michagogo> the .torrent is at https://bitcoin.org/bin/bitcoin-core-0.10.1/bitcoin-0.10.1.torrent
277 2015-04-28 19:25:55 <sipa> Adlai: the blockchain on disk is append-only
278 2015-04-28 19:25:57 <Adlai> will the torrent be faster than -reindex from scratch?
279 2015-04-28 19:26:02 <sipa> Adlai: no
280 2015-04-28 19:26:06 <Adlai> 
ACTION shrugs

281 2015-04-28 19:26:11 <sipa> Adlai: the torrent only downloads files
282 2015-04-28 19:26:19 <sipa> the indexing still has to be done in both cases
283 2015-04-28 19:26:32 <sipa> -reindex just tries to reuse what you already have, which is, in this case, nothing
284 2015-04-28 19:26:36 <Adlai> right
285 2015-04-28 19:26:47 <Adlai> why nothing? I'd done a reindex before
286 2015-04-28 19:27:00 <michagogo> Hm, apparently uTorrent (I don't have the right character on the keyboard and can't be bothered to get it) doesn't convert .!ut files to the actual files until the entire torrent finishes downloading :-/
287 2015-04-28 19:27:07 <sipa> Adlai: it can't index past a missing block
288 2015-04-28 19:27:21 <Adlai> 
ACTION doesn't understand how a block went missing >_<

289 2015-04-28 19:27:23 <sipa> Adlai: indexing happens by processing blocks one by one, if one is missing, it can't use any of the later ones
290 2015-04-28 19:27:43 <sipa> Adlai: 0.10.0 had a bug that caused overwrites of block data sometimes
291 2015-04-28 19:28:02 <michagogo> sipa: that seems kinda weird, though
292 2015-04-28 19:28:14 <michagogo> It should get the headers, and then realize it's just missing that one block
293 2015-04-28 19:28:29 <sipa> michagogo: it doesn't even have the headers
294 2015-04-28 19:28:53 <michagogo> I'm saying, it should get them
295 2015-04-28 19:29:09 <sipa> sure
296 2015-04-28 19:29:18 <sipa> feel free to implement that :)
297 2015-04-28 19:29:34 <Adlai> sipa: this is the "Bugfix: make CreateNewBlock return pindexPrev" ?
298 2015-04-28 19:29:44 <sipa> Adlai: hmm, no
299 2015-04-28 19:30:05 <sipa> maybe this was fixed in 0.10.0 already, and the bug was only present before release
300 2015-04-28 19:30:08 <sipa> i can't remember exactly
301 2015-04-28 19:30:23 <phantomcircuit> Adlai, there was a bug in the parser which would move the offset pointer incorrectly
302 2015-04-28 19:30:24 <Adlai> 
ACTION wants to make sure that his new version is bugfixed, kills and updates git

303 2015-04-28 19:30:44 <sipa> Adlai: use 0.10.1, run from scratch
304 2015-04-28 19:31:09 <phantomcircuit> oh right i hosed this system with gcc:5.1
305 2015-04-28 19:31:11 <phantomcircuit> >.>
306 2015-04-28 19:32:09 <michagogo> I don't get the point of uTorrent having the option to set priority among files in a torrent if it doesn't convert .!ut files to the actual files before it downloads everything
307 2015-04-28 19:32:22 <phantomcircuit> michagogo, dont use utorrent
308 2015-04-28 19:32:32 <michagogo> Okay, great, the one file I wanted downloaded faster, but I can't use it until they're all there
309 2015-04-28 19:32:51 <phantomcircuit> most trackers will significantly reduce your priority if you're using a version newer than a few years old
310 2015-04-28 19:33:23 <michagogo> Eh?
311 2015-04-28 19:33:31 <michagogo> Really? Why?
312 2015-04-28 19:34:59 <phantomcircuit> michagogo, because they dont like things it does now
313 2015-04-28 19:49:29 <jonasschnelli> sipa: https://github.com/bitcoin/bitcoin/pull/6021/files#r29281003
314 2015-04-28 19:49:45 <jonasschnelli> i kept the key unchanged between random subdomains
315 2015-04-28 19:50:02 <jonasschnelli> i can't see a benefit of a subdomain/keys map
316 2015-04-28 19:50:18 <sipa> ... then what's the point?
317 2015-04-28 19:50:29 <sipa> the same ip addresses will end up in all 26 results
318 2015-04-28 19:50:39 <sipa> resulting in all of them being blacklisted if one is
319 2015-04-28 19:51:07 <jonasschnelli> hmm... now i know what you mean. :)
320 2015-04-28 19:51:26 <jonasschnelli> right. Will adapt.
321 2015-04-28 19:51:31 <sipa> :)
322 2015-04-28 19:52:05 <jonasschnelli> sometime im slow on the update. :)
323 2015-04-28 19:52:11 <jonasschnelli> s/update/uptake
324 2015-04-28 20:01:36 <sdaftuar> cfields_: around?
325 2015-04-28 20:01:56 <cfields_> sdaftuar: for a min, yes
326 2015-04-28 20:02:29 <sdaftuar> just thought it might be easier to discuss the travis issues in 5981 here
327 2015-04-28 20:03:00 <sipa> sdaftuar: i can force a re-check of a travis build, if necessary
328 2015-04-28 20:03:14 <sdaftuar> the initial travis failure i ran into was with one of the new python tests i added to rpc-tests.sh; it failed in travis due to a 10-minute, no data received from the build timeout
329 2015-04-28 20:03:24 <cfields_> sdaftuar: sure
330 2015-04-28 20:03:26 <sdaftuar> sipa: not necessary, travis has succeeded now (after i rebased which i needed to do anyway)
331 2015-04-28 20:03:37 <cfields_> sdaftuar: ok. what was the last thing happening before that?
332 2015-04-28 20:03:49 <sdaftuar> it started maxblocksinflight.py, and then no output
333 2015-04-28 20:04:26 <cfields_> can you paste the link to the last busted travis output? assuming you have it around?
334 2015-04-28 20:04:54 <sdaftuar> ugh, i didn't save a link, is there a way for me to retrieve it somehow?
335 2015-04-28 20:05:09 <cfields_> yea, just have to dig through old links :\
336 2015-04-28 20:05:27 <sdaftuar> sorry :(
337 2015-04-28 20:05:31 <cfields_> np
338 2015-04-28 20:07:45 <cfields_> ok, looks like the last one died in the java pull-tester
339 2015-04-28 20:07:59 <sdaftuar> yeah
340 2015-04-28 20:08:22 <sdaftuar> and i believe the build prior to that one succeeded, and the one before that was the one that failed in maxblocksinflight
341 2015-04-28 20:09:03 <cfields_> still looking for that
342 2015-04-28 20:09:30 <cfields_> i'm not too worried about the one that failed in the java tool...
343 2015-04-28 20:10:07 <cfields_> ok: https://travis-ci.org/bitcoin/bitcoin/builds/58940499
344 2015-04-28 20:10:31 <sdaftuar> yes, thanks
345 2015-04-28 20:11:25 <sdaftuar> do we have an understanding of why the java tool fails sometimes?
346 2015-04-28 20:12:04 <cfields_> i haven't really seen it failing lately. BlueMatt would be the one to ask
347 2015-04-28 20:12:18 <sipa> 
ACTION whistles "Java..." and quitely walks away before the flamewar starts

348 2015-04-28 20:13:08 <sdaftuar> sipa: i had hoped that replacing the java tool with python might make some things better, but if these python scripts randomly fail too... sigh!
349 2015-04-28 20:13:19 <sipa> haha
350 2015-04-28 20:13:28 <cfields_> heh. yea, it's pretty heavyweight to setup, so some spurious breakage is reasonable to expect
351 2015-04-28 20:14:01 <cfields_> sdaftuar: well, it's very possible that it's a resource issue. disabling the java tests and running these instead would answer that question
352 2015-04-28 20:14:33 <sdaftuar> well in this build the java tool successfully ran -- or do you think just running it at all might tie up resources that cause later failure?
353 2015-04-28 20:14:34 <cfields_> (meaning: possible that the java tests are sabotaging the new ones)
354 2015-04-28 20:14:37 <sdaftuar> ah ok
355 2015-04-28 20:14:47 <sdaftuar> well we
356 2015-04-28 20:15:09 <sdaftuar> whoops, we're not ready to turn them off yet, have a lot more python tests to write before that would be the case
357 2015-04-28 20:15:26 <cfields_> right
358 2015-04-28 20:16:30 <cfields_> sdaftuar: i just picked up lunch, but i'll look deeper after. Maybe you could add some logging to the testdir init just in case it hangs there again
359 2015-04-28 20:18:27 <cfields_> mm, looks like that should be really quick
360 2015-04-28 20:18:42 <sdaftuar> yeah it's the same init that all the rpc tests do, basically
361 2015-04-28 20:18:59 <sdaftuar> it could have hung on connecting via p2p to the node, which is normally the next thing that prints
362 2015-04-28 20:19:50 <sdaftuar> but it's also now run successfully a few times, so i don't know if this is very reproducible?
363 2015-04-28 20:21:22 <cfields_> well if it's still there, it's something racy or caused by the environment
364 2015-04-28 20:21:50 <sdaftuar> not sure what you mean by still there? (build is passing now)
365 2015-04-28 20:22:24 <cfields_> yes, but these kinds of failures tend to come back randomly
366 2015-04-28 20:23:53 <sdaftuar> sure, well i'm happy to debug to try to figure out what is causing this, what's the best way to do that?  i can keep pushing to this branch i guess to trigger more builds?
367 2015-04-28 20:24:14 <sipa> sdaftuar: at some point, travis may think you're a spammer and stop building
368 2015-04-28 20:24:18 <sipa> i've had that happen
369 2015-04-28 20:24:36 <sdaftuar> ah -- i think i had that happen once too, i didn't realize that was the reason.
370 2015-04-28 20:24:39 <cfields_> sdaftuar: i'm looking into where it could be failing
371 2015-04-28 20:24:59 <sipa> sdaftuar: i'm not sure whether it's dos protection, or just a bug induces by overload :p
372 2015-04-28 20:26:15 <cfields_> sdaftuar: looks like the most likely culprit is the 'bitcoin-cli -rpcwait'
373 2015-04-28 20:28:13 <cfields_> maybe add a bit of logging around the bitcoind/bitcoin-cli -rpcwait calls? so we can track it further if it happens again?
374 2015-04-28 20:29:39 <sdaftuar> sure.  i think that would affect logging for all the tests, should i just enable extra printing for my new ones?
375 2015-04-28 20:30:11 <cfields_> in particular, if bitcoind startup fails, the rpcwait will cause the test to hang forever
376 2015-04-28 20:30:26 <sdaftuar> yes, good point
377 2015-04-28 20:30:27 <cfields_> uhmm, you can use a travis env var if you want
378 2015-04-28 20:31:04 <cfields_> er, less travis-centric than that though. You can use an env var to enable debugging, and we can enable that in travis
379 2015-04-28 20:31:24 <sdaftuar> ok sounds good
380 2015-04-28 20:31:57 <sdaftuar> would it better to have a separate pull that adds that, or should i just add it to this one?
381 2015-04-28 20:32:28 <cfields_> seems like it'd also make sense to have rpcwait take a time arg: ./bitcoin-cli -rpcwait=60
382 2015-04-28 20:32:49 <sdaftuar> agreed
383 2015-04-28 20:32:51 <cfields_> either way's fine
384 2015-04-28 20:33:11 <sdaftuar> ok cool, i'll try to push something up, thanks for your help!
385 2015-04-28 20:34:00 <cfields_> np. sorry it's so vague, but since it's so hard to repro, best we can do is add debugging to help track it down next time :\
386 2015-04-28 20:34:18 <sdaftuar> yep, makes sense...
387 2015-04-28 20:36:56 <cfields_> food time, bbl
388 2015-04-28 20:51:35 <phantomcircuit> sipa, btw as expected, reindex is cpu limited even for the first blocks with a normal hdd
389 2015-04-28 20:51:41 <phantomcircuit> after drop_caches
390 2015-04-28 20:51:54 <sipa> after what?
391 2015-04-28 20:52:05 <phantomcircuit> cold cache
392 2015-04-28 20:52:10 <phantomcircuit> i dropped the page cache
393 2015-04-28 20:52:18 <phantomcircuit> so worst possible case
394 2015-04-28 20:53:18 <sipa> oh ok
395 2015-04-28 20:53:20 <sipa> what -dbcache?
396 2015-04-28 20:53:43 <phantomcircuit> 4096
397 2015-04-28 20:53:57 <phantomcircuit> i'll try again with... what's the minimum?
398 2015-04-28 20:54:03 <sipa> 10 or so
399 2015-04-28 20:54:15 <sipa> the default is 100 i think?
400 2015-04-28 20:54:42 <phantomcircuit> 100 is default 4 is min
401 2015-04-28 20:55:42 <phantomcircuit> still clearly cpu limited
402 2015-04-28 20:56:19 <phantomcircuit> actually it's more bursty but io wait is showing as 0
403 2015-04-28 20:56:49 <phantomcircuit> i suspect there's some pipelining improvements available but i doubt it's worth it
404 2015-04-28 20:56:53 <phantomcircuit> shrug
405 2015-04-28 21:07:27 <phantomcircuit> sipa, ok yeah adjusting that to 4 triggered io limits
406 2015-04-28 21:07:34 <phantomcircuit> cpu -> io -> cpu again
407 2015-04-28 21:12:48 <sipa> cfields_: wrt jni libsecp256k1: it's not a priority of mine (and i wouldn't want you to waste time on it if you're not particularly interested in it), but if there is interest by someone, and they want to maintain the patch until mergable... sure
408 2015-04-28 21:31:52 <gmaxwell> Great. 0.10 (and git master) crash almost instantly at start if you start a new node with gen=1.
409 2015-04-28 21:32:55 <gmaxwell> It attempts to createnewblock; and the blockvalidity test catches fire because it's below the highest checkpoint.
410 2015-04-28 21:34:51 <gmaxwell> Too much of our testing is on regtest and regtest is insufficiently faithful to the production system.
411 2015-04-28 21:43:29 <cfields_> sipa: sure. I'll rebase what i've got and turn it over to him
412 2015-04-28 21:44:17 <cfields_> sipa: last time this i was hacking on it, hearn made it sound as though jni wasn't all that interesting to him from the java side
413 2015-04-28 21:45:05 <cfields_> but as long as it's interesting to someone, i agree, no reason not to add it
414 2015-04-28 21:46:18 <cfields_> gmaxwell: grr
415 2015-04-28 21:49:36 <gmaxwell> I'll have fixed in a moment.
416 2015-04-28 22:38:53 <hulkhogan_> cfields_, sipa: jni is interesting to me for a few reasons, but IIRC i thought there was someone asking about java support on libsecp256k1 somewhere, so i felt there was some business value present. also, i've been working on the jvm recently, and that PR looked like a easy win for a upstream merge. the fact that it wasn't well developed and needs a maintainer was another plus i only just found out
417 2015-04-28 22:38:59 <hulkhogan_>  about :)
418 2015-04-28 22:46:08 <gmaxwell> hulkhogan_: we haven't really been pushing for bindings to mature much since the API is not finished quite yet; though I think we have no planned _major_ changes finally.
419 2015-04-28 22:46:23 <gmaxwell> Though I still need to rename the posix non-compliant types.
420 2015-04-28 22:48:38 <hulkhogan_> its cool, i'm following development, i think libsecp256k1 is an exciting project on the engineering side, so i dont mind the unstable apis so much
421 2015-04-28 22:48:47 <Adlai> ./bu16
422 2015-04-28 23:50:22 <esso> Is Bitcoin-qt still a nondeterministic wallet?
423 2015-04-28 23:50:38 <gmaxwell> esso: Yes.
424 2015-04-28 23:50:51 <gmaxwell> well unless you configure it with an effectively infinite keypool.
425 2015-04-28 23:52:47 <phantomcircuit> esso, which i strongly suggest you do
426 2015-04-28 23:52:48 <michagogo> gmaxwell: no, not even then
427 2015-04-28 23:53:06 <michagogo> s/not//
428 2015-04-28 23:53:27 <michagogo> It's not deterministic, you won't generate the same keys again
429 2015-04-28 23:53:59 <gmaxwell> michagogo: Sure it is. If you've set your keypool to 100k keys and restore from a backup after you'll generate the same keys from getnewaddress over and over again.
430 2015-04-28 23:54:48 <gmaxwell> (so long as 100k is 'infinite' from the perspective of your use)
431 2015-04-28 23:54:51 <esso> gmaxwell: Do you think Bitcoin-qt should be a deterministic wallet?
432 2015-04-28 23:55:49 <michagogo> gmaxwell: that's not infinite
433 2015-04-28 23:55:53 <michagogo> By definition.
434 2015-04-28 23:56:02 <gmaxwell> esso: I basically invented determinstic wallets because I wanted bitcoin-qt to be; other people disagreed, and now having seen the effects in deployment (where people are using the highly specialized and materally less safe public derrivation mode exclusively); I think I may have been a little mistaken.
435 2015-04-28 23:56:06 <phantomcircuit> michagogo, for your average user it approximates infinity close enough
436 2015-04-28 23:56:13 <michagogo> phantomcircuit: maybe
437 2015-04-28 23:56:20 <gmaxwell> michagogo: it may be within the context of some usage.
438 2015-04-28 23:56:34 <gmaxwell> michagogo: which I qualified from my very first statement.
439 2015-04-28 23:56:50 <phantomcircuit> for something giving out addresses automatically? generate 1m
440 2015-04-28 23:57:25 <phantomcircuit> much more than that and the wallet code is too slow anyways
441 2015-04-28 23:57:33 <michagogo> "Infinite" != "more than you'll ever need", in my mind
442 2015-04-28 23:57:57 <michagogo> But that might just be because I'm rigid about things like that…
443 2015-04-28 23:58:15 <gmaxwell> There isn't a single wallet using determinstic generation which has proper key management either; its quite a security disaster. So one set of problems has just been replaced with another (though one usually a little less severe; though the use of public derrivation plus supporting private key export concerns me a lot)
444 2015-04-28 23:58:18 <phantomcircuit> michagogo, that's why he said "effectively"
445 2015-04-28 23:58:48 <esso> gmaxwell: Apologies for my ignorance concerning your involvement.
446 2015-04-28 23:59:11 <michagogo> phantomcircuit: I guess... Still bugs me, though :-/
447 2015-04-28 23:59:14 <moa> michagogo: there is a whole field of mathematics surrounding infinities, i.e there are different varieties.