1 2014-04-10 00:00:04 <kdomanski> let me explain what I'm trying to do here
2 2014-04-10 00:00:09 <sipa> but pulltester does more than just running that; it also compiles and runs test for several platforms
3 2014-04-10 00:00:15 <sipa> including windows
4 2014-04-10 00:00:29 <kdomanski> that's exactly what I want to do
5 2014-04-10 00:00:54 <sipa> you can use gitian for doing windows builds
6 2014-04-10 00:00:57 <kdomanski> my C++ pull request fails to compile, because it requires latest compilers
7 2014-04-10 00:01:05 <kdomanski> I want to check with C++0x enabled
8 2014-04-10 00:01:24 <sipa> well pulltester is on an old ubuntu that only has gcc 4.4 or so
9 2014-04-10 00:01:26 <sipa> or even older
10 2014-04-10 00:01:37 <kdomanski> oh
11 2014-04-10 00:03:48 <sipa> mingw-gcc and gcc are both 4.4 in pulltester
12 2014-04-10 00:04:00 <sipa> we could easily upgrade it to precise if someone with access spent the time for that
13 2014-04-10 00:04:07 <sipa> which would give 4.6
14 2014-04-10 00:12:50 <maaku> kdomanski: we're purposefully supporting older compilers
15 2014-04-10 00:18:05 <dgenr8> I do believe my clean .bitcoin AWS instance was being fed all kinds of garbage by the nodes it kept connecting to for blockchain dl
16 2014-04-10 00:18:23 <sipa> why do you think so?
17 2014-04-10 00:19:05 <dgenr8> 2014-04-10 00:14:02 ProcessBlock: ORPHAN BLOCK 169, prev=0000000000000360cf53553522d0f7d09737482c98059f9662a93fa5bfa68e03
18 2014-04-10 00:19:14 <sipa> that's totally expected
19 2014-04-10 00:19:19 <dgenr8> repeated ad nauseam and then gets unresponsive
20 2014-04-10 00:19:35 <sipa> it's just blocks arriving in the wrong order
21 2014-04-10 00:19:44 <dgenr8> ok
22 2014-04-10 00:19:50 <dgenr8> ill be more patient
23 2014-04-10 00:20:01 <sipa> yeah, it recovers on itself, but can take time
24 2014-04-10 00:20:05 <gmaxwell> AWS servers are usually IO starved, it's expected that it will get slow when processing the blockchain.
25 2014-04-10 00:20:28 <sipa> the block fetching logic is known to be flaky, and it's being worked on (slowly...)
26 2014-04-10 00:21:11 <dgenr8> what makes it choke .. too many orphans locally?
27 2014-04-10 00:21:31 <sipa> no
28 2014-04-10 00:21:39 <sipa> orphan here means block whose parent isn't known
29 2014-04-10 00:22:09 <sipa> that can either because of an actual garbage block someone gives you (unlikely, it takes a lot of effort to produce one that isn't immediately thrown out)
30 2014-04-10 00:22:23 <sipa> but almost always due to the block being received in the wrong order
31 2014-04-10 00:22:34 <sipa> long chains of blocks, with just one parent somewhere missing
32 2014-04-10 00:22:51 <sipa> when that parent eventually arrives, all orphans after it immediately get processed too
33 2014-04-10 00:22:54 <dgenr8> ok, but why does traffic slow down after a lot of orphans arrive .. i see .. tht one is hard to find?
34 2014-04-10 00:23:09 <sipa> there's no "finding"
35 2014-04-10 00:23:15 <sipa> we ask blocks from peers, and they arrive
36 2014-04-10 00:23:22 <sipa> but they can only be processed in order
37 2014-04-10 00:23:31 <sipa> you can't process a block if you haven't processed its parent
38 2014-04-10 00:23:35 <sipa> so we must queue them up
39 2014-04-10 00:23:48 <midnightmagic> huh. Is there anything special getting gitians for old versions like 0.8.0 ?
40 2014-04-10 00:24:00 <sipa> define "gitians" ?
41 2014-04-10 00:24:03 <dgenr8> understood. but the good blocks fly by at first
42 2014-04-10 00:24:14 <sipa> dgenr8: there's no good or bad blocks
43 2014-04-10 00:24:20 <sipa> dgenr8: initially, they just arrive in order
44 2014-04-10 00:24:23 <midnightmagic> sipa: Going through the gitian build process with a target of 0.8.0, for which there appears to be only two signatures, yours and gavin's.
45 2014-04-10 00:24:36 <sipa> then once there is one being missed, everything after it queues up
46 2014-04-10 00:24:48 <sipa> midnightmagic: could be?
47 2014-04-10 00:25:02 <midnightmagic> sipa: :) well okay then.
48 2014-04-10 00:25:19 <sipa> midnightmagic: i still don't know what you mean by "gitians"
49 2014-04-10 00:25:58 <midnightmagic> sipa: I am using it as a term to refer to the output of the gitian build process; namely, the signatures resulting from gsign.
50 2014-04-10 00:26:47 <sipa> so by "getting gitians", you mean "doing a gitian build" ?
51 2014-04-10 00:27:27 <dgenr8> sipa: understood, really :) Is it normal for the same orphan seq# to be repeated hundreds of times eg ORPHAN BLOCK 548 ORPHAN BLOCK 548...
52 2014-04-10 00:27:45 <sipa> dgenr8: yup, the number is the total count of orphans at that point
53 2014-04-10 00:28:02 <midnightmagic> sipa: I mean "doing a gitian build and then subsequently being able to generate not only outputs which match against the sigs in gitian.sigs, but also being able to legitimately create a fresh set of signatures which other people can use to gverify after the fact."
54 2014-04-10 00:28:02 <sipa> dgenr8: so seeing the same number again is good; it means some have been processed in the mean time
55 2014-04-10 00:28:12 <sipa> midnightmagic: got it!
56 2014-04-10 00:30:30 <midnightmagic> i don't think it's cleanly possible with plain gitian..
57 2014-04-10 00:30:43 <sipa> how so?
58 2014-04-10 00:30:55 <sipa> i think at the time you needed qt or something precompiled
59 2014-04-10 00:31:08 <sipa> or depended on the day of building
60 2014-04-10 00:31:15 <midnightmagic> sipa: I'm not even getting matching hashes for bitcoind
61 2014-04-10 00:32:57 <midnightmagic> this suggests a temporal requirement which passes with time, presuming of course I'm not doing something stupid
62 2014-04-10 00:33:06 <sipa> that sounds likely
63 2014-04-10 00:33:23 <midnightmagic> what the temporal requirement?
64 2014-04-10 00:33:27 <midnightmagic> :-P
65 2014-04-10 00:33:30 <sipa> yes
66 2014-04-10 00:33:43 <sipa> i have not heard about temporal requirements which do NOT pass with time though
67 2014-04-10 00:33:47 <sipa> do you have examples?
68 2014-04-10 00:34:56 <midnightmagic> assuming I know what you mean, many of our customers will save nearly-complete machine state for total recreation of build artifacts.
69 2014-04-10 00:35:34 <SoftwareMechanic> bip32 question. When creating wallet nodes, what should I be using for an index for the child nodes?
70 2014-04-10 00:35:42 <SoftwareMechanic> Can I just start at 1 and increment?
71 2014-04-10 00:35:55 <SoftwareMechanic> Or is that a bad idea
72 2014-04-10 00:36:42 <sipa> yes; you should start at 0 and increment :)
73 2014-04-10 00:36:59 <SoftwareMechanic> ok, cool
74 2014-04-10 00:37:23 <midnightmagic> so. whatever existed at the time in the build environment would need to be completely re-fetched and inserted into a vm, downgrading it in-place. hrm..
75 2014-04-10 00:37:40 <SoftwareMechanic> Looking at some reference code, it looks like it's appended to the key, then the required hashing is done
76 2014-04-10 00:41:30 <dgenr8> sipa: it is being limited by MAX_ORPHAN_BLOCKS=750. Many #751's sequentially with only a few ACCEPTED now and then
77 2014-04-10 00:41:43 <sipa> dgenr8: yes, after 750 they get dropped from memory
78 2014-04-10 00:41:50 <sipa> dgenr8: to prevent excessive memory usage
79 2014-04-10 00:42:21 <dgenr8> if the pool is large it could be dropping the one with the parent I need
80 2014-04-10 00:43:21 <sipa> no, it drops the last one
81 2014-04-10 00:53:30 <dgenr8> sipa: out of 24582 total orphans during load, 19954 are unique hashes
82 2014-04-10 00:53:41 <dgenr8> so far
83 2014-04-10 00:55:25 <sipa> and how many total blocks?
84 2014-04-10 00:55:57 <dgenr8> height=182306
85 2014-04-10 00:56:22 <sipa> so some 5000 duplicate blocks out of 180000
86 2014-04-10 00:56:26 <sipa> i've seen worse
87 2014-04-10 00:56:29 <sipa> but it's bad enough
88 2014-04-10 00:57:25 <dgenr8> when at max orphans, it seems to be churning. I'll try increasing the limit.
89 2014-04-10 01:02:54 <dgenr8> ...bingo, a 30-block hit ... but alas, back to 751 and 2-3 dropped blocks/second
90 2014-04-10 01:29:09 <warren> Anyone with Linux distros with glibc 2.13 or older? need testing soon
91 2014-04-10 02:08:55 <dgenr8> sipa: regarding the change https://github.com/bitcoin/bitcoin/commit/bbde1e99c893924dbef135f42c14f4df9828c6e5
92 2014-04-10 02:09:12 <dgenr8> sipa: I think perhaps most of the 750 is connected and not eligible for deletion
93 2014-04-10 02:10:25 <dgenr8> sipa: i have seen a few multi-hundred connected subchains go through ... if one gets close to 750 you've got no space
94 2014-04-10 02:10:48 <dgenr8> sipa: running now with MAX_ORPHAN_BLOCKS=2250
95 2014-04-10 02:13:55 <gmaxwell> dgenr8: that isn't how it works, the count there is just or orphan blocks. They are not connected. If you keep restarting it you'll just prolong how long it takes to sync up.
96 2014-04-10 02:20:18 <JWU42> any ideas what this is about (from debug.log)
97 2014-04-10 02:20:21 <JWU42> https://www.refheap.com/d1645e46d1ee48761ffb60cad
98 2014-04-10 02:20:35 <JWU42> some googling seems this was pasted here recently
99 2014-04-10 02:20:59 <JWU42> on common thread seems to be txindex=1 is on
100 2014-04-10 02:21:25 <JWU42> it has been stuck here with ORPHAN BLOCK 751 for over a minute now
101 2014-04-10 02:22:02 <JWU42> https://www.refheap.com/fead0ca410b46ab15199ed6db
102 2014-04-10 02:24:43 <JWU42> this is nuts
103 2014-04-10 02:24:46 <JWU42> https://www.refheap.com/06892d6867d7868b2ee57310c
104 2014-04-10 02:25:54 <gmaxwell> JWU42: it's not clear to me what problem you're reporting. The orphan blocks during fetch are a known and expected behavior during initial block download which will eventually fixed by the headers first logic.
105 2014-04-10 02:26:42 <JWU42> gmaxwell: yes, this is initial download. Server was running as an electrum server but was having issues building the electrum DB
106 2014-04-10 02:26:54 <JWU42> so redowloading the blockchain
107 2014-04-10 02:27:10 <JWU42> but I am now well past 7500 lines of ORPHAN BLOCK 751
108 2014-04-10 02:27:30 <gmaxwell> Yes. That isn't unexpected.
109 2014-04-10 02:27:39 <JWU42> very well
110 2014-04-10 02:28:05 <JWU42> I guess I haven't watched debug.log in the past
111 2014-04-10 02:29:49 <dgenr8> @gmaxwell: the change from 1/2014 deletes orphans to keep it under 750. but is does not delete orphans connected in a subchain
112 2014-04-10 02:31:13 <gmaxwell> dgenr8: sure it does. It prefers to delete the later dependant orphans because they are strictly less useful (cannot help connect the subchain to the chain so far)
113 2014-04-10 02:32:45 <dgenr8> @gmaxwell: how does that mesh with this .. says it chooses randomly? https://github.com/bitcoin/bitcoin/commit/bbde1e99c893924dbef135f42c14f4df9828c6e5
114 2014-04-10 02:33:06 <JWU42> all I know is that i have been stuck at block 176694 for 25 minutes and counting while it does this...
115 2014-04-10 02:33:57 <gmaxwell> dgenr8: It does choose randomly. But it makes absolutely _NO_ sense to drop a parent block which has dependents instead of the last dependant since dropping the parent makes all the dependants unconnectable until the parent is retrieved again.
116 2014-04-10 02:34:05 <gmaxwell> JWU42: yep.
117 2014-04-10 02:34:47 <dgenr8> @gmaxwell: agreed, but I think the large subchains are leaving an insufficient amount of space left
118 2014-04-10 02:35:01 <gmaxwell> JWU42: it will continue, eventually. This is why we need the headers first change. Unfortunately very few people showed up during 0.9 devlopment to work on or even test the code, so it had to be deferred.
119 2014-04-10 02:35:29 <gmaxwell> dgenr8: there is no question about "insufficient amount of space left" it's always removing one when it goes to remove one.
120 2014-04-10 02:35:42 <dgenr8> with my increased limit I have seen a high water mark of 1256 so far and no stalling
121 2014-04-10 02:35:49 <gmaxwell> And it always removes one which is equal to or less useful than the initial selection.
122 2014-04-10 02:36:02 <gmaxwell> dgenr8: the orphan count going up is not helpful, it just wastes your memory.
123 2014-04-10 02:36:07 <JWU42> ahh crud - forgot to tweak teh cache limit
124 2014-04-10 02:36:51 <gmaxwell> dgenr8: and yes, when its pulling down a bunch of orphans restarting it can sometimes (often even) trigger it into immediately making progress again.
125 2014-04-10 02:37:22 <JWU42> doing that now
126 2014-04-10 02:37:28 <dgenr8> yes why is that?
127 2014-04-10 02:37:49 <gmaxwell> dgenr8: because it continues pulling forward from the selected sync peer.
128 2014-04-10 02:38:15 <gmaxwell> (until that peer fails and stops responding. In which case it will sit around for a bit until it realizes its gone and continues again)
129 2014-04-10 02:38:40 <JWU42> restarted and progress
130 2014-04-10 02:38:59 <gmaxwell> the orphans are a side show triggered by new blocks on the network, giving you blocks which are too new to connect to the chain you have so far. They don't really have anything to do with the normal fetching.
131 2014-04-10 02:39:13 <gmaxwell> JWU42: yes, it would have begun making progress again in a bit on its own.
132 2014-04-10 02:39:31 <dgenr8> I follow re: delete selection but otoh presumably there is a reason for the pool in the first place
133 2014-04-10 02:39:36 <JWU42> You are far wiser than I gmaxwell so I'll trust you
134 2014-04-10 02:40:17 <JWU42> it seemed to be eternally stuck
135 2014-04-10 02:41:00 <gmaxwell> dgenr8: sure, there is a reason, to handle blocks arriving out of order normally, not during IBD.
136 2014-04-10 02:41:54 <gmaxwell> it is unreasonable to batch the chain in memory in reverse order all it will do is out of memory hosts which are 32 bit or have less than 18 gigabytes of free memory.
137 2014-04-10 02:42:36 <dgenr8> that makes sense definitely
138 2014-04-10 02:43:54 <dgenr8> was there any change in IBD time when the change went in in 1/2014
139 2014-04-10 02:44:26 <dgenr8> 0.9.0 actually
140 2014-04-10 03:05:42 <dgenr8> what if during ibd the orphan pool got into a state where there was a chain 749 long, disconnected from tip by 100 or so
141 2014-04-10 03:08:42 <dgenr8> might try an idea ... pruning orphan subchains at max/3
142 2014-04-10 03:09:10 <gmaxwell> dgenr8: you're wasting your time fussing with that.
143 2014-04-10 03:09:43 <gmaxwell> the orphan pool is largely irrelevant to IBD... and if you want to be doing development to improve in this space you should be reviewing and testing the patches that are phasing in headers first.
144 2014-04-10 03:13:57 <dgenr8> @gmaxwell: just scratching an itch. now I got curious about ibd though. i'll take a look at the headers work. thank you
145 2014-04-10 04:49:48 <vetch> -printtoconsole and -logtimestamps seem to be incompatible
146 2014-04-10 04:50:04 <vetch> is that intended for some reason?
147 2014-04-10 04:53:31 <olalonde> Hola
148 2014-04-10 05:23:31 <Nunacoin> Hello all
149 2014-04-10 05:24:14 <Nunacoin> I'm starting a new coin to fully integrate Bitcoin and BOINC
150 2014-04-10 05:24:48 <Nunacoin> Is there anyone who's interested in joining? :)
151 2014-04-10 05:26:48 <warren> There might be more coins than developers.
152 2014-04-10 05:26:55 <vetch> Nunacoin: this is off topic.
153 2014-04-10 05:27:12 <vetch> warren: http://mapofcoins.com/ < yep
154 2014-04-10 05:28:00 <vetch> Nunacoin: you also really need to read up on how distributed consensus works. it's completely incompatible with BOINC.
155 2014-04-10 05:29:24 <Nunacoin> vetch:sorry for the offtopic, but this is the first place came on my mind to look for a help ^^
156 2014-04-10 05:32:10 <Nunacoin> vetch: went to read :)
157 2014-04-10 06:52:36 <warren> wump: coryfields: do the other distros support debuginfo in separate files?
158 2014-04-10 06:52:47 <warren> hmm, no wumpus
159 2014-04-10 06:53:08 <coryfields> warren: of course
160 2014-04-10 06:53:14 <coryfields> that's what -dbg packages are
161 2014-04-10 06:54:00 <wump> warren: why would that need distribution support?
162 2014-04-10 06:54:12 <wump> warren: oh right, for tha ppa that'd make sense
163 2014-04-10 06:55:17 <warren> ok good
164 2014-04-10 06:55:36 <Luke-Jr> warren: Gentoo FEATURES=splitdebug
165 2014-04-10 06:55:51 <warren> well, if the separate debuginfo is a standard, then the gitian debuginfo could be installed there too
166 2014-04-10 06:56:04 <Luke-Jr> most users might not want to download it :p
167 2014-04-10 06:56:12 <warren> if they get crashes they can follow a guide
168 2014-04-10 06:56:16 <wumpus> no no no, we don't want to package the debug info, just build it for ourselves
169 2014-04-10 06:56:26 <Luke-Jr> half the point of split debuginfo is so that you can open core dumps in gdb on another system.
170 2014-04-10 06:56:33 <Luke-Jr> but that's bad
171 2014-04-10 06:56:35 <Luke-Jr> for bitcoin
172 2014-04-10 06:56:42 <wumpus> mind that gitian releases are usually not 'installed' at all, just copied somewhere
173 2014-04-10 06:56:49 <coryfields> well, for bitcoin, the discussion is pretty much irrelevant
174 2014-04-10 06:56:57 <wumpus> (apart from the PPA and distro-specific packages)
175 2014-04-10 06:56:59 <coryfields> since the full (unstripped) binary can be regenerated
176 2014-04-10 06:57:01 <warren> if they get crashes from the deterministic gitian build, they can follow a guide that tells htem to download the matching debuginfo and how to use it.
177 2014-04-10 06:57:45 <coryfields> warren: debug symbols don't do users any good. they send a core dump, we pair it with debug syms in a debugger to get a useful backtrace
178 2014-04-10 06:57:47 <wumpus> warren: sure; for power users that could be useful
179 2014-04-10 06:58:09 <warren> coryfields: we want to avoid a situation where they need to send a core dump ...
180 2014-04-10 06:58:20 <Luke-Jr> coryfields: O.O no plz
181 2014-04-10 06:58:24 <wumpus> coryfields: we want to limit their dependence on 'us' as much as possible
182 2014-04-10 06:58:32 <coryfields> warren: better write better code then...
183 2014-04-10 06:58:32 <wumpus> coryfields: people that can debug themselves, should
184 2014-04-10 06:58:55 <Luke-Jr> coryfields: core dumps will have ECDSA private keys.
185 2014-04-10 06:59:05 <Luke-Jr> we should probably even set ulimit to disable core dumps ourselves..
186 2014-04-10 06:59:25 <gmaxwell> Luke-Jr: well in the someday when we seperate processes they won't. :P
187 2014-04-10 06:59:33 <Luke-Jr> gmaxwell: depends on what crashed ;)
188 2014-04-10 06:59:52 <coryfields> Luke-Jr: right, apologies
189 2014-04-10 06:59:54 <gmaxwell> but we should make the symbols available, since there are technical people who will do debugging that won't have a gitian setup to build them themselves.
190 2014-04-10 06:59:56 <wumpus> I don't think warren's idea is bad, though let's first make gitian spit out the debug symbol files, after that we can determine whether and how to distribute them
191 2014-04-10 07:00:02 <gmaxwell> yea
192 2014-04-10 07:00:03 <coryfields> still seems moot given the determinism though
193 2014-04-10 07:00:39 <Luke-Jr> determinism doesn't help w/o debug symbols
194 2014-04-10 07:00:56 <gmaxwell> Luke-Jr: I mean you can go rebuild and get the debug symbols that way.
195 2014-04-10 07:01:00 <coryfields> Luke-Jr: you can generate them at will...
196 2014-04-10 07:01:14 <warren> coryfields: if it takes X effort to make a standard build and X+3 effort to make a gitian build to debug
197 2014-04-10 07:01:21 <coryfields> point was, compared to other projects where the final build process is sacred, there's no policy decision to be made here
198 2014-04-10 07:01:25 <wumpus> coryfields:right now it means having to run gitian with a different descriptor any time you want to generate debug executables, would be more useful if they were always generated
199 2014-04-10 07:01:30 <warren> it's probably X-10 effort to download the debuginfo for the gitian build instead of building
200 2014-04-10 07:01:40 <Luke-Jr> coryfields: if you can, then you might as well always generate them *shrgu*
201 2014-04-10 07:01:47 <Luke-Jr> uses a tiny bit more RAM during build
202 2014-04-10 07:01:57 <coryfields> Luke-Jr: sure, agreed
203 2014-04-10 07:02:13 <coryfields> i'm not arguing anything, just saying that anyone can do it at any point. so just do it already :)
204 2014-04-10 07:02:25 <warren> yes, gitian spits out the stripped and debuginfo separate
205 2014-04-10 07:02:33 <wumpus> coryfields: but it's autotools changes! we need you! lol
206 2014-04-10 07:02:37 <warren> err, in all builds
207 2014-04-10 07:02:43 <Luke-Jr> wumpus: it shouldn't be..
208 2014-04-10 07:02:44 <coryfields> wumpus: no it's not
209 2014-04-10 07:02:55 <coryfields> just don't strip the binary in gitian
210 2014-04-10 07:02:59 <wumpus> well the build process already strips the executables
211 2014-04-10 07:03:05 <wumpus> not the gitian script
212 2014-04-10 07:03:32 <warren> The host outside of gitian might lack tools to strip the binary outside, or might strip it in a way that differs from other hosts.
213 2014-04-10 07:03:36 <coryfields> wumpus: ok ok.. how bout 'make debug-symbols' or so?
214 2014-04-10 07:03:43 <coryfields> will do tomorrow if that's what you're after.
215 2014-04-10 07:03:48 <Luke-Jr> O.o
216 2014-04-10 07:04:00 <Luke-Jr> --disable-strip
217 2014-04-10 07:04:01 <Luke-Jr> :P
218 2014-04-10 07:04:14 <wumpus> coryfields: or doesn't the make install_strip strip the executable that is built, just the one that is installed?
219 2014-04-10 07:04:44 <coryfields> wumpus: make install_strip just strips it and sticks it in the install prefix
220 2014-04-10 07:04:56 <coryfields> we use the install prefix to mean where we want the target binary to end up
221 2014-04-10 07:05:06 <wumpus> coryfields: so it does strip the executable in the build directory as well
222 2014-04-10 07:05:07 <coryfields> you could just as easily grab the binary, and split/strip it manually
223 2014-04-10 07:05:23 <coryfields> wumpus: iirc it uses 'strip input output'
224 2014-04-10 07:05:29 <coryfields> since strip prefers not to strip in-place
225 2014-04-10 07:06:12 <wumpus> coryfields: good! then we can just split it manually, without having to call make more times (which can be really slow)
226 2014-04-10 07:06:24 <coryfields> wumpus: it's really just a matter of convenience. let me know how you'd prefer it and i'll hook it up
227 2014-04-10 07:06:35 <warren> please correct me if I'm wrong, you want the tools inside gitian to do stripping if you want it to be deterministic right?
228 2014-04-10 07:06:53 <Luke-Jr> what Gentoo does, is 'make install' to get unstripped binaries, then does the split/strip itself
229 2014-04-10 07:07:44 <warren> Fedora does that too, stripped binary goes into packagename-VERSION.rpm and a separate packagename-debuginfo-VERSION.rpm is also output
230 2014-04-10 07:08:12 <wumpus> warren: yes, we are talking about doing it in the gitian descriptors; I thought it'd need autotools changes, but that turned out not to be the case
231 2014-04-10 07:08:19 <Luke-Jr> I wonder if there's a simple script on Ubuntu (or better, platform-independent) we can just run in gitian
232 2014-04-10 07:08:37 <wumpus> a simple script? it's already super-easy to do
233 2014-04-10 07:08:43 <coryfields> Luke-Jr: sure, debian kinda revolves around those
234 2014-04-10 07:09:21 <coryfields> (and ubuntu as a consequence)
235 2014-04-10 07:10:11 <wumpus> (though I wouldn't dare to say whether the debug symbols themselves are deterministic)
236 2014-04-10 07:10:19 <coryfields> you guys are really overcomplicating :)
237 2014-04-10 07:10:36 <wumpus> coryfields: yes, it's just a objcopy --only-keep-debug line per executable as you said
238 2014-04-10 07:10:42 <coryfields> there's a binary, we strip it. we can grab it at any point and do whatever we want with it
239 2014-04-10 07:10:59 <Luke-Jr> there's multiple binaries :P
240 2014-04-10 07:11:40 <wumpus> coryfields: btw; a different question, we pass --disable-dependency-tracking to configure while building bitcoin, does this mean that every 'make' call will do the entire build? (because it ignores the dependencies that are already there)
241 2014-04-10 07:12:04 <wumpus> coryfields: I noticed that in the windows build at least, quite some duplicate building happens
242 2014-04-10 07:12:12 <coryfields> wumpus: nah, it just means that it can't tell what needs to be rebuilt if something changes
243 2014-04-10 07:12:23 <wumpus> ok... must have another reason then
244 2014-04-10 07:12:29 <coryfields> so if you touch a .h, the correct .cpps won't be rebuilt as a result
245 2014-04-10 07:12:48 <coryfields> we use that because that tracking can get in the way of determinism
246 2014-04-10 07:12:53 <wumpus> thanks for explaining
247 2014-04-10 07:12:56 <wumpus> yeah I figured
248 2014-04-10 07:13:11 <coryfields> np
249 2014-04-10 07:13:48 <coryfields> wumpus: we also mess with the maintainer rules. those make it so that if you touch (for example) Makefile.am, automake will re-run
250 2014-04-10 07:13:58 <coryfields> with those disabled, you're on your own for that stuff
251 2014-04-10 07:14:56 <coryfields> iirc we change that one because faketime screws with it
252 2014-04-10 07:15:18 <wumpus> I noticed; the timestamps can get in the way there, causing makefile to be rebuilt time after time, it's painful to watch :)
253 2014-04-10 07:15:33 <coryfields> yea
254 2014-04-10 07:15:48 <coryfields> that's one of the reasons i spent so much time fixing up binutils
255 2014-04-10 07:15:58 <coryfields> if we moved to trusty, we could kill off faketime :)
256 2014-04-10 07:16:13 <hno> Is there any established practice for how to generate orphan block chains? I guess that temporarily isolating some bitcoin nodes on testnet while mining should do the trick?
257 2014-04-10 07:16:14 <coryfields> (my patches are included in trusty's binutils)
258 2014-04-10 07:16:16 <wumpus> yes congratulations on that, especially on getting it upstream :)
259 2014-04-10 07:17:06 <Luke-Jr> hno: orphan block chain would be deleting a parent block :P
260 2014-04-10 07:17:27 <coryfields> i'm very much on gmaxwell's side there. lots has changed in the compiler world in the last few years
261 2014-04-10 07:17:35 <coryfields> but that's a different discussion and it's bed time :)
262 2014-04-10 07:17:44 <wumpus> so, let's switch to trusty tahr for building? can you handle all the symbols?
263 2014-04-10 07:18:22 <hno> Luke-Jr, stale blocks then?
264 2014-04-10 07:18:33 <gmaxwell> coryfields: I might be a bit biased just because there are a number of miscompilation bugs I found that were fixed in 4.7 and 4.8. :P
265 2014-04-10 07:19:00 <coryfields> gmaxwell: i found 4.7 to be a complete dud, personally
266 2014-04-10 07:19:04 <wumpus> it's be almost impossible to get the resulting executables to run on older distributions, and patching over the libc++ version gap would involve importing all the c11++ symbols into bitcoin ....
267 2014-04-10 07:19:06 <coryfields> seems like it was just trying to piss me off
268 2014-04-10 07:19:11 <coryfields> wumpus: you're not going to bait me that easily :)
269 2014-04-10 07:19:38 <wumpus> but if we just use static building for compatibility on the other hand... :-)
270 2014-04-10 07:19:52 <warren> hno: try -regtest
271 2014-04-10 07:20:13 <warren> hno: it's much easier to make competing forks and test software on reorg behavior there.
272 2014-04-10 07:20:44 <coryfields> gmaxwell: on the flip-side though, i don't have a single complaint about 4.8. I'd be curious to hear what you've run into
273 2014-04-10 07:21:42 <coryfields> (though by that time, i'd pretty much gone full-clang, so maybe i didn't try as hard to break 4.8)
274 2014-04-10 07:22:16 <gmaxwell> I'm pretty good at breaking stuff.
275 2014-04-10 07:23:09 <wumpus> how does that work with clang, does it use the same libc++ as gcc?
276 2014-04-10 07:23:33 <coryfields> wumpus: for now, for linux, it uses libstdc++ by default
277 2014-04-10 07:23:52 <coryfields> (careful saying libc++, that very specifically means the exact opposite of what you wanted ;)
278 2014-04-10 07:24:00 <warren> would you be in favor of a deterministic build of gcc/clang, and a set of matching symbols in the compat library?
279 2014-04-10 07:24:02 <wumpus> if not, it'd be an option to switch to clang for gitian building (at least for linux target)
280 2014-04-10 07:24:44 <coryfields> imo the most reasonable choice for gitian is whatever is closest to what devs use day-to-day
281 2014-04-10 07:25:01 <warren> gitian during 0.8 different substantially from that
282 2014-04-10 07:25:15 <warren> it's a little closer now but falling behind again
283 2014-04-10 07:25:27 <coryfields> shipping binaries that are substantially different from what a dev has ever run is just a bad idea.
284 2014-04-10 07:25:28 <wumpus> coryfields: btw... couldn't we just link libstdc++ statically to avoid the issue?
285 2014-04-10 07:25:45 <wumpus> coryfields: we'd still need to patch over GLIBC symbols, but not libstdc++
286 2014-04-10 07:26:01 <coryfields> wumpus: yea, that was my original plan. unfortunately, ubuntu's packagers didn't play nice with my plan.
287 2014-04-10 07:26:08 <wumpus> ugh
288 2014-04-10 07:26:12 <warren> if we replace gcc that might work
289 2014-04-10 07:26:18 <Luke-Jr> wumpus: no.
290 2014-04-10 07:26:31 <Luke-Jr> pretty sure GNU's libc stuff is LGPL
291 2014-04-10 07:26:33 <Luke-Jr> and we use OpenSSL
292 2014-04-10 07:26:39 <coryfields> precice's libstdc++ is built without -fPIC
293 2014-04-10 07:26:41 <warren> oh damn
294 2014-04-10 07:26:43 <warren> liense
295 2014-04-10 07:26:44 <Luke-Jr> a static binary of that would be illegal to distrbiute
296 2014-04-10 07:26:44 <warren> license
297 2014-04-10 07:26:49 <gmaxwell> ACTION shakes his fist at the payment protocol
298 2014-04-10 07:26:51 <wumpus> coryfields: let me guess, you ran into -fPIC problems
299 2014-04-10 07:26:59 <warren> Luke-Jr: then the windows build is already illegal
300 2014-04-10 07:27:05 <gmaxwell> we're _so_ close to not needing openssl at all. :P
301 2014-04-10 07:27:09 <Luke-Jr> warren: it shouldn't be using GNU libs?
302 2014-04-10 07:27:15 <wumpus> gmaxwell: well, if we just drop the wallet...
303 2014-04-10 07:27:19 <wumpus> :P
304 2014-04-10 07:27:20 <coryfields> there's an exemption for static glibc
305 2014-04-10 07:27:28 <coryfields> and libstdc++ as well iirc
306 2014-04-10 07:28:08 <wumpus> how can our distribution ever be illegal? everything is open source
307 2014-04-10 07:28:13 <wumpus> it's even provable due to gitian
308 2014-04-10 07:28:25 <gmaxwell> openssl has a stupid license.
309 2014-04-10 07:28:50 <coryfields> wumpus: you'll find companies who won't let gpl in their building
310 2014-04-10 07:28:57 <coryfields> if that's, for ex, bitpay.. that's a problem
311 2014-04-10 07:29:10 <wumpus> coryfields: I know. But that's not our problem, is it?
312 2014-04-10 07:29:19 <wumpus> coryfields: they don't have to use the gitian build
313 2014-04-10 07:29:36 <coryfields> wumpus: the binary has nothing to do with it
314 2014-04-10 07:29:44 <coryfields> you'd be arguing that they should write their own core
315 2014-04-10 07:29:52 <gmaxwell> wumpus: OpenSSL's license:
316 2014-04-10 07:29:53 <gmaxwell> * 3. All advertising materials mentioning features or use of this
317 2014-04-10 07:29:53 <gmaxwell> * for use in the OpenSSL Toolkit. (http://www.openssl.org/)"
318 2014-04-10 07:29:53 <gmaxwell> * software must display the following acknowledgment:
319 2014-04-10 07:29:53 <gmaxwell> * "This product includes software developed by the OpenSSL Project
320 2014-04-10 07:30:15 <coryfields> gmaxwell: their license did wonders for them on the news this week :p
321 2014-04-10 07:30:17 <Luke-Jr> wumpus: OpenSSL cannot legally be linked with GPL
322 2014-04-10 07:30:20 <wumpus> huh? no I am not arguing that they should write their own core, just that there is nothing inherently in bitcoin that requires those libraries to be linked statically
323 2014-04-10 07:30:28 <gmaxwell> This is incompatible with any copyleft free software licenseâ except things that have exceptions, which I believe lgpl and libc++ do.
324 2014-04-10 07:31:03 <wumpus> we just do that for easy of distribution, but within a company you'd likely build your own version anyway, using dynamic libraries of the operating system
325 2014-04-10 07:31:05 <endofcrypto> price comments?
326 2014-04-10 07:31:17 <warren> kick
327 2014-04-10 07:31:20 <Luke-Jr> /kickban endofcrypto off-topic and spam
328 2014-04-10 07:31:26 <gmaxwell> It's minutia in any case. I wouldn't advise losing sleep over it, esp since we're now pretty close to elimiating the dependency at least in a daemon only build. :)
329 2014-04-10 07:31:56 <Luke-Jr> actually, if we're including any GPL content, we need to distribute source for it too
330 2014-04-10 07:31:58 <coryfields> gmaxwell: agreed
331 2014-04-10 07:32:00 <Luke-Jr> even if it's unmodified..
332 2014-04-10 07:32:03 <warren> so what did we actually decide to do here?
333 2014-04-10 07:32:23 <warren> make available an optional deterministic debuginfo for download?
334 2014-04-10 07:32:28 <coryfields> Luke-Jr: no we don't. that's fsf rumor #3 or so :)
335 2014-04-10 07:32:39 <Luke-Jr> coryfields: GPLv2 at least
336 2014-04-10 07:32:59 <gmaxwell> Luke-Jr: technically we only have to make it available if someone requests it.
337 2014-04-10 07:33:02 <coryfields> Luke-Jr: nah, you just need to be able to point requests to the source.
338 2014-04-10 07:33:07 <gmaxwell> We can also charge our costs of doing so. :)
339 2014-04-10 07:33:11 <gmaxwell> coryfields: that isn't true.
340 2014-04-10 07:33:11 <Luke-Jr> gmaxwell: that option requires us to provide a written offer upfront
341 2014-04-10 07:33:16 <coryfields> if that source is burdened by your request, you can handle it yourself and charge for it
342 2014-04-10 07:33:18 <coryfields> what ^^ said
343 2014-04-10 07:33:30 <wumpus> warren: if someone adds building the symbols to the gitian process, we can afterwards decide how to distrbute it
344 2014-04-10 07:33:41 <wumpus> warren: if not, this discussion is moot
345 2014-04-10 07:33:42 <endofcrypto> why was openssl included in the protocol?
346 2014-04-10 07:33:46 <gmaxwell> seriously, you guys should stop debating licencing minutia in here.
347 2014-04-10 07:33:56 <Luke-Jr> GPLv3 allows you to point at a 3rd party IFF that 3rd party used the source-offer option
348 2014-04-10 07:33:58 <gmaxwell> endofcrypto: what protocol?
349 2014-04-10 07:34:06 <endofcrypto> bitcoin core
350 2014-04-10 07:34:14 <gmaxwell> endofcrypto: it's not a part of the bitcoin protocol.
351 2014-04-10 07:34:17 <coryfields> i really wonder if i get dragged in to all of these discussions, or if i somehow cause them...
352 2014-04-10 07:34:19 <wumpus> endofcrypto: #bitcoin please
353 2014-04-10 07:34:49 <endofcrypto> gmaxwell: wasn't it in the "bitcoin core" the main implementation?
354 2014-04-10 07:35:12 <Luke-Jr> gmaxwell: shoulda done the ban too :P
355 2014-04-10 07:35:33 <gmaxwell> endofcrypto: you've been asked to move into #bitcoin
356 2014-04-10 07:36:16 <warren> wumpus: we didn't agree to a plan?
357 2014-04-10 07:36:23 <wumpus> warren: that's the plan
358 2014-04-10 07:36:26 <coryfields> wumpus: anyway, let me know if i can help with symbol stuff. as for future releases, i'd like to work on a policy and try to stick to it. but not tonight :)
359 2014-04-10 07:36:39 <hno> warren, thanks. That has a different blockchan and networkid, right?
360 2014-04-10 07:37:03 <warren> hno: yes, different magic and some genesis params, same address version as testnet
361 2014-04-10 07:37:18 <warren> hno: it lets you generate blocks whenever you want
362 2014-04-10 07:38:01 <hno> do I understand it correctly that regtest mode is everything of bitcoin except there is no difficulty?
363 2014-04-10 07:38:23 <wumpus> warren: but I'm ok with what you propose re: distributing debug symbols, as I've already said a few times
364 2014-04-10 07:39:25 <warren> wumpus: ok
365 2014-04-10 07:39:47 <hno> I guess the nonstandard check is also disabled like on testnet?
366 2014-04-10 07:39:56 <warren> wumpus: I don't know if there's any debuginfo standard that is consistent across distros, if not then the alternate download is just an unstripped binary
367 2014-04-10 07:41:58 <hno> Not entirely clear how to control the generated block contents in regtest mode. Do one need to use setgenerate, or can one submit blocks like usual?
368 2014-04-10 07:42:28 <wumpus> warren: objcopy --only-keep-debug (raw dwarf) format is standard as it is what gdb gets
369 2014-04-10 07:42:41 <warren> hno: each time you setgenerate it makes one block and stops
370 2014-04-10 07:42:43 <wumpus> warren: sometimes it's gz-compressed, but that's it..
371 2014-04-10 07:42:57 <wumpus> warren: there is really no variance in that
372 2014-04-10 07:43:12 <warren> wumpus: do the distros install debuginfo in the same place for gdb to look for it?
373 2014-04-10 07:43:16 <hno> warren, but how to control the included transactions?
374 2014-04-10 07:43:29 <warren> hno: whatever is in the mempool at the time the block is created, like normal
375 2014-04-10 07:43:40 <coryfields> warren: distro policy goes out the window as soon as we distribute our own binary
376 2014-04-10 07:43:42 <wumpus> warren: I can't say -- I do know that those paths are configurable
377 2014-04-10 07:43:46 <coryfields> warren: man objcopy
378 2014-04-10 07:43:53 <wumpus> it's a simply a gdb configuration option
379 2014-04-10 07:43:56 <coryfields> warren: those are the definitive answers
380 2014-04-10 07:43:57 <warren> ok
381 2014-04-10 07:44:03 <wumpus> messed around a lot with that for embedded debugging
382 2014-04-10 07:44:13 <warren> you two have something in common
383 2014-04-10 07:44:17 <hno> warren, the coinbase transaction is not in mempool.
384 2014-04-10 07:44:23 <warren> oh
385 2014-04-10 07:45:05 <warren> hno: then you probably have to write a hacked up GBT miner, I think it will accept any difficulty
386 2014-04-10 07:45:55 <coryfields> nnite
387 2014-04-10 07:47:26 <wumpus> night coryfields
388 2014-04-10 07:47:27 <endofcrypto> hi gmaxwell
389 2014-04-10 07:49:56 <hno> warren, ok. I'll go and read some bitcoind code I think..
390 2014-04-10 07:51:59 <gmaxwell> (PM context)
391 2014-04-10 07:52:23 <warren> hno: actually, you best look at pool code that makes a custom coinbase tx
392 2014-04-10 07:52:38 <warren> hno: p2pool or eloipool are the most common I think
393 2014-04-10 07:53:45 <warren> Anyone have one of those useless 300MH ASICMiner USB sticks? Hoping for a donation to put it into a data center on a permanent testnet node. It will generate a block if no block happens for 20 minutes.
394 2014-04-10 07:55:06 <vetch> warren: maybe BTC Guild has a few left over? they used to sell them.
395 2014-04-10 08:35:04 <wumpus> testers needed that run Debian Stable, Ubuntu 10.04, or CentOS 6.5 for an alternative to static linking gitian releases: https://github.com/bitcoin/bitcoin/pull/4042
396 2014-04-10 08:36:10 <warren> wumpus: I'll post binaries to make it easier to test
397 2014-04-10 08:36:23 <wumpus> I did that myself already
398 2014-04-10 08:36:32 <wumpus> see bottom post
399 2014-04-10 08:36:36 <warren> ahhh thanks
400 2014-04-10 08:37:33 <warren> wumpus: hmm, coryfields wanted more than just linking testing
401 2014-04-10 08:37:39 <wumpus> (would be pretty pointless otherwise, 'can you help test on these old OSes, oh, you need to build first on a newer OS'
402 2014-04-10 08:37:52 <wumpus> warren: people can do whatever testing they want
403 2014-04-10 08:38:00 <wumpus> step one is to test linking
404 2014-04-10 08:38:08 <warren> yes, step one
405 2014-04-10 08:38:11 <wumpus> if they want to test more that's great
406 2014-04-10 08:38:22 <wumpus> ie, at least run the test_bitcoin and test_bitcoin-qt
407 2014-04-10 08:38:26 <warren> I'm testing 10.04, already tested those other OS's for linking
408 2014-04-10 08:38:38 <wumpus> why didn't you tell me?
409 2014-04-10 08:38:50 <warren> because the tests he wants is far more than linking
410 2014-04-10 08:39:14 <wumpus> still you could have told me that you already tested lining
411 2014-04-10 08:39:28 <wumpus> so that I don't spend time typing a post for nothing
412 2014-04-10 08:40:09 <warren> we wouldn't have posted that at all if we didn't test linking
413 2014-04-10 08:40:15 <wumpus> ...
414 2014-04-10 08:40:25 <wumpus> ok, post removed, you can handle this if you want
415 2014-04-10 08:40:33 <warren> well your build is helpful
416 2014-04-10 08:41:07 <warren> wumpus: please repost, it's helpful to know how old of a distro it works
417 2014-04-10 08:41:14 <warren> you also have binaries
418 2014-04-10 08:46:22 <anddam> do third-party wallet softwares listed at https://bitcoin.org/en/choose-your-wallet follow the dependency checking and code auditing discussed at https://docs.google.com/document/d/1naenR6N6fMWSpHM0f4jpQhYBEkCEQDbLBs8AXC19Y-o/edit?pli=1 ?
419 2014-04-10 08:46:43 <wumpus> ok, I'll repost the binaries
420 2014-04-10 08:47:33 <wumpus> but please try to keep me informed of what work is happening, so that we don't do double work, we're with few people enough as we are
421 2014-04-10 08:47:54 <warren> wumpus: thanks, sorry about the confusion.
422 2014-04-10 08:51:31 <gmaxwell> anddam: none of them are full node implementations, so they have lower consensus implications.
423 2014-04-10 08:58:28 <anddam> got a couple minutes?
424 2014-04-10 08:58:30 <anddam> gmaxwell: ^^
425 2014-04-10 08:59:48 <hno> Hmm.. someone is mining a little too hard on testnet.
426 2014-04-10 09:03:33 <hno> How often do the testnet difficulty adjust?
427 2014-04-10 09:04:51 <hno> every 2016 block as on main chain?
428 2014-04-10 09:05:20 <kinlo> yes but it does readjust down to 1 after 30 min of no activity as an extra rule
429 2014-04-10 09:05:31 <kinlo> besides that extra rule, everything is exactly the same
430 2014-04-10 09:08:16 <vetch> kinlo: hno: after 20 minutes it adjusts to difficulty 1 but only for a single block.
431 2014-04-10 09:10:32 <hno> So I need to wait 40 minuts for testnet to settle at a acceptable rate, or disconnect from the network. Someone generates 1 block/s now at diff 256.
432 2014-04-10 09:21:45 <Luke-Jr> tcatm: you alive?
433 2014-04-10 09:21:57 <kinlo> hno: if you need to perform tests, use testnet in a box
434 2014-04-10 09:22:11 <Luke-Jr> kinlo: testnet-in-a-box is broken :<
435 2014-04-10 09:22:19 <kinlo> instead of waiting for the public testnet to become "free"
436 2014-04-10 09:22:49 <wumpus> even better to use regtest
437 2014-04-10 09:22:59 <kinlo> Luke-Jr: depends on what you call testnet-in-a-box: the pre-made .tar.gz might be broken, but setting up a "testnet in a box" is just a matter of setting up 2 bitcoind's and setting the correct config file
438 2014-04-10 09:23:02 <wumpus> it's like testnet in a box but you can request blocks at will, no need for mining or delays
439 2014-04-10 09:23:12 <Luke-Jr> kinlo: except for the checkpoint lockin
440 2014-04-10 09:23:16 <jouke> regtest ftw
441 2014-04-10 09:23:29 <kinlo> Luke-Jr: there is no checkpoint lockin in testnet afaik?
442 2014-04-10 09:23:32 <dexX7> what would be the way to go, if i wanted to get the position of a tx inside a block? i tried ReadBlockFromDisk + enumerate txs but it appears to be slow
443 2014-04-10 09:23:40 <wumpus> kinlo: there is one checkpoint on testnet, but pretty early
444 2014-04-10 09:23:45 <kinlo> ic
445 2014-04-10 09:23:48 <Luke-Jr> wumpus: but totally breaks TNIAB
446 2014-04-10 09:23:49 <warren> wumpus: regarding the proposed change to -regtest address versions, I'm hoping to talk to jgarzik before writing it.
447 2014-04-10 09:24:01 <wumpus> kinlo: so you could start a testnet in a box after that, if you even wanted to
448 2014-04-10 09:24:21 <Luke-Jr> hm, tcatm was here Apr 2nd at least
449 2014-04-10 09:25:05 <Luke-Jr> wumpus: since you're the main project maintainer, I guess you can bypass tcatm.. ok to add warren too? (transifex)
450 2014-04-10 09:25:34 <wumpus> Luke-Jr: yes, when you're changing the maintainers anyway, can you add me (laanwj) too ?
451 2014-04-10 09:26:12 <warren> hno: FYI, I'm soon putting a permanent miner on testnet that adds one block if nobody else did so in the previous 20 minutes. I need to acquire a tiny miner that can be reliably enabled/disabled by RPC.
452 2014-04-10 09:26:16 <wumpus> Luke-Jr: but adding warren is fine, I already discussed with him
453 2014-04-10 09:26:35 <Luke-Jr> done
454 2014-04-10 09:26:45 <hno> kinlo, I also need distributed connectivity for these tests, and not in a mood to build a full bitcoin network right now. Quicker to just wait for diff to adjust.
455 2014-04-10 09:26:50 <warren> hno: I'm also proposing that -regtest's address versions be changed to be unique.
456 2014-04-10 09:27:24 <warren> hno: what is unsuitable about regtest for what you need to do?
457 2014-04-10 09:27:38 <hno> warren, you can easily split any miner into a tiny portion mining on testnet.
458 2014-04-10 09:28:23 <warren> I have zero mining hardware, and the goal isn't to constantly mine testnet, just to ensure there are at least 3 blocks per hour so people testing transactions don't get stuck.
459 2014-04-10 09:29:32 <hno> warren, I already have a miner running 24/7 at about diff 0.2 rate.
460 2014-04-10 09:29:43 <warren> oh, since when?
461 2014-04-10 09:29:59 <warren> last week my friend tried to test stuff but got frustrated when there were no blocks
462 2014-04-10 09:30:26 <warren> hno: anyway, what about regtest is unsuitable for your needs?
463 2014-04-10 09:30:57 <hno> warren, only that I also need a bitcoin network and third parties participating..
464 2014-04-10 09:30:59 <sipa> running at difficulty 0.2 will still mean only one block per 70 minutes, if difficulty is high
465 2014-04-10 09:31:20 <warren> hno: third parties can join your own regtest network
466 2014-04-10 09:32:03 <warren> hno: no mining client knows about testnet's "drop difficulty to 1 if no blocks in 20 minutes" rule, unless you wrote it yourself
467 2014-04-10 09:33:10 <hno> warren, sure they can, but requires a bit more configuration of everyone involved.
468 2014-04-10 09:33:25 <warren> eh? addnode=IP:PORT is the only config
469 2014-04-10 09:33:46 <hno> warren, how do you make that in android wallets etc?
470 2014-04-10 09:33:46 <warren> hno: what purpose do you have for making orphan chains?
471 2014-04-10 09:34:01 <hno> warren, the orphans is only one of many tests.
472 2014-04-10 09:34:39 <sipa> i wouldn't promote regtest as a public network
473 2014-04-10 09:34:43 <warren> hno: android is hte harder part, you would need to edit the app and build it yourself, testnet android app is a separate build from mainnnet, and I don't think it knows about regtest at all
474 2014-04-10 09:35:00 <warren> the app does have an option to connect to a particular IP
475 2014-04-10 09:35:10 <vetch> warren: cgminer picks up the difficulty drop
476 2014-04-10 09:35:21 <vetch> warren: bfgminer does too with the right settings, according to luke
477 2014-04-10 09:36:43 <warren> hno: even if you make your own testnet-in-a-box, that still requires an equally time consuming custom android app (hacked to not connect to the real testnet)
478 2014-04-10 09:36:44 <warren> vetch: oh? Luke-Jr ?
479 2014-04-10 09:37:00 <vetch> warren: yes, I was talking about it with him earlier
480 2014-04-10 09:37:01 <warren> vetch: only a few weeks ago Luke-Jr said it doesn't
481 2014-04-10 09:37:04 <hno> warren, yes. And is why I don't and use testnet3 as is.
482 2014-04-10 09:37:24 <warren> hard to control tests on testnet3
483 2014-04-10 09:37:26 <warren> but ok
484 2014-04-10 09:38:16 <hno> For our purposes it's just slightly annoying if someone just thrown a bit too much hash power at it like just now.
485 2014-04-10 09:38:23 <vetch> warren: I mentioned that it didn't seem to and he corrected me that it did if you changed the scantime (?) setting
486 2014-04-10 09:38:46 <warren> hno: yeah, I got my friend onto regtest because testnet3 was too unpredictable
487 2014-04-10 09:39:05 <warren> vetch: oh, GBT with a very short scantime?
488 2014-04-10 09:39:19 <vetch> hno: wow, whoever that is has started mining even faster. I was able to outrun them yesterday and fork their blocks out today. doesn't look like I could today.
489 2014-04-10 09:39:23 <vetch> warren: yes
490 2014-04-10 09:39:40 <hno> warren, Yes, we might eventually go there as well.
491 2014-04-10 09:39:54 <warren> "I was able to outrun them yesterday and fork their blocks out today." <--- this is exactly why people can't use testnet to test things =)
492 2014-04-10 09:40:32 <hno> warren, that's exaclty why one should...
493 2014-04-10 09:40:40 <hno> you never know what happens :)
494 2014-04-10 09:40:42 <vetch> warren: hey, they were already being abusive and mining a block few seconds. they only lost a few minutes work at most.
495 2014-04-10 09:41:01 <gmaxwell> warren: thats exactly why testnet is useful. because you actually do get some exceptional events.
496 2014-04-10 09:41:03 <warren> hno: IMHO, tests need to be automated and predictable... sure additionally see how your software deals with 20k reorg
497 2014-04-10 09:41:26 <hno> warren, depends on what is being tested.
498 2014-04-10 09:41:33 <vetch> yeah, I failed my 4000 block reorg by killing my nodes chainstate (somehow) :(
499 2014-04-10 09:41:43 <sipa> vetch: OOM?
500 2014-04-10 09:41:48 <warren> In any case, I hope to get a few of the common tools and wallets with regtest support.
501 2014-04-10 09:41:52 <gmaxwell> vetch: he used your blacklisting patch.
502 2014-04-10 09:42:06 <vetch> sipa: yeah, using your blacklisting patch
503 2014-04-10 09:42:51 <sipa> unless you rebased that, it's likely that a huge reorg will kill a node
504 2014-04-10 09:42:56 <warren> I'm familiar with electrum, android wallet, bitcore and insight, so those will be able to talk regtest if I have time to do it.
505 2014-04-10 09:43:00 <sipa> as it does the whole reorg in ram
506 2014-04-10 09:43:26 <vetch> sipa: http://pastebin.com/raw.php?i=4rUJVePY
507 2014-04-10 09:43:51 <sipa> vetch: oooh that's interesting
508 2014-04-10 09:45:11 <vetch> sipa: I don't think it was a memory issue, I was usually a ridiculously oversized VPS for the task
509 2014-04-10 09:45:25 <sipa> vetch: nope, doesn't look like it
510 2014-04-10 09:47:18 <sipa> vetch: this may be serious
511 2014-04-10 09:47:35 <sipa> i have no time to look into it now
512 2014-04-10 09:47:57 <vetch> sipa: this *wasn't* with a rebased patch
513 2014-04-10 09:48:51 <sipa> yeah, some things changed in the mean time
514 2014-04-10 09:49:08 <sipa> that may affect its behaviour in large reorgs
515 2014-04-10 09:49:29 <sipa> i'll try to get blacklistblock rebased
516 2014-04-10 09:49:40 <sipa> this weekend or so
517 2014-04-10 09:53:09 <vetch> sipa: hopefully not serious and just an issue with the patch somehow
518 2014-04-10 10:13:38 <hno> Looks like whoever it was just stopped mining. Maybe realized testnet coins are not worth their storage..
519 2014-04-10 10:15:30 <vetch> hno: they haven't stopped.
520 2014-04-10 10:16:06 <vetch> hno: I'm seeing height 219319
521 2014-04-10 10:17:27 <sipa> vetch: was this on a datadir that had been touched by newer code (0.9 in particular)?
522 2014-04-10 10:17:57 <vetch> sipa: completely new sync from the network, had never been touched by anything other than your branch checked out from github.
523 2014-04-10 10:18:19 <hno> vetch, hm.... 219287 here.
524 2014-04-10 10:18:44 <sipa> vetch: worries me :)
525 2014-04-10 10:18:51 <vetch> hno: you've fallen behind, the network is at 219382
526 2014-04-10 10:18:53 <warren> vetch: what branch is this?
527 2014-04-10 10:19:12 <sipa> blacklistblock
528 2014-04-10 10:19:28 <vetch> warren: https://github.com/sipa/bitcoin/tree/blacklist
529 2014-04-10 10:19:30 <hno> vetch, I hear you, but why....
530 2014-04-10 10:20:04 <warren> hmm, that code is pretty close to 0.8
531 2014-04-10 10:20:06 <vetch> hno: checked out your log?
532 2014-04-10 10:21:35 <hno> vetch, Not yet.
533 2014-04-10 10:21:51 <vetch> sipa: warren: hold on, I'll see if I have a snapshot of the virtual machine in that state.
534 2014-04-10 10:27:18 <vetch> sipa: warren: doesn't look like it, I was running with -printtoconsole so none of the logs got written to disk.
535 2014-04-10 10:28:41 <hno> vetch, nothing in logs, and plenty of network connections.
536 2014-04-10 10:29:20 <hno> confusing.
537 2014-04-10 10:30:55 <vetch> hno: I'm doing a sync from testnet3 in a fresh instance now, lets see if I can catch up.
538 2014-04-10 10:33:21 <hno> vetch, restarted bitcoind one of the bitcoin testnet nodes and it catched up again.
539 2014-04-10 10:33:38 <vetch> hno: weird.
540 2014-04-10 10:34:33 <hno> vetch, very much.
541 2014-04-10 10:36:29 <hno> Hmm... the blocks did not propagate to the other nodes however.
542 2014-04-10 10:37:26 <hno> all running 0.9.0
543 2014-04-10 10:38:01 <hno> with connect= to the node I restarted.
544 2014-04-10 11:00:22 <warren> wumpus: btw, does bitcoin-qt work in wine? it broke sometime before 0.9 right?
545 2014-04-10 11:02:32 <wumpus> warren: it works great in wine
546 2014-04-10 11:05:58 <warren> ok thanks
547 2014-04-10 11:11:22 <hno> This is worrying. All my testnet nodes need restart to catch up, else they are stuck on 219287.
548 2014-04-10 11:12:03 <warren> ACTION looks at his nodes
549 2014-04-10 11:13:02 <warren> "blocks" : 219393,
550 2014-04-10 11:13:02 <warren> [btc@lego ~]$ bitcoind -testnet getinfo |grep blocks
551 2014-04-10 11:17:46 <hno> No, not all. But many, and one was stuck on 205487.
552 2014-04-10 11:17:54 <hno> Aha.
553 2014-04-10 11:18:15 <hno> error: {"code":-2,"message":"Safe mode: Warning: Displayed transactions may not be correct! You may need to upgrade, or other nodes may need to upgrade."}
554 2014-04-10 11:19:08 <vetch> mm, my local testnet3 node is still trying to reorganise and catch up. sitting around 214952
555 2014-04-10 11:22:36 <hno> Question: Is that "safe mode" error an effect of the node being stuck on a past block, or the cause to it being stuck?
556 2014-04-10 11:23:04 <vetch> I think "safe mode" is triggered when you can see that there's other chains significantly higher than yours
557 2014-04-10 11:23:53 <hno> It refuses to respond to getblockhash rpc calls, so not sure how to understand what state it is in.
558 2014-04-10 11:24:11 <vetch> it's meant to be so you can use -alertnotify to kill whatever service you are running. say you were an exchange, that would be significant cause for you to freeze all transactions until you can work what's going on.
559 2014-04-10 11:25:31 <hno> hmm... it still responds to getblocktemplate.
560 2014-04-10 11:27:11 <vetch> hno: tail -f ~/.bitcoin/testnet3/debug.log
561 2014-04-10 11:30:53 <hno> vetch, I am. But nothing of value logged.
562 2014-04-10 11:32:03 <hno> and a restart of that bitcoind did not help either.
563 2014-04-10 11:38:12 <warren> hno: can you make a backup of that datadir?
564 2014-04-10 11:38:44 <vetch> hno: looks like our high powered miner has finally given up though, #219395 seems to be the current best chain.
565 2014-04-10 11:39:08 <hno> warren, yes.
566 2014-04-10 11:42:29 <hno> Hmm.. maybe didn't catch it in time. It now said
567 2014-04-10 11:42:31 <hno> 2014-04-10 11:41:21 CheckForkWarningConditions: Warning: Large valid fork found
568 2014-04-10 11:42:31 <hno> Chain state database corruption likely.
569 2014-04-10 11:42:31 <hno> forking the chain at height 205487 (00000000f52a446a4166f9dff3f7a14d3d341f84a586332056d24a827bf7a11c)
570 2014-04-10 11:42:31 <hno> lasting to height 219396 (00000000c2ed1c8d70291765d7762bc6520df619cc15d9150063534848146552).
571 2014-04-10 11:42:33 <hno> 2014-04-10 11:41:21 ProcessBlock: ACCEPTED
572 2014-04-10 11:50:42 <vetch> hno: something isn't right there, you really should be able to recover from almost anything like that
573 2014-04-10 11:52:24 <vetch> hno: what'd your current best block hash?
574 2014-04-10 11:56:31 <gmaxwell> petertodd: trustfree was less important when you were sitting across from the person and had red hot tongs in your hands.
575 2014-04-10 11:59:38 <vetch> gmaxwell: you know when I said my node was "crawling" trying to catch up on testnet (unrelated to other issues), I found out why. there's a section of huge blocks in a long run around 208200 that were created last month. that's one mystery solved.
576 2014-04-10 12:00:51 <vetch> gmaxwell: each is just someone spending 0.00001BTC to themselves over and over again for 800kb, but that's the reason.
577 2014-04-10 12:18:37 <hno> vetch, I have reset that node now. It's currently downloading the blockchain again from one of my other nodes.
578 2014-04-10 12:19:00 <hno> but I have the testnet3 data directory saved for further analysis.
579 2014-04-10 12:20:38 <hno> vetch, but the block number reported by getblockcount did not change when the above message was logged. Still reported 205487
580 2014-04-10 12:21:00 <hno> and was still in error condition.
581 2014-04-10 12:22:13 <hno> unfortunatel no time to look at it today, Need to run to celebrate 2 year birthday of one of the kids.
582 2014-04-10 12:59:26 <Luke-Jr> shadders: btw, you're aware PSJ no longer works, right?
583 2014-04-10 13:00:53 <lclc> sipa: does your secp256k1 library also work on ARM?
584 2014-04-10 13:01:27 <sipa> lclc: i believe coryfields has tested it on ARM, and the unit tests passed
585 2014-04-10 13:01:51 <Luke-Jr> ACTION forgets the secp256k1 status on x32
586 2014-04-10 13:02:04 <gmaxwell> it works on arm
587 2014-04-10 13:02:09 <lclc> ok thanks!
588 2014-04-10 13:02:11 <gmaxwell> it's quite slow, but openssl is slower.