1 2014-12-24 00:00:00 <cfields> i've bumped bdb at least once personally
2 2014-12-24 00:00:00 <wumpus> those are built very quickly though
3 2014-12-24 00:00:09 <wumpus> qt is the only one that takes significant time
4 2014-12-24 00:00:19 <cfields> but also, any change to cflags or etc will cause a bump as well
5 2014-12-24 00:01:08 <wumpus> but this will become more important if we also start building toolchains, they shouldn't be handled in the same way
6 2014-12-24 00:01:27 <cfields> which part?
7 2014-12-24 00:01:39 <cfields> if you mean the gitian cache, that was just a nasty hack
8 2014-12-24 00:01:43 <wumpus> rebuilds should be incredibly rare
9 2014-12-24 00:02:00 <wumpus> and probably not happen automatically in the first place
10 2014-12-24 00:02:30 <cfields> wumpus: yea, the process is pretty well settled by now.
11 2014-12-24 00:02:41 <cfields> it'll probably be bumpy for quite a while after adding toolchains though
12 2014-12-24 00:03:10 <michagogo> BTW, what's the policy on bitcoin.org/bin and prereleases?
13 2014-12-24 00:03:26 <michagogo> bins at /bin/0.10.0/test?
14 2014-12-24 00:03:28 <wumpus> it's not acceptable if some flag changed that gcc and everything randomly gets rebuilt
15 2014-12-24 00:03:31 <Luke-Jr> michagogo: I would imagine it's the same "at least 3 signers" rule
16 2014-12-24 00:03:38 <michagogo> Luke-Jr: right
17 2014-12-24 00:03:46 <michagogo> (which we have now, I think?)
18 2014-12-24 00:03:52 <Luke-Jr> wait, that's bitcoincore.org, right? :x
19 2014-12-24 00:03:52 <wumpus> michagogo: need three signers, and need signed executables for windows and mac before uploading
20 2014-12-24 00:04:04 <michagogo> wumpus: ah, right, so waiting on Gavin
21 2014-12-24 00:04:19 <wumpus> anyhow there's no huryr
22 2014-12-24 00:04:23 <michagogo> What was the decision on the OS X signing in the end, btw?
23 2014-12-24 00:04:27 <cfields> wumpus: not sure what you mean there. if gcc changed, we have to assume that everything needs to be rebuilt
24 2014-12-24 00:04:28 <wumpus> any of you actually tested the resulting executables already?
25 2014-12-24 00:04:29 <michagogo> wumpus: yeah, I know
26 2014-12-24 00:04:45 <Luke-Jr> why are we using bitcoin.org for Bitcoin Core downloads still?
27 2014-12-24 00:04:48 <wumpus> cfields: yes *if* gcc changed, that's not what I mean
28 2014-12-24 00:04:59 <wumpus> cfields: I mean if say the major version changes, or the depends change
29 2014-12-24 00:05:23 <wumpus> cfields: that shouldn't trigger a rebuild of the entire toolchain
30 2014-12-24 00:05:39 <wumpus> Luke-Jr: what else would you propose?
31 2014-12-24 00:05:40 <michagogo> Ooh
32 2014-12-24 00:05:55 <Luke-Jr> wumpus: bitcoincore.org seems the obvious location
33 2014-12-24 00:05:55 <michagogo> The splash screen is a window now and not one of those stupid splash screen popup things :D
34 2014-12-24 00:06:00 <cfields> wumpus: oh, sure.
35 2014-12-24 00:06:20 <wumpus> Luke-Jr: bitcoin.org has a lot of bandwidth, bitcoincore is some crappy vm in gavin's basement or such :P
36 2014-12-24 00:06:25 <Luke-Jr> oh
37 2014-12-24 00:06:29 <michagogo> And a percentage on verifying blocks? Woot
38 2014-12-24 00:06:42 <wumpus> (just kidding about the basement, but it's a crappy vm hosted somewhere)
39 2014-12-24 00:07:01 <Luke-Jr> guess no rush if there's a good reason for it
40 2014-12-24 00:08:07 <michagogo> Also, bitcoin.org *is* still Bitcoin Core's site, for better or for worse -- bitcoin.org/download gives you BC
41 2014-12-24 00:08:22 <michagogo> erm, bitcoin.org/en/download *
42 2014-12-24 00:08:31 <Luke-Jr> michagogo: it's not "supposed" to be Bitcoin Core's website
43 2014-12-24 00:08:34 <wumpus> instead of overloading a central server it would be better (or at least more principled) to have a more decentralized way of distributing executables, there is work on a torrent for example
44 2014-12-24 00:08:41 <gmaxwell> We want to be inclusive, but that doesn't extend to presenting a false equivilence.
45 2014-12-24 00:08:51 <michagogo> Luke-Jr: Sure, but it sort of is atm
46 2014-12-24 00:09:08 <michagogo> If we want it not to be, the pretty download page should also move elsewhere
47 2014-12-24 00:09:31 <wumpus> it is not bitcoin core's homepage, just the executables are hosted there
48 2014-12-24 00:09:44 <Luke-Jr> I wonder if saivann has time to maintain a "distinct" simple page for Bitcoin Core
49 2014-12-24 00:10:32 <Luke-Jr> I assume bitcoincore.org's VM is used for something and can't just be moved to the same bitcoin.org infra?
50 2014-12-24 00:11:09 <wumpus> why?
51 2014-12-24 00:11:32 <Luke-Jr> why what?
52 2014-12-24 00:11:47 <wumpus> why are you so held up about bitcoincore.org, it's just a silly dev server, and hosts the fallback for the depends packages currently, there are no plans for it
53 2014-12-24 00:13:54 <gmaxwell> Luke-Jr: you realize that the inclusiveness efforts have primarily had the effect of significantly pushing down the full node count, and falsely presenting to new bitcoin users that it's acceptable for no one to run bitcoin core, and that all these choices are more or less equivilent, right? (I'm supportive of being inclusive too, but there are _severe_ costs even before you get into trying to extra
54 2014-12-24 00:14:00 <gmaxwell> ct out the last bit of minutia around equality)
55 2014-12-24 00:15:48 <Luke-Jr> it's no big deal or urgency, but it makes sense for Bitcoin Core downloads to be hosted on a Bitcoin Core domain, not a community information website. I consider it a different matter than inclusiveness
56 2014-12-24 00:15:50 <wumpus> yes I'd say it more prominently needs to feature a guide to run a full node
57 2014-12-24 00:16:14 <wumpus> there's an item 'contributing to the network' that says you should run a full node but it doesn't actually link to anything
58 2014-12-24 00:16:30 <harding> wumpus: writing that guide is next on my todo list, after I finish updating the docs to cover 0.10.
59 2014-12-24 00:16:45 <wumpus> harding: cool!
60 2014-12-24 00:17:05 <gmaxwell> there is a lot that can be easily improved. Personally I've held off on making that push until we fix a number of things in Bitcoin Core-- headers first was one of them. Ultimately this may turn out to be poor strategically.
61 2014-12-24 00:17:10 <cfields> grr, that cache/cache thing is actually tricky
62 2014-12-24 00:17:47 <michagogo> Win64 binary seems to work
63 2014-12-24 00:17:50 <wumpus> well in my experience there are lots of people that want to run a full node but don't really know what they're getting into, a friendly guide that simply lists what you need and what you need to do could help there
64 2014-12-24 00:17:55 <gmaxwell> (memory usage was another, we've also made big strides there, usable pruning is another, there is a raft of wallet improvements we need, etc.)
65 2014-12-24 00:18:01 <michagogo> At least, in that it's not crashing on laucch or anything
66 2014-12-24 00:18:04 <michagogo> launch*
67 2014-12-24 00:18:12 <gmaxwell> e.g. the approach of make "what you're getting into" less of a big deal. :)
68 2014-12-24 00:18:34 <wumpus> sure, but they don't know in the first place that doesn't help, there's a large aspect of information lackage
69 2014-12-24 00:19:15 <wumpus> of course it helps to reduce the requirements, but they still need to be made clear, I mean
70 2014-12-24 00:19:48 <gmaxwell> Yea, both are required. Mostly I am concerned about a visible improvement there causing a bunch of people who'd held back to come in and have a negative expirence.
71 2014-12-24 00:22:11 <michagogo> hm, wtf is "subver" : "Dain 0.0.1"
72 2014-12-24 00:22:23 <wumpus> I mean if it justs says you need 50GB of disk space, 1GB of memory, then people with less will not even try, and not have a negative experience either
73 2014-12-24 00:22:26 <michagogo> and why is it connecting to me 6-7 times on a freshly booted node
74 2014-12-24 00:22:51 <wumpus> if you don't say anything, then they'll try to run it on say a rpi and be disappointed
75 2014-12-24 00:23:40 <michagogo> Looks like they're all from 130.211.0.0/16
76 2014-12-24 00:24:09 <michagogo> several from one address
77 2014-12-24 00:25:14 <wumpus> none of those on any of my nodes
78 2014-12-24 00:25:41 <michagogo> Ooh, I like the new Peers tab
79 2014-12-24 00:26:29 <michagogo> Though it would be nice if there were an option to add more columns
80 2014-12-24 00:27:44 <cfields> michagogo: would you mind verifying a fix for the gitian cache/cache problem?
81 2014-12-24 00:28:28 <michagogo> cfields: could you make a dummy descriptor that doesn't do all the building, so it can be tested quickly?
82 2014-12-24 00:28:43 <michagogo> If not, I'll do it tomorrow
83 2014-12-24 00:28:48 <JWU42> so 0.10.0 is still a few weeks away?
84 2014-12-24 00:28:49 <michagogo> (it's 2:30 here)
85 2014-12-24 00:28:57 <cfields> michagogo: no build needed
86 2014-12-24 00:29:00 <cfields> sec
87 2014-12-24 00:29:09 <michagogo> ACTION boots his VM back up
88 2014-12-24 00:31:05 <cfields> michagogo: ok, if you were headed to bed, don't worry about it
89 2014-12-24 00:31:30 <michagogo> .../me shuts it down again
90 2014-12-24 00:31:45 <cfields> haha
91 2014-12-24 00:31:45 <michagogo> Goodnight everyone
92 2014-12-24 00:31:49 <cfields> nnite
93 2014-12-24 00:32:47 <cfields> michagogo: for reference, tomorrow you can test with "copy-from-target cache/common cache"
94 2014-12-24 00:32:58 <cfields> if that puts it in the right place, the fix worked
95 2014-12-24 00:43:36 <BCB> anyone have experience with znort987's blockparser
96 2014-12-24 00:44:53 <cfields> osx tests pass, and binary starts up and appears to run sanely
97 2014-12-24 00:46:47 <wumpus> cfields: nice!
98 2014-12-24 00:48:29 <gmaxwell> BCB: ask, don't ask to ask.
99 2014-12-24 00:49:45 <sipa> cfields: i can test
100 2014-12-24 00:49:53 <sipa> i had the same problem as michagogo
101 2014-12-24 00:50:00 <wumpus> cfields: windows version also works
102 2014-12-24 00:50:13 <cfields> sipa: ah great, sec
103 2014-12-24 00:50:24 <sipa> build 0.10 + 5536?
104 2014-12-24 00:50:26 <cfields> wumpus: i'll test linux in a min, though i assume you already have as well
105 2014-12-24 00:50:47 <cfields> sipa: https://github.com/devrandom/gitian-builder/issues/78#issuecomment-68013133
106 2014-12-24 00:50:59 <sipa> oh wait, other issue
107 2014-12-24 00:51:22 <BCB> I am making on ubuntu 14.04 with gcc version 4.9.1 and it is failing with "lto-wrapper: g++ returned 1 exit status." I think this is because they bytecode files were generated with a different version of gcc. Any ideas how to correct?
108 2014-12-24 00:51:25 <cfields> sipa: well, you're welcome to test both. but the cache problem is very quick/easy, no need to build anything
109 2014-12-24 00:51:39 <cfields> (if you still have a vm up with cached contents)
110 2014-12-24 00:52:16 <sipa> yes
111 2014-12-24 00:52:17 <cfields> sipa: "libexec/copy-from-target cache/common cache"
112 2014-12-24 00:52:24 <cfields> with that change in place
113 2014-12-24 00:52:46 <cfields> BCB: that's not the real error. can you pastebin the full output?
114 2014-12-24 00:53:26 <gmaxwell> BCB: bytecode files?! that software is not written in python.
115 2014-12-24 00:54:13 <BCB> cfields: http://pastebin.com/raw.php?i=MFDTttZD
116 2014-12-24 00:54:32 <sipa> gmaxwell: he means object files, which (for lto) contain GIMPLE output (which i guess could be considered bytecode :p)
117 2014-12-24 00:54:50 <cfields> gmaxwell: i was trying to give him credit and assume he meant lto :)
118 2014-12-24 00:55:05 <cfields> hah, sipa did too
119 2014-12-24 00:55:11 <cfields> BCB: ehm, that's not our buildsystem
120 2014-12-24 00:55:21 <wumpus> just build without lto?
121 2014-12-24 00:55:27 <gmaxwell> BCB: there are no object files distributed with the software; do you just need to run make clean?
122 2014-12-24 00:55:27 <sipa> cfields: nor are we znort97
123 2014-12-24 00:55:42 <BCB> gmaxwell: yes
124 2014-12-24 00:55:52 <BCB> (and what happened to znort97)
125 2014-12-24 00:56:06 <sipa> dunno, ask him?
126 2014-12-24 00:56:11 <cfields> oh sorry, i missed that part
127 2014-12-24 00:56:46 <gmaxwell> BCB: fwiw, I', pretty sure that program will not work with 0.10 / git master.
128 2014-12-24 00:57:36 <wumpus> I still don't understand *why* you would want to build in that way, using object files left around by a different version of gcc
129 2014-12-24 00:57:46 <wumpus> just clean the tree and rebuild
130 2014-12-24 00:57:55 <BCB> gmaxwell: I figured. I'm trying it on 903 before I upgrade.
131 2014-12-24 00:58:28 <BCB> wumpus: will do. Thanks
132 2014-12-24 00:59:15 <wumpus> if make clean leaves things behind you could do 'git clean -f -x -d ' to remove all non-git files from the tree
133 2014-12-24 00:59:30 <sipa> cfields: just deleted one of the cached deps and rebuilding
134 2014-12-24 01:00:07 <BCB> gmaxwell: does blkstats.c do the same thing as bitparser?
135 2014-12-24 01:00:28 <BCB> *blockparser
136 2014-12-24 01:05:20 <sipa> wumpus: how many and what size registers does ARM (or NEON, or whatever would be reasonable to write secp256k1 assembly for) have?
137 2014-12-24 01:09:07 <midnightmagic> wait, znort's block parser won't work with 0.10? or do you mean the build whatever you're talking about?
138 2014-12-24 01:09:08 <wumpus> sipa: ARM has 16 32-bit registers (of which, say, r0-r12 can be used general purpose), NEON has 32 registers of 64 (or 2x32) wide
139 2014-12-24 01:09:57 <sipa> midnightmagic: i assume gmaxwell assumes the znort's block parser assumes that blocks are written consecutively
140 2014-12-24 01:10:09 <sipa> midnightmagic: also, how often can you use 3 levels of 'assume' in a sentence?
141 2014-12-24 01:11:27 <sipa> wumpus: so, there are two algorithms for the field multiplication, one which builds the 512-bit result first, and then reduces it, and one that does the reduction and multiplication simultaneously; on amd64 the latter is faster and on x86 the former is faster (presumably because it needs more registers)
142 2014-12-24 01:11:51 <sipa> wumpus: it may be the case that if you have 16ish 32-bit registers, the latter may be faster for arm
143 2014-12-24 01:12:06 <sipa> wumpus: so, if you ever think about implementing it, ping me :)
144 2014-12-24 01:12:25 <Luke-Jr> eh, I would think you'd want to use the NEON 64-bit registers?
145 2014-12-24 01:12:32 <sipa> duh
146 2014-12-24 01:12:34 <wumpus> sipa: though in practice, ARM toolchains usually default to a reduced instruction set called thumb, which has priority access only to the first 8 registers so there's somewhat more register pressure
147 2014-12-24 01:12:37 <sipa> if those are available
148 2014-12-24 01:12:42 <wumpus> yes, NEON would be preferable
149 2014-12-24 01:12:59 <Luke-Jr> are there any ARM CPUs that can run Bitcoin Core but don't have NEON?
150 2014-12-24 01:13:05 <wumpus> sure
151 2014-12-24 01:13:11 <sipa> 32 registers of 64 bits... that would be pretty awesome
152 2014-12-24 01:13:19 <wumpus> but those can also simply run compiled C code :)
153 2014-12-24 01:13:32 <Luke-Jr> yeah
154 2014-12-24 01:13:56 <midnightmagic> hrm.
155 2014-12-24 01:14:14 <midnightmagic> i thought znort's parser actually reads the db for indices
156 2014-12-24 01:14:42 <sipa> i doubt that
157 2014-12-24 01:14:47 <midnightmagic> maybe not.
158 2014-12-24 01:15:13 <gmaxwell> Luke-Jr: probably. neon is far from uniformly supported, far less so than even sse on x86.
159 2014-12-24 01:16:03 <wumpus> sipa: it indeed may well be that the latter is faster, especcially if there is parallelization oppertunity where 2x 32*32->64 bit multiply or mad can be useful
160 2014-12-24 01:16:11 <gmaxwell> Luke-Jr: but it's increasingly okay, and we do have plain C code for where it isn't available.
161 2014-12-24 01:18:32 <wumpus> I still don't understand how my pull could make things slower on x86 and ppc, though
162 2014-12-24 01:19:26 <sipa> TIL https://en.wikipedia.org/wiki/Branch_predication and https://en.wikipedia.org/wiki/Branch_prediction are not the same
163 2014-12-24 01:19:28 <gmaxwell> wumpus: compilers are crazy. I mean it could be as small as just moving some code around and now some branch predictior or cache line aliasing hurts it.
164 2014-12-24 01:19:51 <gmaxwell> sipa: well you mostly use x86 and not arm or itanium sooo.
165 2014-12-24 01:19:55 <wumpus> it's really demotivating to try anything
166 2014-12-24 01:20:22 <sipa> wumpus: except platform-specific asm, which is guaranteed to not hurt anything else :)
167 2014-12-24 01:20:29 <sipa> wumpus: but yeah
168 2014-12-24 01:20:43 <gmaxwell> wumpus: well that was a huge gain on arm. I dunno that we shouldn't just take it. The gain is real. Argument against taking it would be that we really ought to have arm/neon asm.
169 2014-12-24 01:21:23 <sipa> wumpus: in any case, i think over the next few years ARM will become way more important than x86, but right now, there are no real ARM users afaik
170 2014-12-24 01:21:31 <gmaxwell> arm needs a 10%-20% speedup much more than x86 is hurt by a 2% slowdown.
171 2014-12-24 01:21:46 <gmaxwell> sipa: none of bitcoin core, of everything else though.
172 2014-12-24 01:21:47 <sipa> i'll have a look at the generated x86 to try to explain the slowdown
173 2014-12-24 01:22:54 <wumpus> I didn't notice any slowdown nor speedup locally when I tried
174 2014-12-24 01:23:34 <gmaxwell> sipa: aarch64 removes the predication stuff from arm. alas.
175 2014-12-24 01:24:19 <sipa> wumpus: if neon can do 64*64->128 bit multiplications in parallel, peter dettman's multiply/reduce algorithm will be a lot faster if we can use it
176 2014-12-24 01:24:28 <cfields> sipa: thanks for testing
177 2014-12-24 01:24:35 <gmaxwell> if peeking at the asm shows GCC doing crappy things in either case, I've had reasonably good luck throwing examples at GCC developers and just letting them handle it.
178 2014-12-24 01:24:47 <sipa> i'll do that tomorrow
179 2014-12-24 01:25:05 <gmaxwell> I was just kind of depressed about the change after I found it slower on PPC.
180 2014-12-24 01:25:06 <cfields> fwiw, arm intrinsics can be very useful
181 2014-12-24 01:25:11 <sipa> anyway, are we going to announce some 0.10.0rc1?
182 2014-12-24 01:25:48 <wumpus> generally I only announce when the signed executables are uploaded
183 2014-12-24 01:25:52 <sipa> right, sure
184 2014-12-24 01:26:00 <gmaxwell> cfields: I don't think any of the intrinsics are super helpful for the things we're optimizing.
185 2014-12-24 01:26:02 <sipa> and we may some extra release notes
186 2014-12-24 01:26:09 <midnightmagic> huh. he does read the on-disk blk*.dat directly.
187 2014-12-24 01:26:09 <sipa> "0.10.0rc1 crappy christmas"
188 2014-12-24 01:26:22 <gmaxwell> (the may 'feature' we use is the carry from adders and there is a from-c way to get to it: using long long output)
189 2014-12-24 01:26:49 <wumpus> I don't see the advantage of using intrinsics here, we want manual control over register allocation
190 2014-12-24 01:27:24 <cfields> http://gcc.gnu.org/onlinedocs/gcc-4.6.1/gcc/ARM-NEON-Intrinsics.html
191 2014-12-24 01:27:26 <cfields> mm, ok
192 2014-12-24 01:27:31 <wumpus> yes, I know about those
193 2014-12-24 01:27:49 <sipa> yeah, that's my impression: the largest advantage is from a human being able to reason about long-term usage of values, and plan register allocation much better
194 2014-12-24 01:28:03 <sipa> the compiler seems to generate code that moves variables around all the time
195 2014-12-24 01:28:18 <gmaxwell> cfields: the simd stuff is even less useful, because uniformly the compiler botches the register allocation. :)
196 2014-12-24 01:28:39 <wumpus> though maybe if you have so many registers it's less urgent, still it doesn't make that much of a difference, you're still writing assembly in a slightly different syntax
197 2014-12-24 01:28:53 <gmaxwell> It's not unusual on x86 code for the same algorithim hand written out to get 2x the performance of using intrensics. :(
198 2014-12-24 01:28:54 <cfields> gmaxwell: ah. i figured the simd stuff would be the most useful here. clearly i'm out of my element :)
199 2014-12-24 01:29:01 <gmaxwell> er x86_64 even.
200 2014-12-24 01:30:19 <wumpus> gmaxwell: ppc's multiply instructions are strange, a separate instruction to multiply-and-compute-low-word and multiply-and-compute-high-word
201 2014-12-24 01:30:30 <sipa> hmm, no double-wide multiply? :(
202 2014-12-24 01:30:44 <sipa> in those arm intrinsics
203 2014-12-24 01:30:57 <sipa> ah, there's a separate mulh
204 2014-12-24 01:31:00 <wumpus> vmull_u32
205 2014-12-24 01:31:20 <sipa> ah yes!
206 2014-12-24 01:31:25 <sipa> ACTION needs sleep
207 2014-12-24 01:31:34 <midnightmagic> sipa: how does 0.10 change the blk*.dat file storage? it looks like znort just goes through the whole thing and creates a longest-chain manually.
208 2014-12-24 01:31:42 <wumpus> and vmlal
209 2014-12-24 01:31:46 <gmaxwell> sipa: the intrinsics are also not complete. Arm has a lot of funky multiples, and also crazy tricks that don't map well to intrinsics. (well it's worse for plain arm code than simd)
210 2014-12-24 01:31:46 <sipa> midnightmagic: does it deal with out-of-order blocks?
211 2014-12-24 01:32:06 <sipa> midnightmagic: as in: have a child on disk before the parent
212 2014-12-24 01:32:29 <ers35> I'm writing a bitcoin core test case that requires a private key. Is there a key I should use or any key is fine?
213 2014-12-24 01:32:38 <midnightmagic> sipa: the stuff I've read in parser.cpp *appears* to be a plain search and location-agnostic so far
214 2014-12-24 01:33:08 <wumpus> ers35: just create one
215 2014-12-24 01:34:34 <wumpus> i\
216 2014-12-24 01:34:43 <gmaxwell> ers35: 1 is a nice private key.
217 2014-12-24 01:34:46 <sipa> or 11
218 2014-12-24 01:35:15 <ers35> I'm already using 1. I tried 0 but it asserts as you previously discovered.
219 2014-12-24 01:35:33 <gmaxwell> 0 isn't a valid private key. :)
220 2014-12-24 01:35:40 <firelegend> Another question. When parsing the blockchain, how do I ensure that I am on the longest chain? I am currently checking if the previous block hash equals the hash specified in the current block header but I think this might not work as expected
221 2014-12-24 01:36:11 <sipa> firelegend: you know you're on the longest chain by computing the length (well, amount of work) of all chains, and picking the longest :)
222 2014-12-24 01:37:33 <sipa> firelegend: but if you're parsing from disk, there can be side branches, and since 0.10, blocks can be stored out-of-order
223 2014-12-24 01:38:23 <firelegend> Yes, I'm parsing from disk
224 2014-12-24 01:39:04 <sipa> firelegend: look at what the serialize script does (somewhere in contrib), it first requests the longest chain from bitcoind (through RPC), and then parses from disk, locating the the blocks that that chain consists of
225 2014-12-24 01:39:19 <sipa> sorry, linearize
226 2014-12-24 01:39:49 <sipa> firelegend: you can also just operate on the output of the linearize script, which is guaranteed to be valid, not contains holes, be fully linear, and not contain side branches
227 2014-12-24 01:41:52 <firelegend> Okay I will check them out
228 2014-12-24 01:43:35 <sipa> gmaxwell: should we mention RFC6979 in the release notes?
229 2014-12-24 01:45:30 <gmaxwell> probably. I mention we used derandomized DSA. I think at the time I wrote that text I wasn't sure if what we'd end up implementing would be exactly 6979. Feel free to add that in.
230 2014-12-24 01:45:37 <gmaxwell> sipa: working on a release note for headers first?
231 2014-12-24 01:47:06 <sipa> gmaxwell: nope, working on sleeping; but i'll write something tomorrow
232 2014-12-24 01:47:43 <wumpus> we do mention deterministic signing separately in the release notes
233 2014-12-24 01:48:22 <wumpus> hm although it's just called "improved signing security"
234 2014-12-24 01:48:25 <sipa> wumpus: yeah, i do mean RFC6979 specifically (it seems to be this buzzword that people complain about now)
235 2014-12-24 01:49:18 <wumpus> I don't think it matters that much whether the actual buzzword is used
236 2014-12-24 01:49:48 <sipa> even though I don't think that RFC6979 is all that good (it requires a dozen invocations of the hash compression function, which is actually a significant amount of time compared to the rest of signing)
237 2014-12-24 01:50:00 <sipa> ACTION .standby();
238 2014-12-24 01:50:44 <wumpus> if you create enough bitcoin transaction for signing performance to be a problem, you're probably abusing the network
239 2014-12-24 01:52:57 <gmaxwell> meh, we should just leave that as is.
240 2014-12-24 01:53:07 <gmaxwell> I'm not keen on catering to buzzwords.
241 2014-12-24 01:53:09 <wumpus> yes
242 2014-12-24 01:53:13 <wumpus> agreed
243 2014-12-24 02:04:38 <Luke-Jr> 504b6cb989b2f6967e3aa9ef1fd9ec15aaa28b060351f064323f3972cc199320 bitcoin-0.10.0-win32-setup.exe
244 2014-12-24 02:04:39 <Luke-Jr> 24a94035641cbb84d9f8d04e290e1aa008346fc8b867029717475e2991d510a6 bitcoin-0.10.0-win32.zip
245 2014-12-24 02:04:41 <Luke-Jr> 835559d963cb7d1e6fb594f15e19b4c46b811deb6b78afa25621f62af8be73ae bitcoin-0.10.0-win64-setup.exe
246 2014-12-24 02:04:42 <Luke-Jr> d89d976778c8c1fd752e1171a4221bdfa26c626b342a9d5c21570c66903ea76f bitcoin-0.10.0-win64.zip
247 2014-12-24 02:04:44 <Luke-Jr> ac9ee4a32602ccfe56980cee04bd9bc21ed79044b866f0509a2b27bfb241e1cb src/bitcoin-0.10.0.tar.gz
248 2014-12-24 02:05:37 <Luke-Jr> looks good match with michagogo
249 2014-12-24 02:18:49 <Luke-Jr> cfields: can I just update my gitian from git, to get the new stuff, or is it more complicated and I should setup from scratch?
250 2014-12-24 02:19:21 <cfields> updating to git head should work fine
251 2014-12-24 02:19:33 <sipa> gmaxwell, wumpus: ok
252 2014-12-24 02:19:54 <Luke-Jr> great
253 2014-12-24 02:20:06 <Luke-Jr> glad to not have hit any weird bumps with the old version
254 2014-12-24 02:22:02 <cfields> Luke-Jr: if you're updating and building now, you might want to take this while you're at it: https://github.com/devrandom/gitian-builder/pull/79
255 2014-12-24 02:22:46 <Luke-Jr> I don't use/trust LXC, but thanks
256 2014-12-24 02:22:52 <cfields> ah, ok
257 2014-12-24 03:00:28 <Luke-Jr> 90c5eb90ed3dc710c56edb92a93c9d6f5fb8b8243db48edbca316a4cece0d6ad bitcoin-0.10.0-linux32.tar.gz
258 2014-12-24 03:00:29 <Luke-Jr> 4b5a2c43ae9e40131e7d654f22ba99c53b1affb1da61af6c905317dab9ed80f3 bitcoin-0.10.0-linux64.tar.gz
259 2014-12-24 03:00:31 <Luke-Jr> ac9ee4a32602ccfe56980cee04bd9bc21ed79044b866f0509a2b27bfb241e1cb src/bitcoin-0.10.0.tar.gz
260 2014-12-24 03:00:51 <Luke-Jr> wumpus: your tar.gz for linux was different?
261 2014-12-24 03:08:03 <Luke-Jr> hmm
262 2014-12-24 03:08:09 <Luke-Jr> osx and only osx seems to be having DNS issues
263 2014-12-24 03:31:57 <cfields> dns?
264 2014-12-24 03:36:40 <Luke-Jr> cfields: yeah, looks like it's an issue on my end
265 2014-12-24 03:36:52 <Luke-Jr> IPv6 is apparently broken, and for some reason gitian is setting a short 10s timeout
266 2014-12-24 03:37:11 <cfields> Luke-Jr: oh.. osx build. thought you meant runtime
267 2014-12-24 03:37:28 <cfields> Luke-Jr: you can prime the cache so that gitian doesn't have to download
268 2014-12-24 03:37:37 <Luke-Jr> cfields: not with old gitian ;)
269 2014-12-24 03:38:03 <cfields> oh, right
270 2014-12-24 03:38:07 <gavinandresen> Huh. My linux32.tar.gz gitian build matches, but my linux64 does NOTâ¦
271 2014-12-24 03:38:31 <cfields> gavinandresen: you missed a few rounds of fun today. how are you building?
272 2014-12-24 03:38:38 <Luke-Jr> cfields: I need to figure out what's wrong with my IPv6 anyway
273 2014-12-24 03:38:40 <gavinandresen> cfields: VirtualBox
274 2014-12-24 03:39:02 <cfields> gavinandresen: can you paste your checksums?
275 2014-12-24 03:39:25 <gavinandresen> 90c5eb90ed3dc710c56edb92a93c9d6f5fb8b8243db48edbca316a4cece0d6ad bitcoin-0.10.0-linux32.tar.gz
276 2014-12-24 03:39:28 <gavinandresen> 41ce51aa64c96e6b149a363dafb903bdaacfbb69c0e2f95413139d1adcafa7ce bitcoin-0.10.0-linux64.tar.gz
277 2014-12-24 03:40:40 <gavinandresen> windows gitian build matches, except for the source tarball (I saw that pull request, so know it is a known issue)
278 2014-12-24 03:42:12 <gavinandresen> hmm⦠osx build doesnât match at all, EXCEPT for the source tarballâ¦
279 2014-12-24 03:42:48 <cfields> gavinandresen: sigh. can you post em somewhere?
280 2014-12-24 03:43:03 <gavinandresen> cfields: I can push them to the gitian.sigs repoâ¦
281 2014-12-24 03:43:30 <cfields> gavinandresen: i meant the tarballs, so i can compare
282 2014-12-24 03:43:46 <gavinandresen> sure, one sec
283 2014-12-24 03:45:47 <gavinandresen> cfields: uploading to http://bitcoincore.org/~gavin/0.10.0rc1/
284 2014-12-24 03:46:48 <gavinandresen> cfields: done
285 2014-12-24 03:48:04 <gavinandresen> cfields: it is very possible I should re-create my gitian VMs⦠there are a lot of extra packages installed on them that might be getting picked up.
286 2014-12-24 03:48:33 <cfields> ok
287 2014-12-24 03:57:22 <cfields> gavinandresen: for osx, 2 of the binaries themselves are different
288 2014-12-24 03:57:57 <cfields> so, probably necessary to retry with clean VMs before hunting further
289 2014-12-24 04:00:31 <gavinandresen> cfields: that might not happen for a few days, company coming for Christmas....
290 2014-12-24 04:00:51 <Techguy305> http://420weedgirls.tumblr.com/
291 2014-12-24 04:04:11 <cfields> gavinandresen: checking linux to see if maybe there's someething obvious you can uninstall
292 2014-12-24 04:13:01 <cfields> gavinandresen: nothing obvious
293 2014-12-24 04:23:20 <cfields> gavinandresen: libc-dev has been updated a few times this month. i would assume it's that
294 2014-12-24 04:23:49 <Luke-Jr> e4b60408c8917c3a1420529b6b4d9eff6fe080e4c7db86240effd0903f341dda bitcoin-0.10.0-osx-unsigned.dmg
295 2014-12-24 04:23:51 <Luke-Jr> ea4084948adba09a2ed7d7e4412a0250ed8feebc8b55076dc7077625dba1b2ac bitcoin-0.10.0-osx-unsigned.tar.gz
296 2014-12-24 04:23:52 <Luke-Jr> e9ccabda307d06a409609ef01a4585851005e0b232a79da7901a5903dfe11438 bitcoin-0.10.0-osx64.tar.gz
297 2014-12-24 04:24:35 <cfields> Luke-Jr: match
298 2014-12-24 04:25:04 <Luke-Jr> great
299 2014-12-24 04:25:47 <cfields> gavinandresen: can you install a deb?
300 2014-12-24 04:26:26 <Luke-Jr> cfields: do you mind if I move your sigs to the same dir as everyone else?
301 2014-12-24 04:27:20 <cfields> ugh, did i screw that up? pretty sure you can't move em or it'll break verification
302 2014-12-24 04:27:26 <Luke-Jr> :/
303 2014-12-24 04:28:10 <Luke-Jr> cfields: only difference to the actual files seems to be the 'release' in the yml - maybe change that and re-sign?
304 2014-12-24 04:28:18 <cfields> blah. i'll rebuild and PR
305 2014-12-24 04:35:33 <Luke-Jr> done reviewing changes to gitian since last pull, rebuilding with new version to cache stuff..
306 2014-12-24 04:41:12 <Luke-Jr> nuts, I forgot to copy the Mac bins; oh well, at least they matched XD
307 2014-12-24 05:52:41 <proserpine-> why isn't bitcoin core using poll instead of select?
308 2014-12-24 06:42:30 <Luke-Jr> proserpine-: because we support Windows?
309 2014-12-24 06:44:12 <Luke-Jr> yes, sad, I know.. :p
310 2014-12-24 06:44:27 <proserpine-> right, almost forgot :)
311 2014-12-24 06:48:24 <phantomcircuit> Luke-Jr, although would it be very hard to support both?
312 2014-12-24 06:48:33 <phantomcircuit> iirc select is only used in about 2 places
313 2014-12-24 06:49:25 <Luke-Jr> phantomcircuit: but why?
314 2014-12-24 06:49:44 <phantomcircuit> poll is iirc slightly more efficient
315 2014-12-24 06:49:56 <phantomcircuit> and supports larger numbers of connections easier
316 2014-12-24 06:50:02 <proserpine-> supposedly WSAPoll is even WORSE than select
317 2014-12-24 06:50:16 <Luke-Jr> phantomcircuit: but why care?
318 2014-12-24 06:50:19 <ajweiss> it doesn't have the fd_set 1024 limitation, but i don't think that's anywhere near an issue here
319 2014-12-24 06:50:21 <Luke-Jr> proserpine-: it's also buggy
320 2014-12-24 06:50:24 <phantomcircuit> yeah i was saying use select() on windows and poll() on linux
321 2014-12-24 06:50:49 <Luke-Jr> proserpine-: hard to see how it can perform works than select tho, Windows select uses a list for fd_set and interates it for every FD_*
322 2014-12-24 06:51:32 <proserpine-> lol list
323 2014-12-24 07:29:36 <wumpus> proserpine-: if you want to optimize bitcoin core that's fabulous, but you should start with benchmarking where performance problems are, not just clamp into something random
324 2014-12-24 07:30:25 <wumpus> I severly doubt poll versus select will make any different to anything measureable about the P2P network stack at this point
325 2014-12-24 07:30:47 <wumpus> but feel free to show differently, of course
326 2014-12-24 07:31:16 <gmaxwell> We should change to use one of poll family functions so we can scale to more FDs, would have long ago except for windows compat.
327 2014-12-24 07:31:28 <gmaxwell> It's absolutely not a performance concern.
328 2014-12-24 07:31:55 <wumpus> otherwise it's like putting more aerodynamic tires,on a boat
329 2014-12-24 07:40:27 <proserpine-> not performnace per se, but the 1024 fd_max limitation annoying
330 2014-12-24 07:40:44 <proserpine-> i can not get over 870 connections on my node
331 2014-12-24 07:41:02 <gmaxwell> you should absolutely not be running that many connections.
332 2014-12-24 07:41:20 <ajweiss> holy moly how much outgoing bandwidth do you have?
333 2014-12-24 07:41:29 <wumpus> a program to stress-test and benchmarks various aspect of the network stack so that such changes can be compared objectively would be extremely useful though
334 2014-12-24 07:41:37 <gmaxwell> Its harmful to the network to have high order nodes (and presumably you're achieving that via outbound sockets with other modifications, which is a tremendous waste of other peoples resources)
335 2014-12-24 07:42:11 <wumpus> it could be 870 incoming!
336 2014-12-24 07:42:15 <wumpus> :p
337 2014-12-24 07:43:47 <proserpine-> most of them are incoming, i dont see the harm really
338 2014-12-24 07:44:25 <gmaxwell> proserpine-: it's unlikely that they're incoming unless you're also attacking the network in some other way.
339 2014-12-24 07:44:51 <gmaxwell> (or being attacked)
340 2014-12-24 07:46:06 <gmaxwell> there would need to be approximately a million nodes without any increase in the number of nodes accepting incoming connections before we'd expect single nodes to see inbound connection counts that high.
341 2014-12-24 07:46:11 <Luke-Jr> gmaxwell: or long-running and been lucky hit by DNS seed caching
342 2014-12-24 07:46:16 <gmaxwell> (and there aren't anywhere nearly that)
343 2014-12-24 07:46:52 <gmaxwell> Luke-Jr: s'not a behavior I've observed elsewhere, if thats happening its concerning.
344 2014-12-24 07:47:22 <Luke-Jr> gmaxwell: some ISPs are really bad about caching DNS beyond TTL
345 2014-12-24 07:47:56 <gmaxwell> proserpine-: it wastes a ton of bandwidth, and gives peers less diverse views of the network. It doesn't serve any useful purpose to have connection coins an order of magnitude higher than typical.
346 2014-12-24 07:50:29 <ajweiss> it actually screws things up on the block transfer side as peers are "loyal" during initial sync and steady state relays to one peer.
347 2014-12-24 07:51:12 <ajweiss> so if you have a ton of peers but not the bandwidth to support them, there are cases where it will slow or break block transfers to your clients
348 2014-12-24 07:51:48 <proserpine-> yea that's definitely not the case, 10gbit uncapped
349 2014-12-24 07:54:20 <gmaxwell> proserpine-: thats the case for _you_ but it wastes bandwidth for your peers as well. The higher the order nodes have in the network the more traffic is wasted sending redundant data that crosses in flight.
350 2014-12-24 07:54:47 <gmaxwell> It also serves no real purpose. It doesn't aid the network.
351 2014-12-24 07:56:04 <Luke-Jr> proserpine-: mind sharing the output from getpeerinfo?
352 2014-12-24 07:58:21 <proserpine-> gmaxwell: you're absolutely right and i have nothing to add, once i compare select to poll ill cap the # of conns
353 2014-12-24 08:07:41 <Arnavion> Clearly it should use libevent !
354 2014-12-24 08:08:48 <Luke-Jr> Arnavion: libevent just uses select on Windows I think?
355 2014-12-24 08:09:32 <Arnavion> Yes
356 2014-12-24 08:10:21 <Arnavion> The Windows way is IOCP (notify on completion) but nobody wraps that in a select-like interface (notify on ready)
357 2014-12-24 08:10:34 <Arnavion> You can implement the latter with the former but not the former with the latter
358 2014-12-24 08:11:18 <wumpus> Luke-Jr: so does libuv2, looking at it
359 2014-12-24 08:12:29 <gmaxwell> Yea libevent is nice, I've liked it in the past... libuv2 is more recently popular and I've heard good things about it.
360 2014-12-24 08:12:41 <Luke-Jr> Arnavion: you have that backward, it's possible to implement IOCP with select, but not select with IOCP (unless you override all reading) - but IOCP on Windows is just implemented by synchronous threads for each (group of?) sockets anyway
361 2014-12-24 08:13:29 <Luke-Jr> so while IOCP may scale larger, it is less efficient than select
362 2014-12-24 08:13:43 <Arnavion> It's a threadpool, not a thread per socket
363 2014-12-24 08:14:39 <wumpus> it would be a nice experiment to rip out bitcoind's P2P code and change it to use one of those libraries, although as long as there is no program to quantify network performance ond'd be working blindly
364 2014-12-24 08:16:32 <Luke-Jr> "I can see clearly now 2.3 has come. I can see all the objects on the page." â©âªâ«â¬ :p
365 2014-12-24 08:17:23 <Luke-Jr> did Qt 5 get any songs? <.<
366 2014-12-24 08:20:04 <wumpus> I don't think so, they were more fun in the trolltech times
367 2014-12-24 08:33:57 <gmaxwell> so my 0.10rc1 test on the odroid crashed ... suspect it over heated after it got to the cpu heavy part (I should put a fan on it)
368 2014-12-24 08:34:42 <gmaxwell> power cycled it and it came up.. synced a thousand more blocks or two then threw a
369 2014-12-24 08:34:45 <gmaxwell> 2014-12-24 08:33:08 LevelDB read failure: Corruption: block checksum mismatch
370 2014-12-24 08:34:48 <gmaxwell> 2014-12-24 08:33:08 Corruption: block checksum mismatch
371 2014-12-24 08:34:51 <gmaxwell> and has kept going.
372 2014-12-24 08:35:14 <gmaxwell> we really cannot keep going when leveldb throws an error: thats how 0.7 boosted that BDB issue into a forking bug.
373 2014-12-24 08:35:47 <wumpus> so leveldb goes through all these paranoid checks, and we just ignore the result :-)
374 2014-12-24 08:36:12 <Luke-Jr> gmaxwell: that particular error, we might be able to, provided we then sync correctly to the right chain
375 2014-12-24 08:36:17 <gmaxwell> well mostly we don't ignore the result, but @#$@ exceptions.
376 2014-12-24 08:36:18 <gmaxwell> https://0bin.zertrin.org/paste/c7b3c9228f173b4f063700152b7b2d29d5718f32#MpWZG3gUYdQWxUxmTuyjEMArJ84tqYT8rmJs/AD4QGo=
377 2014-12-24 08:36:26 <Luke-Jr> otoh, who knows if we can detect that exact scenario reliably
378 2014-12-24 08:36:32 <CA91201> When will V0.10
379 2014-12-24 08:36:35 <CA91201> be ready for pub rlease
380 2014-12-24 08:36:47 <gmaxwell> Luke-Jr: it's just not acceptable from a design perspective we cannot be sure if any particular failure is safe or not. The only thing that is assuradly safe is to shut down.
381 2014-12-24 08:36:55 <Luke-Jr> CA91201: when it's ready. test rc1 if you want that sooner
382 2014-12-24 08:36:58 <CA91201> Also why isn't it called Version 10.
383 2014-12-24 08:37:04 <CA91201> I mean V.1
384 2014-12-24 08:37:15 <Luke-Jr> because v0.1 was years ago
385 2014-12-24 08:37:27 <CA91201> So why not v1.0
386 2014-12-24 08:37:31 <wumpus> gmaxwell: it should certainly trigger AbortNode()
387 2014-12-24 08:37:36 <CA91201> instead of v0.10 since you have v0.1
388 2014-12-24 08:37:39 <Luke-Jr> CA91201: because after 9 is 10. #bitcoin for dumb questions please
389 2014-12-24 08:38:05 <CA91201> Don't be patronizing.
390 2014-12-24 08:38:25 <wumpus> because 1.0 has no initial zero, and we use the initial zero for signaling something.
391 2014-12-24 08:38:28 <CA91201> I am asking why isn't it called Version 1, instead of Version 0.10 since Version 0.1 is already been released
392 2014-12-24 08:38:44 <Luke-Jr> 10 is also not 1
393 2014-12-24 08:39:03 <CA91201> But after 9, it should really be 1.0 instead of 0.10
394 2014-12-24 08:39:07 <wumpus> that's all, take further discussion to #bitcoin
395 2014-12-24 08:40:22 <gmaxwell> CA91201: no, this is a very common versioning mechenism (the most common in foss software today, I believe). Versions are not decimal numbers (you might have noticed there is more than one dot)
396 2014-12-24 08:40:24 <CA91201> What is the ETA for Version 1.0 (Non-Beta)?
397 2014-12-24 08:40:36 <CA91201> Will Version 1.0 be non-beta?
398 2014-12-24 08:42:47 <Luke-Jr> this is going to be a long next month. :|
399 2014-12-24 08:43:20 <CA91201> Why is that Luke-Jr
400 2014-12-24 08:43:37 <Luke-Jr> CA91201: you can ask in #bitcoin
401 2014-12-24 08:43:47 <wumpus> #bitcoin, last warning
402 2014-12-24 08:43:58 <lewellyn> i swear one of these days i'm going to number releases of something in binary.
403 2014-12-24 08:44:20 <lewellyn> version 100110 is released! get it now! many improvements since 100101!
404 2014-12-24 08:44:21 <Luke-Jr> lewellyn: Perl actually has a "version" type, which encodes it as UTF-8
405 2014-12-24 08:44:58 <lewellyn> Luke-Jr: i prefer the idea of "my numbers grow faster than chrome's" ;)
406 2014-12-24 08:46:24 <wumpus> or just use dates, it avoids any discussion
407 2014-12-24 08:46:47 <wumpus> bitcoin core 1501, bitcoin core 1507, etc
408 2014-12-24 08:46:53 <lewellyn> so... 141213... ok :D
409 2014-12-24 08:46:56 <ajweiss> BITCOIN CORE X
410 2014-12-24 08:47:02 <CA91201> How about letters?
411 2014-12-24 08:47:07 <Luke-Jr> Bitcoin is 513 years old!
412 2014-12-24 08:47:10 <gmaxwell> please move this out of this channel.
413 2014-12-24 08:47:19 <wumpus> yes, good idea
414 2014-12-24 08:50:00 <CA91201> Would it be possible to consolidate the blockchain data from 2009 to 2014 and condense the size?
415 2014-12-24 08:50:28 <gmaxwell> CA91201: please, #bitcoin
416 2014-12-24 08:51:40 <gmaxwell> sipa: so thoughts on having leveldbwrapper directly call AbortNode in HandleError? that seems somewhat safer than hoping that every use of it manages to catch the exception.
417 2014-12-24 09:42:43 <Luke-Jr> michagogo: I'm already building pruning bins for him; if you want to provide a second signature on it, let me know and I'll push my branch (it's 0.10rc1 + pruning)
418 2014-12-24 09:53:04 <wumpus> re: secp256k1, it looks like for ARM, it is generally preferred to provide .s files than to use inline assembly as you can set the architecture flags (ARM, Thumb, Neon) without having to force the entire project to be compiled with certain compiler flags like -mthumb -fpu=neon etc
419 2014-12-24 09:57:44 <wumpus> within inline assembler you have to conform to the settings of the .o file it gets compiled in, which may or may not be the ones you want, although, granted, we have control over that as these are not inline functions defined in a header but everything ends up in secp256k1.o
420 2014-12-24 10:01:11 <Luke-Jr> wumpus: also, we can do the flags for just libsecp256k1
421 2014-12-24 10:01:13 <wumpus> using -marm instead of -mthumb (generally the default) for the entire secp256k1 appears to result in a ~4% speedup for verify, although at the expense of larger code size; now going to try what happens if just the _inner functions are -marm and the rest is compiled with standard options
422 2014-12-24 10:02:03 <wumpus> Luke-Jr: sure, but secp256k1 would force those flags, it wouldn't compile without
423 2014-12-24 10:02:34 <buZz> thumb instructions arent really faster
424 2014-12-24 10:02:36 <buZz> just smaller
425 2014-12-24 10:03:15 <wumpus> buZz: but cache-wise they are preferable as they have a much larger code density, though can access less registers easily
426 2014-12-24 10:03:16 <buZz> also fyi, not all ARM cores have neon or thumb
427 2014-12-24 10:03:45 <Luke-Jr> buZz: yeah, we went over that earlier ;)
428 2014-12-24 10:03:48 <wumpus> well on cores that don't have thumb it wouldn't be the defaultk, would it
429 2014-12-24 10:03:49 <buZz> ah ok
430 2014-12-24 10:03:52 <buZz> no worries
431 2014-12-24 10:04:04 <buZz> just wanted to make sure you guys knew this
432 2014-12-24 10:07:27 <gmaxwell> wumpus: yes, actually it's preferable to have the .s in the arm format too, and there are script that translate to inputs for other toolchains. Fortunately the arm stuff has a consistent ABI so less trouble with dealing with that.
433 2014-12-24 10:08:02 <wumpus> gmaxwell: ok, great
434 2014-12-24 10:09:51 <wumpus> I've compiled with -S -O3 -marm, extracted those functions, and put them in a separate .s, and made it part of the project, now looking if everything is still sane
435 2014-12-24 10:12:19 <Luke-Jr> gmaxwell: it does? ARM has at least 2 different ABIs I thought?
436 2014-12-24 10:13:07 <buZz> EABI , OABI
437 2014-12-24 10:13:15 <wumpus> Luke-Jr: but they have the same conventions re: saving which registers, afaik?
438 2014-12-24 10:13:31 <wumpus> Luke-Jr: and passing arguments
439 2014-12-24 10:13:33 <buZz> and probably you consider armhf as seperate ABI aswell
440 2014-12-24 10:13:40 <wumpus> no, I don't.
441 2014-12-24 10:13:44 <buZz> oh
442 2014-12-24 10:13:45 <wumpus> I don't do floats
443 2014-12-24 10:13:49 <buZz> k :)
444 2014-12-24 10:14:09 <buZz> ;)
445 2014-12-24 10:14:09 <buZz> no worries, i'll borrow you a raft when floating time is here
446 2014-12-24 10:14:53 <Luke-Jr> wumpus: I don't know the details
447 2014-12-24 10:16:18 <gmaxwell> Luke-Jr: it's way more consistent than x86, fortunately.
448 2014-12-24 10:16:53 <wumpus> buZz: haha thanks, in that case I'd prefer softfloat to hardfloat though
449 2014-12-24 10:17:45 <buZz> :D lol
450 2014-12-24 10:17:50 <buZz> no houtvlot?
451 2014-12-24 10:22:04 <wumpus> Luke-Jr: although you did make me wonder what those .eabi_attribute at the top of the .s stand for
452 2014-12-24 10:27:36 <Luke-Jr> wumpus: hm, we're claiming in the release notes (IIRC) that watch-only wallets will be compatible with pruning in the future - but as of right now, the pruning code disables the wallet entirely
453 2014-12-24 10:29:43 <wumpus> Luke-Jr: hence 'future'
454 2014-12-24 10:30:11 <wumpus> that can be anywhere from tomorrow to the next million years, right :<
455 2014-12-24 10:30:11 <wumpus> the current pruning just isn't useful with the current wallet
456 2014-12-24 10:30:28 <Luke-Jr> oh well
457 2014-12-24 10:30:32 <Luke-Jr> FWIW, I'm uploading 0.10.0rc1+pruning bins to http://luke.dashjr.org/programs/bitcoin/files/bitcoind/misc/0.10.x/0.10.0rc1.autoprune/
458 2014-12-24 10:31:22 <wumpus> incidentally that's also one of the reasons why it wasn't merged for 0.10
459 2014-12-24 10:35:15 <wumpus> now that 0.10 is split off it may be a good idea to merge pruning soon, so that people can build further on it
460 2014-12-24 10:36:49 <Luke-Jr> I imagine we should get at least a few more testers with these binaries
461 2014-12-24 10:39:15 <wumpus> yes
462 2014-12-24 10:49:04 <wumpus> I've looked them up and most of the EABI attributes deal with the FP or VFP https://gist.github.com/laanwj/d8d023729614d4212d0f although there are also some like align_needed, align_preserved, enum_size, ...
463 2014-12-24 11:25:38 <goodbtc> This is the time of the year when we gather around the table with friends and family to enjoy each others company and celebrate the work of our beloved developers.
464 2014-12-24 11:25:39 <goodbtc> Thanks, gavinandresen and co!
465 2014-12-24 11:26:35 <wumpus> ok, in principle it works, but this is too much fighting with autotools, adding the same .s file for libsecp256k1_SOURCES as well as the tests_SOURCES results in "object 'src/asm/secp256k1_arm_o3.$(OBJEXT)' created both with libtool and without", unlike for c files it does not call the object file differently
466 2014-12-24 11:26:59 <wumpus> goodbtc: you're welcome
467 2014-12-24 11:35:47 <sipa> wumpus: hmm, you may want to look at an older version of libsecp, where the amd64 asm was in yasm files
468 2014-12-24 11:38:49 <wumpus> ah, it created a common lib, ok yes that'd work
469 2014-12-24 11:39:29 <sipa> goodbtc: you're welcome :)
470 2014-12-24 11:40:10 <goodbtc> sipa, you are an workaholic - thank you for that!
471 2014-12-24 11:40:10 <sipa> yasm was annoying as a dependency, though
472 2014-12-24 11:44:25 <wumpus> I use gnu assembler not yasm, so you'd expect this work better
473 2014-12-24 11:44:58 <sipa> right
474 2014-12-24 11:45:00 <wumpus> somehow the logic for automatic renaming of objects doesn't trigger, I wonder if there's a way to do so manually, the common library looks annoying
475 2014-12-24 11:45:17 <wumpus> (ie, everything that links against secp256k1 also needs to link against that)
476 2014-12-24 11:55:10 <sipa> wumpus: no, that common library was linked into libsecp256.a iirc
477 2014-12-24 11:57:38 <wumpus> sipa: I'd expected that to only work when compiling the end product into a shared library, as there is a linker step, but possibly autotools is smart enough to compile static libs as well, let's see
478 2014-12-24 11:57:48 <wumpus> s/compile/merge/
479 2014-12-24 11:59:21 <sipa> yeah, libtool magic i suppose
480 2014-12-24 11:59:31 <wumpus> yes, it works
481 2014-12-24 12:03:41 <sipa> so what are you doing now, just compiling the mul/sqr code separately to asm in some other mode, and linking that in as asm?
482 2014-12-24 12:07:21 <wumpus> yes, that's my starting point
483 2014-12-24 12:08:49 <wumpus> I've compiled secp256k1 with -O3 -S -marm to get assembly output, extracted those field10x26 _inner functions, and use them as external .s assembly input
484 2014-12-24 12:13:25 <wumpus> this works as expected, and gives the same speedups as #170, even slightly better because of -marm (where this cross compiler at least defaults to -mthumb)
485 2014-12-24 13:37:44 <wumpus> the gcc-generated code does a lot of useless shuffling between the registers, the stack and input arguments, there is certainly still scope for optimization
486 2014-12-24 13:56:27 <sipa> wumpus: yeah, expected
487 2014-12-24 14:43:28 <hearn> sipa: do you know how to invoke the rpc regtests, off hand? it doesn't seem to be documented in the readme and just running them with python doesn't work
488 2014-12-24 14:44:06 <sipa> hearn: yes, just run them :)
489 2014-12-24 14:44:11 <sipa> ah
490 2014-12-24 14:44:17 <sipa> what fails if you just run them with python?
491 2014-12-24 14:46:44 <hearn> sipa: http://pastebin.com/k95MLrHG
492 2014-12-24 14:47:55 <hearn> ah
493 2014-12-24 14:48:05 <hearn> it expects to be run "python forknotify.py" in the same directory
494 2014-12-24 14:48:17 <hearn> i should have guessed that'd be the issue, sorry for the noise
495 2014-12-24 14:49:16 <sipa> that should work, afaik
496 2014-12-24 14:49:54 <sipa> works here
497 2014-12-24 14:50:03 <sipa> one requirement is that you have a wallet compiled into bitcoind
498 2014-12-24 14:51:18 <sipa> and i again i misread, good to hear it works :)
499 2014-12-24 14:54:23 <hearn> sipa: thanks, and merry christmas :)
500 2014-12-24 15:48:31 <sipa> hearn: thanks, but will be glad when it's over :)
501 2014-12-24 15:48:45 <hearn> christmas?
502 2014-12-24 15:48:59 <hearn> btw the "cannot build gitian due to no network connectivity" issue still seems to be present on the 0.10 branch
503 2014-12-24 15:49:38 <sipa> yes, but you can fetch the sources outside of gitian
504 2014-12-24 15:50:28 <sipa> do a make download in the depends directory, and then copy the inputs to gitian-builder/cache/common directory
505 2014-12-24 15:50:39 <hearn> right
506 2014-12-24 15:50:51 <hearn> there is a command listed in the new docs but it's listed as optional .... i'm trying it now
507 2014-12-24 15:58:04 <wumpus> hearn: it's not a problem with the 0.10 branch, but with gitian
508 2014-12-24 15:58:13 <wumpus> (or at least, its network setup)
509 2014-12-24 15:58:17 <hearn> ok
510 2014-12-24 15:59:43 <wumpus> also cfields made a fix in gitian yesterday, not sure if it's already been merged, but it caused depends caching to go wrong in some cases resulting in unncessary rebuilds
511 2014-12-24 16:00:10 <wumpus> https://github.com/devrandom/gitian-builder/pull/79
512 2014-12-24 16:00:50 <wumpus> not merged yet
513 2014-12-24 16:13:30 <michagogo> Specifically, when using LXC it caused the cache output to be copied into cache/cache/<name> instead of cache/<name>
514 2014-12-24 16:14:18 <michagogo> (with the reading from the cache into the target done correctly, from cache/*, resulting in the files not being where the descriptor expects them)
515 2014-12-24 16:20:54 <sipa> ;;blocks
516 2014-12-24 16:20:55 <gribble> 335698
517 2014-12-24 18:30:16 <BCB> wumpus gmaxwell "make clean" worked on the file yesterday. Just got around to trying it. Thank you and have a nice holiday.
518 2014-12-24 20:40:57 <devrandom> cfields: I merged https://github.com/devrandom/gitian-builder/pull/79 just now, thank you
519 2014-12-24 20:42:28 <michagogo> devrandom: thanks
520 2014-12-24 20:42:40 <michagogo> ACTION is glad he realized what was happening pretty quickly
521 2014-12-24 21:44:28 <XDOGEX> hi
522 2014-12-24 21:44:32 <souljah1h-Lucky8> hi
523 2014-12-24 21:57:48 <Luke-Jr> cfields: why are there 3 fairly-different OSX outputs?
524 2014-12-24 22:03:15 <Luke-Jr> wumpus: is it a bug that our Linux binaries don't use 64bit_asm for libsecp256k1?
525 2014-12-24 22:08:23 <Luke-Jr> ditto for OSX bins
526 2014-12-24 22:08:32 <XDOGEX> is rain gift ??
527 2014-12-24 22:12:28 <gmaxwell> Luke-Jr: I thought that was intentional; we didn't update libsecp256k1 to one that doesn't require yasm before release.
528 2014-12-24 22:12:52 <gmaxwell> Luke-Jr: it doesn't matter since performance isn't really interesting for signing for us.
529 2014-12-24 22:14:47 <Luke-Jr> ok
530 2014-12-24 22:19:15 <Luke-Jr> note to self: KVM in 32-bit mode can't handle -j4 on OSX building
531 2014-12-24 22:34:54 <cfields> Luke-Jr: yea, gcc is very picky in gitian
532 2014-12-24 22:35:08 <cfields> our version seems especially crashy when low on mem
533 2014-12-24 22:35:25 <Luke-Jr> ACTION wonders why 32-bit KVM is limited to 2 GB rather than 4 GB
534 2014-12-24 22:36:31 <cfields> Luke-Jr: the osx outputs are all intended. one is an unsigned dmg, usable for anyone who wants a fully deterministic one, but will have to put up with gatekeeper complaints. the second is a set of binaries in the same structure as the rest, bin/ llib/, etc. for users who want to run bitcoin-cli, bitcoind, etc
535 2014-12-24 22:37:28 <cfields> the third is a package that contains scripts needed to create a detached signature from osx. that payload can be re-combined with the original output in gitian to create a deterministic signed dmg
536 2014-12-24 22:37:49 <cfields> (fully deterministic if you have the priv key, of course :)
537 2014-12-24 22:39:40 <Luke-Jr> cfields: oh, so we can't go dmg -> signed dmg
538 2014-12-24 22:39:56 <cfields> no. it's not the dmg that's signed, it's the contents
539 2014-12-24 22:40:49 <Luke-Jr> right, I meant unpack, sign, repack
540 2014-12-24 22:41:04 <Luke-Jr> so no point distributing the osx64 file?
541 2014-12-24 22:41:41 <cfields> the final dmg, and the tarball with binaries should be distributed
542 2014-12-24 22:41:49 <cfields> (same as we do with the .exe installer and the zip)
543 2014-12-24 22:42:38 <Luke-Jr> cfields: there are two tarballs, I'm trying to clarify which one is useless to share
544 2014-12-24 22:42:39 <cfields> uhmm, i don't remember off the top of my head why i didn't just unpack the unsigned dmg and sign it
545 2014-12-24 22:42:52 <cfields> the -unsigned.tar.gz
546 2014-12-24 22:43:15 <Luke-Jr> oh, unexpected âº
547 2014-12-24 22:46:10 <cfields> Luke-Jr: ah, right, the -unsigned.tar.gz is needed again as an input to gitian for signing. i believe it presented some problem to unpack a dmg, was easier to leave as a tarball
548 2014-12-24 22:46:23 <cfields> s/signing/attaching the detached sig/
549 2014-12-24 22:46:32 <Luke-Jr> i c
550 2014-12-24 22:46:58 <cfields> probably doable though, i just went the easiest way for the sake of getting it done
551 2014-12-24 22:47:37 <Luke-Jr> might as well if it's easy; I was just confused by the 3 files
552 2014-12-24 22:52:51 <cfields> reasonable enough
553 2014-12-24 22:53:13 <cfields> off for family time, cya
554 2014-12-24 22:55:33 <Luke-Jr> ttyl
555 2014-12-24 22:57:39 <XDOGEX> good night
556 2014-12-24 23:17:09 <gmaxwell> our memory usage is really crazy high.
557 2014-12-24 23:51:55 <ers35> Any interest in a feature to load a new bitcoind executable without losing incoming and outgoing connections?
558 2014-12-24 23:59:44 <gmaxwell> ers35: I don't know why anyone would want that... also you need to restart connections to flush peers knoweldge of what we already know.