1 2013-09-02 00:17:18 <gmaxwell> phantomcircuit: there are users on some version of debian who are reporting signature validation errors.
2 2013-09-02 00:18:06 <gmaxwell> my best theory is that some behavior difference in some dependency (or the compiler) is exposing an issue, but I have no idea if the issue is in the bitcoin code or otherwise.
3 2013-09-02 00:19:54 <phantomcircuit> oh
4 2013-09-02 00:19:56 <phantomcircuit> hmm
5 2013-09-02 00:20:02 <phantomcircuit> gmaxwell, what version of debian?
6 2013-09-02 00:20:52 <TheLordOfTime> gmaxwell: i've heard that as well, one of those bugs trickled down into Ubuntu but was "Invalid"'d after a while because it was proven wrong.
7 2013-09-02 00:21:20 <gmaxwell> phantomcircuit: current sid/unsable
8 2013-09-02 00:21:28 <gmaxwell> er unstable.
9 2013-09-02 00:21:58 <jcorgan> hmm, ipsec vpn doesn't work over hotel network, but Tor does :)
10 2013-09-02 00:21:58 <jgarzik> gmaxwell, <self interested tangent> another reason to switch to sipa's libsecp256k1
11 2013-09-02 00:22:04 <gmaxwell> TheLordOfTime: sometimes people get just random corruption from bad hardware, so there are some bogus reports too... but this issue on debian is quite repeatable for some people and doesn't appear to be hardware.
12 2013-09-02 00:22:10 <jgarzik> has the side effect of embedded our signature checking code in repo
13 2013-09-02 00:22:18 <jgarzik> no OpenSSL differences, fewer compiler variations
14 2013-09-02 00:22:32 <jgarzik> no Fedora problems
15 2013-09-02 00:22:43 <jgarzik> *embedding
16 2013-09-02 00:22:49 <gmaxwell> jgarzik: yup. It's a ton of wins. I think we will eventually.
17 2013-09-02 00:23:40 <TheLordOfTime> gmaxwell: got a bug number?
18 2013-09-02 00:24:25 <gmaxwell> TheLordOfTime: the best report is here: https://github.com/bitcoin/bitcoin/issues/2726
19 2013-09-02 00:26:40 <TheLordOfTime> gmaxwell: ah. i see. that hasn't yet reached the debian bug trackers, so maybe it should be added to the debian BTS as a bug.
20 2013-09-02 00:26:52 <TheLordOfTime> gmaxwell: although i see something more problematic... who maintains the package for bitcoin in Debian?
21 2013-09-02 00:27:53 <gmaxwell> TheLordOfTime: well bitcoin in debian has other problems, e.g. they're ripping out our embeded leveldb and swapping in a questionable patched up one, and also applying a number of other patches that no one from bitcoin core has reviewed.
22 2013-09-02 00:28:05 <TheLordOfTime> gmaxwell: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=718272 <-- related?
23 2013-09-02 00:28:17 <gmaxwell> But my understanding is that these issues on unstable happen to people running our source from git without the patches too.
24 2013-09-02 00:28:19 <TheLordOfTime> looks like one of a billion unfixed issues
25 2013-09-02 00:28:43 <TheLordOfTime> gmaxwell: okay, that means Ubuntu has the bugs then, because `bitcoin` in Ubuntu by default is pulled from Debian
26 2013-09-02 00:28:56 <TheLordOfTime> ACTION wouldn't mind replacing the version in Debian with the PPA versions that're maintained by someone from ehre
27 2013-09-02 00:28:58 <gmaxwell> TheLordOfTime: uhg. "Don't do that."
28 2013-09-02 00:29:05 <TheLordOfTime> gmaxwell: heh
29 2013-09-02 00:29:24 <TheLordOfTime> gmaxwell: there's enough there to get it removed AFAICT from Debian :/
30 2013-09-02 00:29:28 <gmaxwell> Bluematt had been maintaining a ubuntu ppa ... we should probably update the gitian process to write out a ppa too, in addition to our normal binaries. But I know nothing of ppas.
31 2013-09-02 00:30:11 <gmaxwell> TheLordOfTime: its actually unusable in unstable, at least on multicore systems, it just dies syncing the chain. Gotta love completely untested packages. :P
32 2013-09-02 00:30:43 <gmaxwell> (not that bitcoin-test detects the problem here, AFAIK, but the debian package doesn't even run it on non-crosscompiled builds)
33 2013-09-02 00:33:19 <TheLordOfTime> gmaxwell: there's a Debian team that maintains this apparently
34 2013-09-02 00:33:47 <phantomcircuit> er
35 2013-09-02 00:33:48 <phantomcircuit> https://github.com/bitcoin/bitcoin/issues/2726#issuecomment-21622639
36 2013-09-02 00:33:48 <TheLordOfTime> gmaxwell: if you can support me with the claim I might be able to get it removed from Debian and Ubuntu. I'm checking the Ubuntu packages now, but they're outdated and I don't have the latest from Debian
37 2013-09-02 00:34:00 <TheLordOfTime> i'd like upstream dev support on that removal request too
38 2013-09-02 00:34:04 <phantomcircuit> has anybody considered that he ran bitcoind -daemon and not ./bitcoind -daemon ?
39 2013-09-02 00:34:05 <TheLordOfTime> then maybe assign BlueMatt as the sole maintainer
40 2013-09-02 00:34:42 <gmaxwell> phantomcircuit: good question! can you ask?
41 2013-09-02 00:35:50 <phantomcircuit> yeah
42 2013-09-02 00:45:20 <TheLordOfTime> gmaxwell: can you do me a favor and compile a list of all the issues reported of the package not working in Debian? I'm going to partly use that and my own testing to try and get the package removed/blacklisted in Ubuntu, and then possibly Debian
43 2013-09-02 00:45:29 <TheLordOfTime> gmaxwell: at least, reported on the github tracker.
44 2013-09-02 00:45:34 <TheLordOfTime> ACTION is doing his own testing too
45 2013-09-02 00:46:04 <Belxjander> TheLordOfTime: what package?
46 2013-09-02 00:46:14 <TheLordOfTime> Belxjander: 'bitcoin'
47 2013-09-02 00:46:15 <gmaxwell> I'd really like to understand the actual issue first. Otherwise I would have been gun-ho on getting it removed already.
48 2013-09-02 00:46:26 <TheLordOfTime> gmaxwell: but it does fail to sync the blockchain?
49 2013-09-02 00:46:35 <TheLordOfTime> in comparison to, say, BlueMatt's PPA version which I KNOW syncs the chain
50 2013-09-02 00:46:51 <TheLordOfTime> Belxjander: specifically, there's reports debian's version is (1) butchered, and (2) unusable
51 2013-09-02 00:47:04 <TheLordOfTime> Belxjander: and as long as that remains the case, I can at least get it removed in ubuntu (which may trickle down to Mint)
52 2013-09-02 00:47:18 <TheLordOfTime> or try to get it removed
53 2013-09-02 00:47:46 <gmaxwell> TheLordOfTime: bluematt's ppa works fine indeed. I don't know if anything currently fails on ubuntu. I know it fails on debian unstable. I'd like to know if the bug is ours or theirs, and at least current reports (ignoring phantom circuit's theory) suggest that removing the package doesn't fix it. (because it's reported even binaries compiled directly from git also fail)
54 2013-09-02 00:48:03 <gmaxwell> (Bluematt's ppa would work simply because it's mostly static, iirc)
55 2013-09-02 00:48:03 <TheLordOfTime> gmaxwell: the failure to use hte blockchain?
56 2013-09-02 00:48:05 <Belxjander> TheLordOfTime: I find most non-source packages get butchered...and a few source based ones too
57 2013-09-02 00:48:18 <TheLordOfTime> Belxjander: yeah, well, in ubuntu there's methods that i can use to blacklist the Debian package
58 2013-09-02 00:48:25 <TheLordOfTime> and then we can force everyone to use BlueMatt's PPA which works
59 2013-09-02 00:48:36 <Belxjander> good
60 2013-09-02 00:48:55 <phantomcircuit> upgrading a wheezy vm to sid now
61 2013-09-02 00:49:07 <phantomcircuit> gmaxwell, the invalid block stuff should contain a ton more information
62 2013-09-02 00:49:12 <gmaxwell> and yea, debian has no business modifying itâ but at the same time, I don't want to go flaming them too much on that until I understand the failure.
63 2013-09-02 00:49:17 <TheLordOfTime> phantomcircuit: i'm downloading Ubuntu Saucy, which is the first release it'd get removed from
64 2013-09-02 00:49:24 <TheLordOfTime> gmaxwell: i can poke the maintainer team and ask why they butchered the code
65 2013-09-02 00:49:36 <TheLordOfTime> they ahve a mailing list http://packages.qa.debian.org/b/bitcoin.html <-- useful
66 2013-09-02 00:49:37 <phantomcircuit> it's a rare enough and important enough occurrence that it should be lots of info
67 2013-09-02 00:49:44 <phantomcircuit> even if it's mostly nonsense to users
68 2013-09-02 00:49:46 <gmaxwell> phantomcircuit: it actually gives what we would need to know here, a signature validation failed. The specific failure point is non-determinstic, so it's unlikely that looking at the signature would help.
69 2013-09-02 00:50:03 <gmaxwell> phantomcircuit: my biggest complaint is that it gets burried in the logs, debug.log is full of worthless noise.
70 2013-09-02 00:50:38 <gmaxwell> In particular debug log reports a bunch of "ERROR" for things that are totally normal, harmless, and expected.. so when users have issues they fret about the error lines and also report them rather than the important ones.
71 2013-09-02 00:50:43 <phantomcircuit> gmaxwell, iirc there are a number of ways VerifySignature can fail though
72 2013-09-02 00:51:23 <gmaxwell> true, what we should at least do is for signatures in blocks log the data that was being validated.
73 2013-09-02 00:51:37 <gmaxwell> (outside of blocks is a problem because logging a bunch there is a dos vector :( )
74 2013-09-02 00:51:44 <TheLordOfTime> gmaxwell: phantomcircuit: what exactly do i look for in the debug logs to confirm failure to sync?
75 2013-09-02 00:51:51 <TheLordOfTime> at least, part of a string that i can pull as evidence
76 2013-09-02 00:51:54 <gmaxwell> TheLordOfTime: INVALID
77 2013-09-02 00:52:10 <gmaxwell> (I might have the case wrong there, is it Invalid? will have to look)
78 2013-09-02 00:52:19 <TheLordOfTime> gmaxwell: okay. and 'VALID' would be the data to look for in, say, BlueMatt's PPA version or a source-built version?
79 2013-09-02 00:52:24 <TheLordOfTime> (i'm going to test both of those to confirm)
80 2013-09-02 00:52:31 <TheLordOfTime> (so a total of 3 tests)
81 2013-09-02 00:53:07 <gmaxwell> no, valid doesn't get logged. the thing to look for is it just making it up to the current height. e.g. "blocks" : 255591,
82 2013-09-02 00:53:44 <gmaxwell> you can probably make the broken ones fail faster by running with -checkpoints=0
83 2013-09-02 00:54:26 <phantomcircuit> lol who actually reads the dist-upgrade changelogs
84 2013-09-02 00:54:38 <phantomcircuit> seriously im dist-upgrade'ing to unstable
85 2013-09-02 00:54:42 <phantomcircuit> im not reading that stuff
86 2013-09-02 00:56:40 <tgs3> base-system notice (urgency: HIGH)
87 2013-09-02 00:56:46 <tgs3> * installing this packages can make you gay
88 2013-09-02 00:57:16 <phantomcircuit> tgs3, fabulous
89 2013-09-02 00:57:26 <phantomcircuit> wait wat
90 2013-09-02 00:57:37 <tgs3> ACTION puts TODO to do this for real when occasion arises
91 2013-09-02 00:58:38 <phantomcircuit> tgs3, bump the version number of your LICENSE by a subversion and add a clause requiring all distributors to love their neighbors
92 2013-09-02 00:58:51 <phantomcircuit> proceed to laugh when debian finds out and cant figure out what to do
93 2013-09-02 00:59:10 <tgs3> heh
94 2013-09-02 00:59:18 <gmaxwell> "no no, it's all fine because on a DESERT ISLAND you have no neighbors!"
95 2013-09-02 00:59:53 <tgs3> except toad. hypno toad
96 2013-09-02 01:07:43 <gmaxwell> oh #@$@ is debian unstable now shipping random bitcoin from git?$@$#@$@#$?#@$
97 2013-09-02 01:08:47 <phantomcircuit> lol are they?
98 2013-09-02 01:10:42 <phantomcircuit> gmaxwell, correct me if im wrong but uh
99 2013-09-02 01:10:51 <phantomcircuit> 0.8.3-2 is not a release
100 2013-09-02 01:11:08 <gmaxwell> phantomcircuit: https://bitcointalk.org/index.php?topic=286013.msg3060553
101 2013-09-02 01:11:28 <gmaxwell> seems it's just a user compiling from git themselves and I was misreading a bit, I asked to clarify.
102 2013-09-02 01:12:14 <gmaxwell> phantomcircuit: and right debian is shipping some patched up stuff. Which is bad but may be unrelated to the issues we're having there. (super bad considering whomever is packaging it is clearly not even testing it)
103 2013-09-02 01:16:12 <TheLordOfTime> phantomcircuit: 0.8.3-2 = 0.8.3 + debian packaging + second revision by Debian
104 2013-09-02 01:16:41 <phantomcircuit> * Use included leveldb instead of system library. See debian/README.source
105 2013-09-02 01:16:43 <TheLordOfTime> phantomcircuit: debian packaging doesn't accept just the version number. 0.8.3 is the base "version" that is either from upstream or based on upstream, and -# is the revision number, -1 being initial, -2 being initial + 1 revision...
106 2013-09-02 01:16:44 <TheLordOfTime> etc.
107 2013-09-02 01:16:48 <phantomcircuit> seems like they're cherrypicking stuff
108 2013-09-02 01:16:53 <TheLordOfTime> phantomcircuit: they have to for revisions
109 2013-09-02 01:16:58 <TheLordOfTime> that's Debian / Ubuntu bugfixing policy
110 2013-09-02 01:17:10 <TheLordOfTime> until there's a new version that includes the fixes in which case unstable gets that
111 2013-09-02 01:17:19 <TheLordOfTime> but Ubuntu isn't fixable like Debian is, if it's broken it'll stay broken
112 2013-09-02 01:17:38 <phantomcircuit> TheLordOfTime, i dont believe anything fixed in leveldb had anythign to do with linux
113 2013-09-02 01:17:42 <TheLordOfTime> also, if ANY of you have bugtracker links that have any Debian or Ubuntu relation, feel free to share them with me.
114 2013-09-02 01:17:43 <phantomcircuit> iirc that was all intended to fix OSX
115 2013-09-02 01:17:49 <TheLordOfTime> phantomcircuit: *shrugs* it's Debian. i don't understand themn
116 2013-09-02 01:17:51 <phantomcircuit> so a debian bug fix for an OS X issue
117 2013-09-02 01:18:22 <TheLordOfTime> phantomcircuit: i can report that on the BTS if (a) that's the issue and (b) you show me the leveldb upstream info about that.
118 2013-09-02 01:18:28 <TheLordOfTime> maybe THEN it might work
119 2013-09-02 01:18:33 <TheLordOfTime> assumign this IS the source of the problems.
120 2013-09-02 01:19:17 <phantomcircuit> TheLordOfTime, im just going to killall -9 bitcoind randomly for a few hours and see what i can break
121 2013-09-02 01:19:28 <TheLordOfTime> (still, Debian butchers the code so much I wouldn't mind forcing Ubuntu people to use BlueMatt's PPA which works)
122 2013-09-02 01:19:43 <gribble> 255598
123 2013-09-02 01:19:43 <phantomcircuit> ;;bc,blocks
124 2013-09-02 01:19:46 <TheLordOfTime> phantomcircuit: feel free to share results if you want.
125 2013-09-02 01:20:09 <TheLordOfTime> preferably somewhere public other than IRC reports.
126 2013-09-02 01:20:10 <TheLordOfTime> :P
127 2013-09-02 01:21:09 <TheLordOfTime> phantomcircuit: any bugs in Debian, please report them on the Debian BTS
128 2013-09-02 01:21:18 <TheLordOfTime> because that DESPERATELY needs the help.
129 2013-09-02 01:21:27 <TheLordOfTime> phantomcircuit: then i can further use that as evidence towards removal.
130 2013-09-02 01:22:22 <gmaxwell> careful, you don't want to inspire more bullshit patches on debian.
131 2013-09-02 01:22:38 <gmaxwell> I'd take a bet that if you post to much they'll monkey patch it to deactivate parallel signature validation.
132 2013-09-02 01:22:57 <gmaxwell> s/to/too/
133 2013-09-02 01:26:06 <TheLordOfTime> gmaxwell: and then it'll fully break but by that point I'll have had the package removed from autosync with Debian on Ubuntu
134 2013-09-02 01:26:10 <TheLordOfTime> so... my half will be fixed :p
135 2013-09-02 01:26:24 <TheLordOfTime> fixed by E: No Longer Included In Ubuntu
136 2013-09-02 01:27:31 <phantomcircuit> heh
137 2013-09-02 01:27:50 <phantomcircuit> processing blocks faster than tail -f can read from debug.log
138 2013-09-02 01:28:38 <gmaxwell> yea, I think early in the chain the logging actually slows it down now.
139 2013-09-02 01:29:05 <gmaxwell> I think I'd like to move most of the non-critical failure logging into an in-memory ringbuffer.
140 2013-09-02 01:29:06 <TheLordOfTime> gmaxwell: CLI argument -debug enables debug logging?
141 2013-09-02 01:29:50 <gmaxwell> TheLordOfTime: yes, I suggest technical people run with logtimestamps=1 debug=1 (which also turns on a few more asserts)
142 2013-09-02 01:30:02 <gmaxwell> debug also disables the autolog rotation.
143 2013-09-02 01:30:46 <gmaxwell> If we used only in-memory logging we could also log more potentially privacy violating data.. and on crash save the end of that log in out dying gasp.
144 2013-09-02 01:31:09 <gmaxwell> e.g. so you'd actually learn which @#$@ peer gave you the poison pill transaction that killed your node.
145 2013-09-02 01:32:36 <TheLordOfTime> heh
146 2013-09-02 01:32:49 <TheLordOfTime> gmaxwell: there should be a global blacklist of nodes that do stupid stuff like that IMO
147 2013-09-02 01:32:57 <TheLordOfTime> but of course, that'll be harder to implement, maybe, than it sounds.
148 2013-09-02 01:33:30 <gmaxwell> ...
149 2013-09-02 01:33:37 <gmaxwell> "Global blacklist"
150 2013-09-02 01:33:46 <TheLordOfTime> ... ehehehe, apparently someone heard i'm on the warpath to remove bitcoin from Ubuntu and Debian and is asking questions in priv...
151 2013-09-02 01:34:16 <TheLordOfTime> gmaxwell: a list of "bad nodes" on IPs or proxies, but that's like having DNSBLs: they're constantly changing.
152 2013-09-02 01:34:21 <gmaxwell> pointless in any case, knowing the peer is useful so someone else can go and try to replicate the behavior.
153 2013-09-02 01:34:23 <TheLordOfTime> mhm
154 2013-09-02 01:34:49 <gmaxwell> thats all, if the peer is actually malicious it'll be some ever shifting tor node or 0wned host, blacklisting has basically no value for us.
155 2013-09-02 01:35:14 <TheLordOfTime> mhm
156 2013-09-02 01:35:16 <gmaxwell> and potential costs too. Instead we just accept that the world is a hostile place and build software that works regardless.
157 2013-09-02 01:35:33 <TheLordOfTime> phantomcircuit: you're testing sid right?
158 2013-09-02 01:35:38 <TheLordOfTime> aka unstable
159 2013-09-02 01:38:16 <TheLordOfTime> phantomcircuit: share your results from testing Debian, because I'd rather *not* dump more disk space on this temporary system for another VM
160 2013-09-02 01:41:38 <phantomcircuit> shit
161 2013-09-02 01:41:46 <phantomcircuit> i managed to install this snapshot on the wrong drive
162 2013-09-02 01:41:51 <phantomcircuit> out of disk space
163 2013-09-02 01:41:52 <phantomcircuit> >.>
164 2013-09-02 01:43:30 <Luke-Jr> gmaxwell: Debian should just package some official release as-is and pull stable backports >.>
165 2013-09-02 01:43:53 <TheLordOfTime> phantomcircuit: lol
166 2013-09-02 01:44:08 <TheLordOfTime> Luke-Jr: it's Debian, they butcher everything.,
167 2013-09-02 01:44:22 <Luke-Jr> sigh
168 2013-09-02 01:44:23 <TheLordOfTime> the only nonbutchered package i think is 'nginx' because the code matches upstream to a tee
169 2013-09-02 01:44:30 <TheLordOfTime> except for patches, which are made from upstream anyways
170 2013-09-02 01:44:31 <TheLordOfTime> so...
171 2013-09-02 01:44:41 <TheLordOfTime> oh and the 'znc' package :p
172 2013-09-02 01:44:45 <TheLordOfTime> (same situation)
173 2013-09-02 01:45:03 <gmaxwell> They don't even submit their patches unstream, pretty horrifying.
174 2013-09-02 01:45:13 <Luke-Jr> Debian wants me to do whatever to maintain their bfgminer package
175 2013-09-02 01:45:15 <TheLordOfTime> phantomcircuit: seriously though share your results, if you want you can put your results on github or something and share that info with me
176 2013-09-02 01:45:20 <gmaxwell> I wonder if I give them a patch that backdoors bitcoin if they'll take it?
177 2013-09-02 01:45:23 <Luke-Jr> maybe I should deal with that, then maintain bitcoind/bitcoin-qt for them as well
178 2013-09-02 01:45:38 <Luke-Jr> gmaxwell: you'd want to do it under a pseudonym
179 2013-09-02 01:45:43 <gmaxwell> sure.
180 2013-09-02 01:45:51 <TheLordOfTime> Luke-Jr: if you took over bfgminer package in Debian, think you could let me borrow the fruits of your labor and make it be included in Ubuntu correctly?
181 2013-09-02 01:46:00 <TheLordOfTime> because as it stands Debian needs to be cleaned with fire...
182 2013-09-02 01:46:06 <TheLordOfTime> no offense to Debian users, of course.
183 2013-09-02 01:46:12 <Luke-Jr> TheLordOfTime: won't adding it to Debian automatically add it to Ubuntu?
184 2013-09-02 01:46:21 <Belxjander> Luke-Jr: no...
185 2013-09-02 01:46:23 <TheLordOfTime> Luke-Jr: not right now, i'd have to request a sync for the dev version
186 2013-09-02 01:46:32 <TheLordOfTime> Luke-Jr: it wouldn't be included at this point until 14.04
187 2013-09-02 01:46:35 <Luke-Jr> TheLordOfTime: the Debian 'bitcoin' team actually already made source packages for bfgminer, they just want me to do some final touches I'm not really sure what I think
188 2013-09-02 01:46:35 <TheLordOfTime> because autosync is off now
189 2013-09-02 01:46:49 <TheLordOfTime> Luke-Jr: tell them you want to take over sole maintainershipo
190 2013-09-02 01:46:52 <TheLordOfTime> because it's your code
191 2013-09-02 01:46:55 <TheLordOfTime> then make the package actually work
192 2013-09-02 01:46:59 <Belxjander> Luke-Jr: Ubuntu pulls a lot from debian but does thier own post-butcher-carcass cleaning
193 2013-09-02 01:47:04 <Luke-Jr> the bfgminer package they have should work..
194 2013-09-02 01:47:14 <TheLordOfTime> while you're there tell the Debian bitcoin team someone says for them to shoot themselves.
195 2013-09-02 01:47:16 <Luke-Jr> or you mean bitcoind?
196 2013-09-02 01:47:32 <TheLordOfTime> Luke-Jr: "they" as in Debian or "they" as in ubuntu?
197 2013-09-02 01:47:39 <Luke-Jr> Debian
198 2013-09-02 01:47:43 <TheLordOfTime> ah
199 2013-09-02 01:47:48 <Luke-Jr> the only bfgminer for Ubuntu afaik is the one unit3 does in a PPA
200 2013-09-02 01:47:58 <TheLordOfTime> Luke-Jr: and what's autosynced from Unstable
201 2013-09-02 01:48:08 <Luke-Jr> TheLordOfTime: ?
202 2013-09-02 01:48:08 <TheLordOfTime> Luke-Jr: so if the packages in Unstable right now are broken, they'll be broken in Ubuntu 13.10
203 2013-09-02 01:48:18 <Luke-Jr> I'm lost
204 2013-09-02 01:48:21 <TheLordOfTime> Luke-Jr: Ubuntu development: they sync from unstable for a list of packages
205 2013-09-02 01:48:30 <TheLordOfTime> Luke-Jr: so what's in Debian Unstable now is also in Ubuntu Saucy's repositories
206 2013-09-02 01:48:37 <Luke-Jr> TheLordOfTime: AFAIK there is no broken Debian bfgminer, just bitcoind
207 2013-09-02 01:48:40 <TheLordOfTime> Luke-Jr: good.
208 2013-09-02 01:48:46 <TheLordOfTime> then that's one less thing I have to worry about
209 2013-09-02 01:48:59 <TheLordOfTime> ACTION continues loading a VM to start testing to remove bitcoin from Ubuntu for E: Broken
210 2013-09-02 01:49:40 <TheLordOfTime> Luke-Jr: still, I wouldn't let any Debian team touch my code with a ten foot pole, unless they paid me billions of dollars, they have a track-record of fubaring packages
211 2013-09-02 01:51:05 <Belxjander> TheLordOfTime: just don't do what I did and force apt-get to uninstall vim followed by a system reboot
212 2013-09-02 01:51:15 <TheLordOfTime> Belxjander: heh.
213 2013-09-02 01:51:19 <TheLordOfTime> Belxjander: you did that in Ubuntu?
214 2013-09-02 01:51:26 <Belxjander> I ended up having a system self-strip everything executable off a Debian system
215 2013-09-02 01:51:31 <TheLordOfTime> Belxjander: heh
216 2013-09-02 01:51:37 <TheLordOfTime> Belxjander: honestly? I leave everything as is normally.
217 2013-09-02 01:51:44 <TheLordOfTime> fix things by hand if they'll break
218 2013-09-02 01:51:49 <TheLordOfTime> but i'm by no means a debian dev
219 2013-09-02 01:51:50 <Belxjander> ACTION has fubared a debian just making a symlink
220 2013-09-02 01:51:55 <TheLordOfTime> Belxjander: i did that twice
221 2013-09-02 01:51:57 <TheLordOfTime> but that was in a test VM
222 2013-09-02 01:52:03 <TheLordOfTime> on... oh wow, i forget how long ago that was
223 2013-09-02 01:52:18 <TheLordOfTime> ... what was Debian stable back in 2009...?
224 2013-09-02 01:52:28 <TheLordOfTime> 5 or 6?
225 2013-09-02 01:52:36 <Belxjander> TheLordOfTime: I did it three times directly on hardware then broke the debian disks and won't ever touch their non-compilable code ever again
226 2013-09-02 01:52:57 <TheLordOfTime> Belxjander: and now you know why I'm OK working with iUbuntu but I tell Debian to suck it
227 2013-09-02 01:53:07 <TheLordOfTime> except on certain packages because the code matches and/or I help maintain it.
228 2013-09-02 01:53:43 <Belxjander> TheLordOfTime: I have less of an opinion regards ubuntu and redhat and other "binary" package systems than I do about just getting sources and building the entire system "from scratch"
229 2013-09-02 01:53:52 <TheLordOfTime> ACTION shrugs
230 2013-09-02 01:54:08 <TheLordOfTime> Belxjander: to each their own
231 2013-09-02 01:54:13 <Belxjander> yup
232 2013-09-02 01:54:23 <TheLordOfTime> at least someone here who uses Ubuntu has enough information and know-how to get bitcoin removed from Ubuntu
233 2013-09-02 01:54:41 <Belxjander> definitely
234 2013-09-02 01:54:58 <Belxjander> I wouldn't trust a non-source "bitcoind" with any kind of wallet personally
235 2013-09-02 01:55:11 <Belxjander> if given the choice at least
236 2013-09-02 01:55:21 <TheLordOfTime> Belxjander: i trust BlueMatt's PPAs, because those work and are from source code, although i triple encrypt my wallets anyways...
237 2013-09-02 01:55:33 <TheLordOfTime> so you have to decrypt said wallet BEFORE bitcoind / bitcoin-qt can read it
238 2013-09-02 01:59:13 <gmaxwell> As I said before, we really should get the gitian build process building the PPAs, but I know nothing about debian or ubuntu packaging or ppas.
239 2013-09-02 01:59:47 <gmaxwell> Bluematt knows about both ppas and gitian is the obvious person to do it, but he hasn't been very active lately... presumably busy with his studies.
240 2013-09-02 01:59:55 <TheLordOfTime> gmaxwell: PPAs use bzr if iwe set up an autobuild recipe for the PPAs, but the Ubuntu packages need to be separately uploaded/maintained and they need a good reason to permanently split off of Debian autosync
241 2013-09-02 02:00:00 <TheLordOfTime> although i'm still pursuing removing it from Ubuntu
242 2013-09-02 02:02:45 <gmaxwell> TheLordOfTime: the point there being that the ppas can be build determinstically so there is no particular need to trust the person who builds them.
243 2013-09-02 02:03:37 <TheLordOfTime> mhm
244 2013-09-02 02:03:45 <TheLordOfTime> gmaxwell: however, i know some upstream teams maintain a PPA
245 2013-09-02 02:10:08 <Luke-Jr> gmaxwell: it's impossible
246 2013-09-02 02:10:14 <Luke-Jr> gmaxwell: PPA = Canonical builds it
247 2013-09-02 02:10:30 <TheLordOfTime> Luke-Jr: PPA = anyone builds it. I have a dozen PPAs myself, their builders aren't all run by canonical
248 2013-09-02 02:10:45 <TheLordOfTime> the builders are automated, the packages and the code are from scratch Ubuntu chroots for the build environment
249 2013-09-02 02:10:48 <Luke-Jr> TheLordOfTime: I don't see any way to upload builds
250 2013-09-02 02:11:28 <TheLordOfTime> Luke-Jr: you upload the source package, the system autobuilds it but it's a clean from scratch build chroot with only ubuntu core dependencies and the build-deps as stated in the package.
251 2013-09-02 02:11:39 <Luke-Jr> exactlyu
252 2013-09-02 02:11:41 <TheLordOfTime> but it's not "Canonical" that's building it
253 2013-09-02 02:11:48 <TheLordOfTime> it's "the build farm"
254 2013-09-02 02:11:50 <Luke-Jr> so there's no way to use gitian for it
255 2013-09-02 02:11:52 <TheLordOfTime> not all of it's maintained by canonical
256 2013-09-02 02:11:59 <Luke-Jr> that's even worse
257 2013-09-02 02:12:01 <TheLordOfTime> remind me what gitian is again?
258 2013-09-02 02:12:09 <Luke-Jr> TheLordOfTime: deterministic building, using a VM
259 2013-09-02 02:12:25 <gmaxwell> oh thats crud.
260 2013-09-02 02:12:42 <gmaxwell> Luke-Jr: perhaps we could get the canonical build to match the determinstic one with some proper kicking?
261 2013-09-02 02:13:06 <Luke-Jr> maybe
262 2013-09-02 02:13:08 <phantomcircuit> lolol
263 2013-09-02 02:13:23 <phantomcircuit> and maybe you'll run into a unicorn selling pixie dust
264 2013-09-02 02:13:43 <Luke-Jr> phantomcircuit: it's not really a stretch
265 2013-09-02 02:13:53 <phantomcircuit> i bet they dont even produce binaries consistent between themselves...
266 2013-09-02 02:14:02 <Luke-Jr> phantomcircuit: gitian's VM is Ubuntu itself
267 2013-09-02 02:16:24 <gmaxwell> yea, it's not that unlikely that its fine.
268 2013-09-02 02:17:40 <gmaxwell> with enough work we could make a build script that actually bootstrapped the entire dependency chain inside the build enviroment and got a consistent result.
269 2013-09-02 02:17:51 <gmaxwell> "Hey, it's not _our_ cpu cycles we're burning there"
270 2013-09-02 02:18:20 <Luke-Jr> something tells me there might be a resource limit :p
271 2013-09-02 02:19:29 <gmaxwell> I dunno, you'd be surprised. Someone eventually noticed on the debian build farm that one of my packages was taking 1.5 weeks to finish its make check when building. (Their build settings were wrong and they were building with fpu on an arch that had only software emulation).. but they only noticed because it was delaying the package releases.
272 2013-09-02 02:19:39 <phantomcircuit> gmaxwell, stick a cpu miner into the bootstrap process
273 2013-09-02 02:19:42 <phantomcircuit> ???
274 2013-09-02 02:19:43 <phantomcircuit> profit
275 2013-09-02 02:20:55 <Luke-Jr> btw, I told vbuterin he should hang out here more
276 2013-09-02 02:22:32 <gmaxwell> Really the whole gitian thing depending on a random ubuntu vm is a bit lame in any case. It's a good start but better could be done.
277 2013-09-02 02:22:44 <gmaxwell> It's ashame we have so many non-trivial dependencies though.
278 2013-09-02 02:22:51 <Luke-Jr> gmaxwell: I'd like to get Gentoo's portage made deterministic :P
279 2013-09-02 02:23:25 <Luke-Jr> then the whole OS can be verified
280 2013-09-02 02:24:16 <Luke-Jr> *and* have real dependency management
281 2013-09-02 02:53:49 <TheLordOfTime> gmaxwell: what command line arguments do you suggest I use to get the information i need for this testing?
282 2013-09-02 02:57:00 <TheLordOfTime> gmaxwell: who reported the package broken? It seems to be syncing right in Ubuntu, and that's directly from Debian
283 2013-09-02 02:57:05 <TheLordOfTime> or is there a specific point where it fails to sync
284 2013-09-02 02:58:01 <gmaxwell> TheLordOfTime: it fails at a fairly high point around block 240000 and I have never seen a reported failure on ubuntu that I considered credible.
285 2013-09-02 02:58:43 <TheLordOfTime> gmaxwell: okay, i'm testing myself right now, on the version from Debian
286 2013-09-02 02:58:53 <TheLordOfTime> gmaxwell: if/when it fails i'll document that as a bug
287 2013-09-02 02:59:05 <TheLordOfTime> and the failuire is consistently able to be replicated in the Debian version near that point?
288 2013-09-02 02:59:16 <TheLordOfTime> based on reports so far
289 2013-09-02 03:01:32 <gmaxwell> Based on reports. In the debian version ON DEBIAN UNSTABLE. No reports of this not on debian unstable.
290 2013-09-02 03:01:53 <TheLordOfTime> okay, Ubuntu Saucy's version of bitcoin-qt is direct from Debian Unstable without any code changes
291 2013-09-02 03:02:07 <TheLordOfTime> so if it breaks in Ubuntu then i'll work on the Ubuntu side, before moving to attacking the Debian side
292 2013-09-02 03:02:18 <TheLordOfTime> gmaxwell: but it consistently fails around 200k + blocks?
293 2013-09-02 03:02:25 <TheLordOfTime> (give or take 40k)
294 2013-09-02 03:03:13 <gmaxwell> so it is reported. (Though if you're testing you should let it sync up to current before calling it working)
295 2013-09-02 03:03:46 <TheLordOfTime> gmaxwell: mhm
296 2013-09-02 03:04:25 <TheLordOfTime> gmaxwell: if it fails once, i'll have that, then wipe the data directory and start again, saving the debug.log each time, so if i can repeatedly replicate it then it's a bad bug.
297 2013-09-02 03:41:44 <Krellan> gmaxwell: I updated my pull request for ping/pong, I instrumented it to log more anomalies but so far haven't seen any, unfortunately. Wanted to find out what those weird clients were.
298 2013-09-02 03:50:43 <Luke-Jr> Krellan: oh, is that why I was ping flooding? <.<
299 2013-09-02 03:51:29 <phantomcircuit> Luke-Jr, lol
300 2013-09-02 04:14:23 <maaku> i'm looking at the code but not sure the answer to this : is it legal to OP_PUSHDATAn , when n is strictly greater than necessary?
301 2013-09-02 04:16:26 <Krellan> Luke-Jr: :)
302 2013-09-02 04:18:53 <Krellan> gmaxwell: Checked the log again and did notice something from last night. Received some pongs even when there was no outstanding ping. The nonce in those pongs was all 0. http://pastebin.com/rQwRN1i1
303 2013-09-02 04:19:41 <Krellan> They happened over the space of a few hours, not all bunched together, and all versions reported /Satoshi:0.8.99/
304 2013-09-02 06:39:55 <jorash> anyone knwo where the cointerra people hang out (if on irc at all
305 2013-09-02 06:39:57 <jorash> )?
306 2013-09-02 06:42:18 <sipa> maaku: in theory legal, but ugly, and there's a pull request to just make it illegal
307 2013-09-02 09:23:24 <warren> is anyone running into the RPC stalling issue with 0.8.2+?
308 2013-09-02 09:23:40 <warren> it's related to the complaints about getwork causing other RPC's to stall
309 2013-09-02 09:23:45 <warren> something to do with keep-alive
310 2013-09-02 09:26:39 <gmaxwell> see double's comments in here he gave a reproduction program.
311 2013-09-02 09:26:50 <gmaxwell> (can't ask for more than that!)
312 2013-09-02 09:27:39 <warren> one of our users is compiling in a larger RPC thread limit to workaround the issue ...
313 2013-09-02 09:27:44 <warren> s/4/64/
314 2013-09-02 09:28:59 <gmaxwell> seems like not a great idea to me.
315 2013-09-02 09:45:13 <tgs3> gmaxwell: debian patches are bad?
316 2013-09-02 09:48:09 <gmaxwell> I have no idea, I've never reviewed them, they've never even been submitted for our review AFAIK. I do know that users report bitcoin in debian unstable does not work correctly now, though that may have nothing to do with the patches (and there is even some evidence that it doesn't)
317 2013-09-02 09:57:12 <jouke> thread limit?
318 2013-09-02 09:57:17 <jouke> amount of threads?
319 2013-09-02 10:07:25 <tgs3> gmaxwell: what process of a distribution patching bitcoin would please you? :)
320 2013-09-02 10:07:54 <tgs3> like, distro discovers bitcoind say uses random memory write&execute (like java, firefox) and replaces it to make code safer and compatible with grsecurity
321 2013-09-02 10:13:04 <gmaxwell> and leaves it insecure for all other users with simultaneously indirectly announcing (whatever) vulnerability to the world? Or haphazardly changing code which they do not understand and leaving the software completely insecure as debian did with openssl?
322 2013-09-02 10:13:52 <gmaxwell> The bitcoin core development team is reachable on IRC nearly 24/7. There is an official private security list for it that can be emailed. Patches can be submitted for review on github.
323 2013-09-02 10:14:11 <gmaxwell> Silently patching things and shipping things out to users is pretty much the worst possible path.
324 2013-09-02 10:15:22 <gmaxwell> Doubly so since the fact that debian unstable bitcoin not syncing the chain suggests that the parties responsible for the package are not testing it.
325 2013-09-02 10:15:39 <gmaxwell> If we're rejecting some change than thats another matter, but at least then there would be an oppturnity for review and a reason given for suggesting it.
326 2013-09-02 10:17:56 <tgs3> gmaxwell: ok we will keep this in mind ;)
327 2013-09-02 10:18:48 <gmaxwell> Who is we?
328 2013-09-02 10:21:35 <tgs3> there is new distribution in planning, to be most secure+private one, I'm helping
329 2013-09-02 10:22:55 <tgs3> by taking best possible software like tor, Qubes-OS, selected hardware, bitcoin :-) vpns and everything and preconfiguring it.. like tails or johndonym but even more (and installable)
330 2013-09-02 10:23:18 <gmaxwell> I'd recommend not packaging the Bitcoin software. Separately, if you're doing something security&privacy oriented, I hope your distribution will do something like gitian so the binaries are reproducable.
331 2013-09-02 10:23:18 <tgs3> (aimed at being primary system, with option to live-cd maybe, not viceversa)
332 2013-09-02 10:23:36 <tgs3> gmaxwell: yes, we work on verificable builds righ now
333 2013-09-02 10:23:40 <gmaxwell> Fantastic!
334 2013-09-02 10:23:43 <tgs3> :-)
335 2013-09-02 10:24:28 <gmaxwell> (okay, well, if you're going to be that focused maybe I'd downgrade my recommendation not to ship bitcoin... :) ... but keep in mind: Bitcoin is a consensus protocol, and at times running an old version is a security problem for the user even if it didn't use to have any bugs)
336 2013-09-02 10:25:27 <tgs3> we will upgrade such software in matter of days
337 2013-09-02 10:25:36 <tgs3> (I hope that is the plan)
338 2013-09-02 10:27:38 <gmaxwell> Okay, great. Well of course there is a transparency tradeoff ... rapid updates can't get much public review. But sure, my concern _generally_ ends if the upgrade cycle, even just for user initiated upgrades, can be reduced to "a couple months"
339 2013-09-02 10:28:32 <tgs3> few month are doable
340 2013-09-02 10:28:43 <tgs3> :)
341 2013-09-02 10:31:18 <gmaxwell> as far as things like patches to make bitcoin more secure in a hardened enviroment, I'd very much like them upstream... I can't promise to instantly merge any of them, especially ones that were only useful on non-primary platforms... but making it stronger is certantly a goal.
342 2013-09-02 10:31:55 <tgs3> gmaxwell: ok. that was just an example. in general bitcoind is compatible with grsecurity (W^X / NX / noexec protection)
343 2013-09-02 10:34:13 <gmaxwell> yes, sure, some bitcoind users have those patches... (though there are some network behavior patches that have behaved poorly on interaction with the bitcoin network, but no one running grsecurity who has mentioned them has been technically adept enough to troubleshoot)
344 2013-09-02 10:40:41 <tgs3> yeah, as I said, bitcoind (vanilla) is behaving nicelly with grsecurity too :)
345 2013-09-02 10:46:42 <_dr> is there an easy way to set up a listen-only bitcoind? (i.e. don't relay blocks, etc.)
346 2013-09-02 10:47:18 <sipa> _dr: only connect to a sibgle peer?
347 2013-09-02 10:47:39 <sipa> depending on what your "etc." means
348 2013-09-02 10:49:08 <_dr> what I want is an up-to-date copy of the blockchain, without causing too much outbound traffic (which i have to pay for)
349 2013-09-02 10:49:29 <sipa> -connect=IP
350 2013-09-02 10:49:30 <_dr> maybe rsyncing .bitcoin/blocks of my full node would be a better idea
351 2013-09-02 11:06:45 <TD> Good day
352 2013-09-02 11:06:59 <sipa> Greetings from Bratislava
353 2013-09-02 11:11:46 <gmaxwell> This is where I get to be american and admit that I had no idea where that was until I looked it up.
354 2013-09-02 11:13:01 <sipa> i'm sure the same would be true for most state capitals in the us for me
355 2013-09-02 11:14:21 <gmaxwell> yes, but thats also the case for most americans. (can't identify more than a few state capitals :P )
356 2013-09-02 11:15:22 <sipa> i took a night train to vienna, which had 2.5 delay "due to a police and rescue operation in germany", there i missed my connection to bratislava (which runs only every hour), where i missed my connection to poprad (whuh runs only every 4h), the latter of which seema to be full, so i get to take a later and even slower train
357 2013-09-02 11:15:34 <sipa> yay for public transport
358 2013-09-02 11:15:56 <TD> bratislava is a nice city
359 2013-09-02 11:16:11 <sipa> total: 5.5h delay due to missed connections
360 2013-09-02 11:16:27 <TD> when i came back via vienna the checkin lady asked where i'd been
361 2013-09-02 11:16:33 <TD> so i told her, and she said
362 2013-09-02 11:16:40 <TD> "bratislava? like in the horror movie?"
363 2013-09-02 11:16:43 <TD> (she means Hostel)
364 2013-09-02 11:16:45 <sipa> lol
365 2013-09-02 11:16:52 <TD> apparently that's what makes the city famous internationally these days
366 2013-09-02 11:17:02 <sipa> i've never seen it
367 2013-09-02 11:17:06 <TD> http://en.wikipedia.org/wiki/Hostel_(2005_film)
368 2013-09-02 11:17:13 <sipa> ok
369 2013-09-02 11:17:13 <TD> College students Paxton and Josh, along with their friend Ãli, are backpacking across Europe. After a night of partying, they meet a Dutch man named Alexei who informs them about an undocumented hostel near Bratislava filled with beautiful women.
370 2013-09-02 11:17:13 <TD> neither have i. horror isn't my thing
371 2013-09-02 11:17:34 <sipa> Alexei is not a Dutch bame!
372 2013-09-02 11:17:37 <sipa> *name
373 2013-09-02 11:17:38 <TD> you can tell it's a US film because the names are all weird. dutch man named alexei?
374 2013-09-02 11:17:41 <TD> lol. beat me to it.
375 2013-09-02 11:17:48 <gmaxwell> hahah!
376 2013-09-02 11:18:00 <sipa> Alex or Alexander
377 2013-09-02 11:18:17 <TD> the stereotype of eastern europe being filled with beautiful women is also classic, albiet somewhat more accurate
378 2013-09-02 11:18:36 <TD> however in this case they torture and kill the travellers
379 2013-09-02 11:18:56 <sipa> so different from google...
380 2013-09-02 11:18:56 <sipa> yeah, like 51% of the population is female here!
381 2013-09-02 11:19:10 <TD> hah
382 2013-09-02 11:19:53 <sipa> another apparently true stereotype: *everyone* is smoking here
383 2013-09-02 11:20:37 <TD> well, that's not a surprise to me. basically the further east you go the more smokers there are. the step up from UK->Switzerland was noticeable, Switzerland->former soviet bloc even moreso
384 2013-09-02 11:21:00 <sipa> japan must be horrible!
385 2013-09-02 11:21:00 <TD> though the swiss smoke like crazy too. i think the only thing that "dilutes" it is the large number of northern european immigrants
386 2013-09-02 11:21:29 <sipa> at least their train stations don't smell like tobacco
387 2013-09-02 11:21:32 <TD> unfortunately the brush i'm painting with isn't broad enough to reach japan
388 2013-09-02 11:21:48 <sipa> analogies are best when they break down
389 2013-09-02 11:22:05 <TD> a bad analogy is like a broken window .... actually n/m
390 2013-09-02 11:22:08 <TD> ACTION -> lunch
391 2013-09-02 11:22:13 <sipa> cy
392 2013-09-02 11:22:16 <sipa> cya
393 2013-09-02 11:24:33 <gmaxwell> thats what the great wall of china must be blocking, all the smokers. :P
394 2013-09-02 11:29:36 <phantomcircuit> gmaxwell, there's a ton of smokers in china
395 2013-09-02 11:29:40 <phantomcircuit> it's not working very well
396 2013-09-02 11:30:03 <gmaxwell> Like many chinese firewalls!
397 2013-09-02 11:30:09 <phantomcircuit> lol
398 2013-09-02 11:42:06 <TD> new cryptdb release, nice++
399 2013-09-02 12:14:48 <TD> sipa: you might find this paper interesting: http://eprint.iacr.org/2013/535.pdf
400 2013-09-02 12:14:57 <TD> it's about hardware acceleration of koblitz curve scalar mult
401 2013-09-02 12:15:16 <TD> actually for double lazy reduction of scalers into wNAF form
402 2013-09-02 12:15:23 <TD> scalars*
403 2013-09-02 12:27:48 <sipa> TD: it's for actual koblitz curves
404 2013-09-02 12:28:04 <TD> ours is not an actual koblitz curve?
405 2013-09-02 12:28:16 <sipa> no, koblitz curves are over binary fields
406 2013-09-02 12:28:26 <sipa> we have a koblitz-like curve on a prime field
407 2013-09-02 12:28:37 <TD> ah
408 2013-09-02 12:28:39 <sipa> though perhaps the same applies
409 2013-09-02 12:28:48 <TD> well, do you have to reduce scalars to wNAF form?
410 2013-09-02 12:28:52 <sipa> yes
411 2013-09-02 12:28:54 <TD> i thought yes
412 2013-09-02 12:29:13 <TD> that's the part their proposed chip accelerates
413 2013-09-02 12:29:17 <TD> so i guess it might still apply?
414 2013-09-02 12:29:19 <sipa> well you don't have to, but it's much faster to do so
415 2013-09-02 12:29:24 <TD> right
416 2013-09-02 12:29:31 <TD> wNAF form is larger, but it's faster to perform the mult
417 2013-09-02 12:29:45 <gmaxwell> Wee: Mtgox spends immature coins .... now explains why customers in #mtgox have been complaining about not-relaying transactions for some time.
418 2013-09-02 12:30:28 <gmaxwell> also, bc.i displays spends of immature coins, and doesn't even mark them as unusual.
419 2013-09-02 12:30:28 <TD> ah ha
420 2013-09-02 12:30:32 <sipa> TD: though iirc the time to compute the wNAF is very small compared to the actual EC operations
421 2013-09-02 12:31:19 <sipa> anyway, i'll read the paper but not on a cell phone in a train :)
422 2013-09-02 12:31:31 <TD> sure. also do we need montgomery multiplication anywhere?
423 2013-09-02 12:31:40 <TD> i thought the answer is yes but now i can't find any mention of it
424 2013-09-02 12:31:41 <sipa> no
425 2013-09-02 12:31:45 <TD> ok
426 2013-09-02 12:31:54 <sipa> at least my implementation doesn't use it
427 2013-09-02 12:32:45 <sipa> due to secp256k1 field size being very close to 2^256, montgomery doean't gain you much on 64-bit systems
428 2013-09-02 12:32:58 <sipa> but maybe it still gains you something
429 2013-09-02 12:34:31 <TD> there's a paper on simd accelerating that, but i guess you already extracted max juice out of simd
430 2013-09-02 12:34:37 <sipa> openssl's generic prime-field ec code uses montgomery (and an assembly-imolemented version of it even)
431 2013-09-02 12:34:53 <sipa> i don't use simd
432 2013-09-02 12:35:27 <sipa> and neither does openssl iirc
433 2013-09-02 12:36:15 <TD> http://eprint.iacr.org/2013/519.pdf
434 2013-09-02 12:36:22 <TD> that's the paper if it has any useful ideas
435 2013-09-02 12:57:22 <jgarzik> sipa, IRC'ing from a beach or mountaintop chalet, I hope? :)
436 2013-09-02 12:57:29 <jgarzik> gmaxwell, lovely
437 2013-09-02 12:57:42 <sipa> jgarzik: train
438 2013-09-02 12:58:01 <jgarzik> gmaxwell, what's immature? 0 conf? 1 conf?
439 2013-09-02 12:58:24 <jgarzik> sipa, cool. Slow but a great way to travel.
440 2013-09-02 12:58:43 <sipa> jgarzik: scroll up :)
441 2013-09-02 12:59:14 <gmaxwell> jgarzik: a newly generated coin with only 92 confirmations... so a somewhat odd case!
442 2013-09-02 12:59:34 <gmaxwell> (well I think it had far fewer when they authored the transactionâ user said he'd been waiting for hours)
443 2013-09-02 13:15:16 <coingenuity> http://www.youtube.com/watch?v=1mYqY5YELd0 <<< how i imagine bratislava, sipa
444 2013-09-02 13:15:32 <coingenuity> lmao
445 2013-09-02 13:15:50 <bizoro> you guys know about bitmessage?!
446 2013-09-02 13:16:17 <runeks> Yes.
447 2013-09-02 13:16:28 <coingenuity> MIAMI WIZE IS NUMBER ONE NEW SHOW xD
448 2013-09-02 13:16:34 <coingenuity> too funny
449 2013-09-02 13:17:39 <runeks> LOL. dog with a human hand in his mouth :)
450 2013-09-02 13:17:53 <coingenuity> https://www.youtube.com/watch?v=B5NuuJQT8ZM << i have to imagine that sipa lives like this in bratislava
451 2013-09-02 13:19:27 <gmaxwell> torists like that in bratislava
452 2013-09-02 13:19:49 <coingenuity> btw gmaxwell - just a random thought
453 2013-09-02 13:20:10 <coingenuity> you cant use config files now to manage datadirs, or so it appears from the documentation on .conf's
454 2013-09-02 13:20:24 <coingenuity> but since the chain is getting so big, a lot of peeps will be using external storage for blocks
455 2013-09-02 13:20:48 <gmaxwell> symlinks are your friend.
456 2013-09-02 13:20:54 <coingenuity> perhaps a blockdir directive in the .conf system might be good, so you can just offload the 'big' crap to non-default storage locations
457 2013-09-02 13:21:01 <coingenuity> so that no symlinks are needed
458 2013-09-02 13:21:03 <coingenuity> heh
459 2013-09-02 13:21:41 <sipa> not sur eit was merged
460 2013-09-02 13:21:53 <sipa> but there was a pull to choose the wallet file
461 2013-09-02 13:26:08 <gmaxwell> coingenuity: yea, doesn't sound unreasonable, esp since there is nothing like a cross device symlink in windows that would work for this as far as I know.
462 2013-09-02 13:27:15 <coingenuity> right-o, can't even imagine trying to do it in doze8 :X
463 2013-09-02 13:27:30 <Luke-Jr> gmaxwell: eh, depends..
464 2013-09-02 13:27:52 <Luke-Jr> gmaxwell: even back in DOS, you could define Z:\ to be the same as C:\dummydir\
465 2013-09-02 13:28:01 <Luke-Jr> more like a hardlink
466 2013-09-02 13:30:04 <sipa> ASSIGN ?
467 2013-09-02 13:30:30 <coingenuity> ACTION amuses self while rsync moves blocks around
468 2013-09-02 13:30:35 <sipa> or JOIN
469 2013-09-02 13:30:40 <gmaxwell> Luke-Jr: hm.. does that actually change anything but the cli?
470 2013-09-02 13:31:03 <Luke-Jr> gmaxwell: yes
471 2013-09-02 13:31:23 <Luke-Jr> or at least it did..
472 2013-09-02 13:31:29 <gmaxwell> cool.
473 2013-09-02 13:31:35 <gmaxwell> I have no recollection of that. Odd.
474 2013-09-02 13:31:53 <sipa> http://en.wikipedia.org/wiki/List_of_MS-DOS_commands#JOIN
475 2013-09-02 13:32:17 <sipa> hmm, that may be the opposite
476 2013-09-02 13:33:24 <coingenuity> mklink still works, i think
477 2013-09-02 13:33:30 <coingenuity> but its a bit strange on windows
478 2013-09-02 13:34:39 <Luke-Jr> subst I think
479 2013-09-02 13:34:54 <Luke-Jr> http://en.wikipedia.org/wiki/SUBST
480 2013-09-02 13:35:57 <coingenuity> cmd != dos anyway
481 2013-09-02 13:36:09 <coingenuity> its missing a lot of shit in winxp+
482 2013-09-02 13:36:42 <_dr> dblspace.exe, most importantly!
483 2013-09-02 13:37:44 <gmaxwell> stacker 4 evar
484 2013-09-02 13:38:27 <Luke-Jr> _dr: that's called ramzswap on Linux
485 2013-09-02 14:17:23 <jgarzik> sipa, bratislava? Bratislava???
486 2013-09-02 14:17:30 <jgarzik> sipa, http://www.youtube.com/watch?v=cqEsE-IKBI8
487 2013-09-02 14:17:52 <jgarzik> sipa, Classic scene from "Eurotrip", wherein the backpacking-across-Europe characters get stranded in Bratislava
488 2013-09-02 14:18:11 <jgarzik> even includes some exchange rate joke than any bitcoiner may enjoy :)
489 2013-09-02 14:19:31 <gmaxwell> coingenuity: ^ you might like that video. :P
490 2013-09-02 14:19:59 <gmaxwell> ACTION waits for them to overdub it so they have $1.85 ... and one bitcoin.
491 2013-09-02 14:22:43 <sipa> jgarzik: no longer stranded there!
492 2013-09-02 14:24:16 <jgarzik> "a bitcoin??? I quit. I'll buy my own hotel!"
493 2013-09-02 14:27:49 <jgarzik> You know you're a techie, when you start to fill up a 7-port USB hub
494 2013-09-02 14:28:12 <sipa> jgarzik: that's trivial
495 2013-09-02 14:28:17 <sipa> (with 6 block erupters)
496 2013-09-02 14:29:30 <Belxjander> jgarzik: then I definitely qualify with 2x4port hubs and both are almost full with 1 port "free" each :P
497 2013-09-02 14:29:49 <sipa> just enough to connect another hub :p
498 2013-09-02 14:30:10 <jgarzik> heh
499 2013-09-02 14:30:46 <jgarzik> people used to test the SCSI driver by physically connecting SCSI bridges (like USB hubs) together, to see how deeply they could nest
500 2013-09-02 14:31:29 <jgarzik> of course, this was in the era before virtual-everything, when that sort of thing became trivial to simulate
501 2013-09-02 14:32:25 <sipa> seems pulltester's problem with headersfirst was just the "you slow peer, i disconnect you!" mechanism triggering
502 2013-09-02 14:33:06 <sipa> apparently not
503 2013-09-02 14:35:55 <sipa> ah no, gavin was right
504 2013-09-02 14:36:33 <jgarzik> ACTION is looking forward to testing that after US holiday ends
505 2013-09-02 14:39:00 <gmaxwell> sipa: what did gavin think the issue was?
506 2013-09-02 14:39:09 <gmaxwell> oh infinite reorg loop
507 2013-09-02 14:39:11 <gmaxwell> cool
508 2013-09-02 14:39:13 <sipa> indeed
509 2013-09-02 14:39:18 <sipa> \o/ bugs to fix
510 2013-09-02 14:39:49 <gmaxwell> Disconnection sounded unoptimal at first, but on reflection if it were doing that it might be right toâ I suppose it should protect -connect/-addnode hosts.
511 2013-09-02 14:40:32 <jgarzik> sigh. i will miss fedora. they do such much better than ubuntu for Fighting The Good Open Source Fight. but for bringing up a new node w/ bitcoind, it's just easier to use ubuntu LTS server.
512 2013-09-02 14:49:17 <sipa> gmaxwell: there's no other way of saying "cancel that, i don't need your blocks!" otherwise
513 2013-09-02 15:07:52 <sipa> so fun... coding on a train, and using pulltester as a syntax checker :p
514 2013-09-02 15:08:13 <michagogo> Lol :-P
515 2013-09-02 15:08:35 <michagogo> Don't you have a compiler?
516 2013-09-02 15:08:46 <michagogo> Or are you working on your phone or tablet or something?
517 2013-09-02 15:09:45 <sipa> nah, just lazy
518 2013-09-02 15:10:47 <michagogo> Does the pull tester not take longer to give a result than a simple "make -f mak<tab>unix"?
519 2013-09-02 15:10:53 <sipa> yes
520 2013-09-02 15:11:00 <gmaxwell> yes it takes longer
521 2013-09-02 15:11:07 <TD> it's a train. everything takes longer :)
522 2013-09-02 15:11:23 <gmaxwell> michagogo: do not asked encrypted questions if you want unencrypted answers. :P
523 2013-09-02 15:11:27 <Luke-Jr> michagogo: it runs a lot of tests too
524 2013-09-02 15:11:37 <sipa> but if you write a single-line patch, there's no way in knowing that it'd contain a typo, right? :p
525 2013-09-02 15:11:38 <michagogo> Encrypted questions?
526 2013-09-02 15:12:05 <michagogo> sipa: Oh, are you using pulltester because you want the tests to run too?
527 2013-09-02 15:12:13 <gmaxwell> I should sleep. :P I misread your question due to reading it out of context. :)
528 2013-09-02 15:12:21 <michagogo> lol
529 2013-09-02 15:12:33 <sipa> no, just because i don't expect the compilation to feel :p
530 2013-09-02 15:12:45 <michagogo> ...does not compute...
531 2013-09-02 15:13:00 <michagogo> Feel what? o_O
532 2013-09-02 15:13:40 <Luke-Jr> sipa: O.o
533 2013-09-02 15:14:09 <michagogo> If you're just using the pull tester as a syntax checker, you could achieve the same thing faster with `make -f makefile.unix`, unless I'm missing something?
534 2013-09-02 15:14:41 <sipa> michagogo: i also want to run the test of course
535 2013-09-02 15:14:47 <michagogo> Ah
536 2013-09-02 15:15:01 <michagogo> (you just said you didn't want to run said tests)
537 2013-09-02 15:15:02 <sipa> michagogo: but since i don't expect the compilation to fail, i just write and push, and wait for a mail
538 2013-09-02 15:15:15 <sipa> and then notice that *again* a just made a typo :)
539 2013-09-02 15:15:18 <michagogo> Also, is there no automated test script?
540 2013-09-02 15:15:26 <sipa> yes, pulltester?
541 2013-09-02 15:15:35 <sipa> or you mean the unit tests?
542 2013-09-02 15:15:40 <michagogo> ...I mean, a script that runs the pulltester's tests locally
543 2013-09-02 15:15:42 <sipa> those wouldn't catch it
544 2013-09-02 15:15:51 <sipa> no, don't have all that set up
545 2013-09-02 15:16:24 <michagogo> Ah, so it's not as simple as "run this programs against it", it requires a whole complicated setup?
546 2013-09-02 15:16:29 <Luke-Jr> oooooooooooo
547 2013-09-02 15:16:54 <Luke-Jr> michagogo: some of it is
548 2013-09-02 15:17:13 <Luke-Jr> michagogo: I have my setup so it runs the test_bitcoin every time I build
549 2013-09-02 15:17:58 <sipa> but i never bothered
550 2013-09-02 15:17:58 <sipa> in practice it probably is that simple
551 2013-09-02 15:17:58 <sipa> it means a java compiler at the least *shudder*
552 2013-09-02 15:21:18 <michagogo> A java compiler? o_O
553 2013-09-02 15:21:37 <michagogo> Also, why *shudder*?
554 2013-09-02 15:21:45 <michagogo> sudo apt0get install openjdk
555 2013-09-02 15:21:49 <michagogo> -typos
556 2013-09-02 15:22:40 <Luke-Jr> michagogo: Java = suck
557 2013-09-02 15:23:37 <Luke-Jr> let's all hate Java together!
558 2013-09-02 15:24:03 <michagogo> Luke-Jr: Not great, but not inherently horrible either
559 2013-09-02 15:24:06 <sipa> i'm exaggerating of course
560 2013-09-02 15:24:08 <sipa> but i haven
561 2013-09-02 15:24:10 <Luke-Jr> :p
562 2013-09-02 15:24:39 <sipa> but i haven't touched it in a long time, and all that maven stuff and whatever around it scares me
563 2013-09-02 15:25:18 <Luke-Jr> ACTION sees no reason Java shouldn't use Make like everything else
564 2013-09-02 15:25:28 <TheLordOfTime> ... o... kay this is weird... gmaxwell: any reports of partially-synced blockchains returning as "Unable to read block" after a shutdown of bitcoin-qt and a restart on the Debian Unstable version?
565 2013-09-02 15:25:55 <michagogo> Luke-Jr: Doesn
566 2013-09-02 15:26:03 <michagogo> Luke-Jr: Doesn't make just call different compilers?
567 2013-09-02 15:26:04 <gmaxwell> TheLordOfTime: yep, thats coingenuity's report!
568 2013-09-02 15:26:14 <gmaxwell> TheLordOfTime: have the debug log from that?
569 2013-09-02 15:26:23 <Luke-Jr> michagogo: exactly
570 2013-09-02 15:26:38 <michagogo> Luke-Jr: So what do you mean, you see no reason Java shouldn't use Make like everything else?
571 2013-09-02 15:26:40 <gmaxwell> TheLordOfTime: and how did you reproduce? just sync, stop the software and restart it?
572 2013-09-02 15:26:48 <michagogo> Can't you use make to invoke javac?
573 2013-09-02 15:26:56 <coingenuity> XD
574 2013-09-02 15:26:56 <Luke-Jr> michagogo: maven is some kind of make "replacement" Java programs are apparently using
575 2013-09-02 15:27:03 <coingenuity> gmaxwell: "coingenuity syndrome"
576 2013-09-02 15:27:06 <michagogo> Ah
577 2013-09-02 15:27:09 <gmaxwell> sipa: think that post threading changes might be quiting before getting data to disk?
578 2013-09-02 15:27:26 <michagogo> http://en.wikipedia.org/wiki/Apache_Maven ?
579 2013-09-02 15:27:27 <sipa> gmaxwell: ?
580 2013-09-02 15:28:02 <TheLordOfTime> gmaxwell: well... reproduction was as follows: (1) loaded the program in the VM yesterday. (2) closed program and VM due to me not wanting the system to overheat and restart, (3) turned on VM today, (4) ran bitcoin-qt -debug, (5) error popup: E: FAIL
581 2013-09-02 15:28:03 <sipa> TheLordOfTime: was this in the mid of a resync?
582 2013-09-02 15:28:11 <TheLordOfTime> sipa: read last line
583 2013-09-02 15:28:29 <sipa> sorry, i mean was this during initial block sync
584 2013-09-02 15:28:30 <TheLordOfTime> ... note to self: don't do cat debug.log
585 2013-09-02 15:28:38 <michagogo> Looks like maven is more than *just* a builder -- it also does things like fetch dependencies, etc.
586 2013-09-02 15:28:45 <gmaxwell> thats why I asked for the log.
587 2013-09-02 15:28:46 <sipa> TheLordOfTime: had the node synced fully already?
588 2013-09-02 15:28:46 <TheLordOfTime> sipa: after bitcoin-qt was closed, and later reopened, yes.
589 2013-09-02 15:28:52 <TheLordOfTime> sipa: no, this isn't my normal bitcoin node
590 2013-09-02 15:28:55 <michagogo> ;;google maven vs make
591 2013-09-02 15:28:56 <gribble> Java Build Systems: Make vs. Ant vs. Maven | GrokCode: <http://grokcode.com/538/java-build-systems-a-sad-state-of-affairs/>; ant - Why is no one using make for Java? - Stack Overflow: <http://stackoverflow.com/questions/2209827/why-is-no-one-using-make-for-java>; java - Maven vs. Ant - Stack Overflow: <http://stackoverflow.com/questions/745261/maven-vs-ant>
592 2013-09-02 15:29:05 <TheLordOfTime> this is a VM setup, it's me testing the Debian Unstable package which also got into ubuntu saucy
593 2013-09-02 15:29:18 <sipa> TheLordOfTime: ok, during initial sync, blocks aren't flushed to disk all the time, as it would kill syncup speed
594 2013-09-02 15:29:25 <TheLordOfTime> sipa: partly-synced for the initial time, but however it was terminated/turned off at a point
595 2013-09-02 15:29:30 <TheLordOfTime> sipa: it didn't even continue syncing this time
596 2013-09-02 15:29:36 <michagogo> Wow, I'd never associated "maven" with ××××
597 2013-09-02 15:29:48 <sipa> TheLordOfTime: but it does mean that in theory you can get a block that is not on disk, but manages to get written to the index
598 2013-09-02 15:30:04 <sipa> so yes, this is possible in theory
599 2013-09-02 15:30:09 <TheLordOfTime> sipa: well, i'm pulling debug.log now so...
600 2013-09-02 15:30:10 <runeks> Is there a nifty way of reading the blocks from the best chain faster than using getblock()? The getblock RPC call is the main bottleneck for me.
601 2013-09-02 15:30:10 <sipa> but only during initial sync
602 2013-09-02 15:30:17 <TheLordOfTime> let's make conclusions only *after* i've pulled the logs for us, okay?
603 2013-09-02 15:30:23 <sipa> TheLordOfTime: sure
604 2013-09-02 15:30:31 <TD> maven is just a zip you extract and add to your $PATH
605 2013-09-02 15:30:32 <sipa> runeks: apart from reading it from disk yourself, not really
606 2013-09-02 15:30:33 <TD> not very difficult
607 2013-09-02 15:30:44 <TheLordOfTime> ... geez.......... it's still loading the file... o.O
608 2013-09-02 15:31:14 <runeks> sipa: How would I go about that? Make sure bitcoind is shut down and read the file position of each block in the best chain from bitcoind's leveldb database?
609 2013-09-02 15:31:35 <sipa> runeks: that's one way, or you parse it entirely yourself, like armory does
610 2013-09-02 15:31:53 <sipa> runeks: it's not exactly a use case the software is optimized for
611 2013-09-02 15:32:50 <runeks> sipa: That's what I've been thinking about. Should an RPC call to localhost really add that much latency? Or is it bitcoind that is slow at fetching random blocks from disk?
612 2013-09-02 15:33:19 <runeks> Cause bitcoind should, in theory, be just as fast to read blocks from disk as my program can do. It's only the RPC interface that might add latency.
613 2013-09-02 15:33:20 <sipa> RPC has some overhead, but mostly, it grabs the internal cs_main lock
614 2013-09-02 15:33:42 <sipa> so everything else important that it's doing blocks the RPC processing
615 2013-09-02 15:33:59 <gmaxwell> runeks: are you one of those wizards in the forum who is running system('bitcoind ...') to use the rpc from php? :P
616 2013-09-02 15:34:14 <michagogo> lol
617 2013-09-02 15:34:18 <runeks> sipa: Ah, I see. Is there a simple way of not making it do that?
618 2013-09-02 15:34:21 <gmaxwell> (sorry, its obligatory to ask the dumb questions)
619 2013-09-02 15:34:40 <sipa> runeks: not without making it crash
620 2013-09-02 15:35:08 <TheLordOfTime> sipa: 90MB log file o.O
621 2013-09-02 15:35:13 <TheLordOfTime> give it a minute to upload somewhere
622 2013-09-02 15:35:16 <sipa> TheLordOfTime: is that all?
623 2013-09-02 15:35:25 <TheLordOfTime> sipa: that's all the log file has before E:Fail
624 2013-09-02 15:35:26 <gmaxwell> xz is your friend. :P
625 2013-09-02 15:35:29 <sipa> 33G /home/pw/.bitcoin/debug.log
626 2013-09-02 15:35:29 <sipa> $ du -sh ~/.bitcoin/debug.log
627 2013-09-02 15:35:41 <runeks> gmaxwell: :) I am not no. Speaking of PHP wizardry, have you seen this? http://www.reddit.com/r/PHP/comments/1l7baq/creating_a_user_from_the_web_problem/
628 2013-09-02 15:35:53 <TheLordOfTime> sipa: note this isn't a fully synced node and it's quite possible it got overwritten somehow :/
629 2013-09-02 15:36:03 <sipa> ?
630 2013-09-02 15:36:06 <TheLordOfTime> but i don't know the structure of the logfile so...
631 2013-09-02 15:36:10 <gmaxwell> runeks: (I ask not just because of the general foolishness but because bitcoind as an rpc client is really really slow)
632 2013-09-02 15:36:10 <TheLordOfTime> we'll see what happens.
633 2013-09-02 15:36:26 <gmaxwell> overwritten?
634 2013-09-02 15:36:34 <runeks> gmaxwell: Yeah I've come to realize that.
635 2013-09-02 15:36:47 <runeks> Would be nice if there was some way of speeding it up.
636 2013-09-02 15:37:01 <gmaxwell> runeks: as a _client_ you should not be using it as a client thats why I was asking.
637 2013-09-02 15:37:23 <gmaxwell> hahahaha nice link.
638 2013-09-02 15:37:27 <runeks> gmaxwell: What do you mean exactly?
639 2013-09-02 15:37:38 <sipa> but i don't see what that has to do with your logfile being overwritten?
640 2013-09-02 15:37:38 <sipa> it may be broken
641 2013-09-02 15:37:38 <sipa> or there may be anothe rproblem with debian or the hardware where we've seen that problem observed
642 2013-09-02 15:37:38 <TheLordOfTime> sipa: were you not here when gmaxwell informed me the debian unstable package is broken and i started going on the warpath to get it removed from Debian and Ubuntu?
643 2013-09-02 15:37:39 <TheLordOfTime> sipa: as i said i don't know
644 2013-09-02 15:37:40 <runeks> gmaxwell: Yeah :) notsureifserious.jpg
645 2013-09-02 15:37:47 <TheLordOfTime> sipa: in any case let's wait to see what debug.log says
646 2013-09-02 15:38:25 <sipa> runeks: how are you doing the getblock RPC call?
647 2013-09-02 15:38:36 <gmaxwell> runeks: when you type "bitcoind getinfo" you are using the bitcoind executable as an rpc client. This is slow because it loads and excutes a multimegabyte binary with multiple megabytes of libraries... and god knows what C++ magic... just to make a single rpc call.
648 2013-09-02 15:38:45 <runeks> sipa: Using jsonrpc in Python.
649 2013-09-02 15:38:51 <sipa> ok
650 2013-09-02 15:38:53 <sipa> you're good
651 2013-09-02 15:39:10 <gmaxwell> okay, good for that issue, jsonrpc in python is also really slow but I don't know if it would effect the getblock call.
652 2013-09-02 15:39:13 <runeks> gmaxwell: Oh, I see. I thought you were talking about the RPC interface in general.
653 2013-09-02 15:39:19 <TD> in fairness, that execute means adjusting some kernel memory maps and hitting "go" on a new thread/address space. it's not necessarily slow just because the binary is large
654 2013-09-02 15:39:37 <runeks> The binary is probably already in memory.
655 2013-09-02 15:39:38 <gmaxwell> TD: and running zillions of constructors.
656 2013-09-02 15:39:53 <gmaxwell> it's actually very slow. go time it instead of arguing with me on IRC. :P
657 2013-09-02 15:40:04 <michagogo> ACTION wonders if there's a rubygem that is a bitcoin jsonrpc client
658 2013-09-02 15:40:24 <TD> 1290.00000000
659 2013-09-02 15:40:24 <TD> real
660 2013-09-02 15:40:24 <TD> time ./bitcoind -regtest getbalance
661 2013-09-02 15:40:39 <michagogo> -regtest?
662 2013-09-02 15:40:54 <gmaxwell> yea, so you're limited to < 100 requests per second right there.
663 2013-09-02 15:41:07 <jrmithdobbs> TD: now time curl to the same interface
664 2013-09-02 15:41:26 <sipa> why would loading the curl binary be faster? :D
665 2013-09-02 15:41:32 <runeks> Fetching 100 hex blocks over RPC takes me 81.294925 seconds. They're probably like 200 KB each, so that's 250 KB/s over localhost :\
666 2013-09-02 15:41:33 <jrmithdobbs> it's smaller
667 2013-09-02 15:41:54 <sipa> runeks: wow
668 2013-09-02 15:41:57 <gmaxwell> (77 req/s I guess)
669 2013-09-02 15:41:58 <sipa> that's very slow
670 2013-09-02 15:42:02 <gmaxwell> runeks: what thats busted.
671 2013-09-02 15:42:21 <runeks> That's slower than my ADSL line...
672 2013-09-02 15:42:45 <gmaxwell> time (for i in {251378..251478} ; do export bh=`./bitcoind getblockhash $i` ; for tx in `./bitcoind getblock $bh | grep ' "' | head -1 |cut -d'"' -f2` ; do ./bitcoind getrawtransaction $tx 1 | grep value | cut -d':' -f2 | cut -d ',' -f1 | awk '{aa+=$1} END {print aa}' ; done ; done)
673 2013-09-02 15:42:51 <gmaxwell> real 0m5.945s
674 2013-09-02 15:42:53 <gmaxwell> user 0m2.508s
675 2013-09-02 15:42:56 <gmaxwell> sys 0m0.958s
676 2013-09-02 15:43:24 <gmaxwell> three calls to bitcoind to use the rpc... for 100 blocks and it takes <6 seconds.
677 2013-09-02 15:43:38 <runeks> And it's not the general RPC calls that are slow. I do a getblockhash() right before every getblock() and it only takes 5 ms.
678 2013-09-02 15:44:02 <sipa> what else is running on that system?
679 2013-09-02 15:44:04 <sipa> what disk?
680 2013-09-02 15:44:06 <TheLordOfTime> sipa: how many lines do you want from the end of debug.log
681 2013-09-02 15:44:25 <TheLordOfTime> 1,087,376 lines so far...
682 2013-09-02 15:44:35 <runeks> sipa: SanDisk SDSSDH2256G SSD
683 2013-09-02 15:44:39 <gmaxwell> runeks: can you just try my commandline there to see if its your calling enviroment or not?
684 2013-09-02 15:45:17 <runeks> gmaxwell: real
685 2013-09-02 15:45:50 <gmaxwell> okay, go fix your python enviroment. :P
686 2013-09-02 15:46:25 <runeks> Odd...
687 2013-09-02 15:46:38 <TheLordOfTime> gmaxwell: how many lines should I grab from the end of debug.log?
688 2013-09-02 15:46:41 <gmaxwell> first rule of computers: everything is crap.
689 2013-09-02 15:46:46 <TheLordOfTime> or do you want all 1.1 million lines.
690 2013-09-02 15:46:54 <gmaxwell> TheLordOfTime: enough to see where it went wrong. :P
691 2013-09-02 15:47:05 <gmaxwell> last 10000 or whatever should be fine.
692 2013-09-02 15:49:05 <TheLordOfTime> gmaxwell: how about the 456 from launch of bitcoind today which is from start to end where it ifailed
693 2013-09-02 15:49:42 <sipa> but i don't think there'll be much to see
694 2013-09-02 15:49:42 <sipa> well ideally we'd see the lines at the end of the the previous run (which still worked)
695 2013-09-02 15:49:45 <gmaxwell> need the logs from just before it went down too.
696 2013-09-02 15:50:09 <gmaxwell> yea, not likely to learn much we don't know now, but some confirmation would be good.
697 2013-09-02 15:50:22 <gmaxwell> I'm pretty sure I know what happened here (what sipa said above)
698 2013-09-02 15:50:57 <TheLordOfTime> sipa: i can pull some data from there
699 2013-09-02 15:51:22 <TheLordOfTime> gmaxwell: it went down after the VM was shut down and then loaded again this morning, so it was trying to pull what was on disk i guess
700 2013-09-02 15:51:24 <TheLordOfTime> give me a minute
701 2013-09-02 15:52:10 <sipa> hmmm... half of my vps's disk space is bitcoind log files
702 2013-09-02 15:54:26 <TheLordOfTime> sipa: gmaxwell: ehhh nothing supports 10000+ lines... :;/
703 2013-09-02 15:55:02 <sipa> tail -n 10000 ~/.bitcoin/debug.log | gzip -9 >foobar.log.gz
704 2013-09-02 15:55:10 <TheLordOfTime> not what i meant
705 2013-09-02 15:55:12 <TheLordOfTime> nowhere to paste it
706 2013-09-02 15:55:14 <gmaxwell> TheLordOfTime: hm? tail -10000 debug.log | xz -dc - > foo.xz
707 2013-09-02 15:55:17 <TheLordOfTime> nowhere to upload it to
708 2013-09-02 15:55:51 <gmaxwell> TheLordOfTime: presumably it's only a few hundred lines from the failure to the last lines before it failed?
709 2013-09-02 15:56:29 <TheLordOfTime> gmaxwell: it pastes about 12000 lines to gist, 9000 to pastebin before they fail...
710 2013-09-02 15:56:53 <TheLordOfTime> one moment here...
711 2013-09-02 15:57:01 <TheLordOfTime> *kicks his SSH and tries to get into one of his web servers*
712 2013-09-02 15:57:58 <TheLordOfTime> gmaxwell: E: format not recognized with your command
713 2013-09-02 15:58:23 <gmaxwell> sorry leave off the -d
714 2013-09-02 16:00:29 <TheLordOfTime> gmaxwell: sipa: i have a gzipped version now, but you'll have to download it and gunzip it
715 2013-09-02 16:00:50 <TheLordOfTime> http://lordoftime.info/debug.log.gz if you want it (i was able to get in because i allow passcode auth for my user on that server)
716 2013-09-02 16:00:55 <TheLordOfTime> last 10000 lines
717 2013-09-02 16:02:18 <TheLordOfTime> the web server doesn't serve gzipped data very well :/
718 2013-09-02 16:03:40 <sipa> gmaxwell: can you deal with this; i'm still on a train :)
719 2013-09-02 16:04:28 <gmaxwell> uh
720 2013-09-02 16:04:30 <gmaxwell> hm
721 2013-09-02 16:04:35 <gmaxwell> this isn't what I .. ah.
722 2013-09-02 16:04:40 <gmaxwell> okay weird.
723 2013-09-02 16:05:03 <gmaxwell> prior run ends with:
724 2013-09-02 16:05:03 <gmaxwell> received block 00000000000006de1735ac51cb5168d5d5edd1a7f86425a83eb66cdcb3550424
725 2013-09-02 16:05:03 <gmaxwell> received: block (7316 bytes)
726 2013-09-02 16:05:06 <gmaxwell> SetBestChain: new best=00000000000006de1735ac51cb5168d5d5edd1a7f86425a83eb66cdcb3550424 height=139919 log2_work=65.969005 tx=1226744 date=2011-08-07 03:21:29 progress=0.018441
727 2013-09-02 16:05:11 <gmaxwell> ProcessBlock: ACCEPTED
728 2013-09-02 16:05:13 <gmaxwell> ^@^@^@^@^@^@^@^
729 2013-09-02 16:05:34 <gmaxwell> Rescanning last 111 blocks (from block 139402)...
730 2013-09-02 16:05:37 <gmaxwell> is successful
731 2013-09-02 16:05:44 <gmaxwell> then SetBestChain: new best=00000000000006f69e5c8394cc9e3916745b2d9f1a8b05999403273e5cc895ad height=139514 log2_work=65.901831 tx=1203613 date=2011-08-04 00:17:20 progress=0.018055
732 2013-09-02 16:06:08 <gmaxwell> which continues until
733 2013-09-02 16:06:10 <gmaxwell> SetBestChain: new best=00000000000006de1735ac51cb5168d5d5edd1a7f86425a83eb66cdcb3550424 height=139919 log2_work=65.969005 tx=1226744 date=2011-08-07 03:21:29 progre
734 2013-09-02 16:06:13 <gmaxwell> ss=0.018402
735 2013-09-02 16:06:15 <gmaxwell> ERROR: CheckProofOfWork() : nBits below minimum work
736 2013-09-02 16:06:18 <gmaxwell> ERROR: CBlock::ReadFromDisk() : errors in block header
737 2013-09-02 16:06:20 <gmaxwell> *** Failed to read block
738 2013-09-02 16:06:51 <sipa> is that just SetBestChain lines, with nothing in between?
739 2013-09-02 16:07:04 <gmaxwell> yep.
740 2013-09-02 16:07:08 <sipa> ok
741 2013-09-02 16:07:18 <gmaxwell> and then
742 2013-09-02 16:07:19 <gmaxwell> Flush(false)
743 2013-09-02 16:07:19 <gmaxwell> init message: Loading addresses...
744 2013-09-02 16:07:19 <gmaxwell> Loaded 9038 addresses from peers.dat 48ms
745 2013-09-02 16:07:19 <gmaxwell> wallet.dat refcount=0
746 2013-09-02 16:07:20 <sipa> so the chainstate was last written at 139514
747 2013-09-02 16:07:21 <gmaxwell> wallet.dat checkpoint
748 2013-09-02 16:07:24 <gmaxwell> wallet.dat detach
749 2013-09-02 16:07:26 <gmaxwell> wallet.dat closed
750 2013-09-02 16:07:29 <gmaxwell> DBFlush(false) ended 22ms
751 2013-09-02 16:07:31 <gmaxwell> StopNode()
752 2013-09-02 16:07:34 <gmaxwell> but without timestamps, I'm not sure how long it sat there.
753 2013-09-02 16:07:38 <sipa> but the blockindex and the blocks on disk ere written further
754 2013-09-02 16:07:40 <sipa> including a block index entry for a block that never manged to get on disk
755 2013-09-02 16:07:48 <sipa> please don't paste such long chunks
756 2013-09-02 16:07:55 <gmaxwell> sorry. :) And yep.
757 2013-09-02 16:08:01 <sipa> actually, we could deal perfectly with that case
758 2013-09-02 16:08:14 <sipa> by just stopping the auto-reconnect if it fails
759 2013-09-02 16:08:17 <gmaxwell> sure. that particular one isn't hard. just recover when it fails.
760 2013-09-02 16:08:55 <sipa> and usually, whenever we write *anything* to the chainstate, blocks and blockindex are synced to disk
761 2013-09-02 16:09:28 <sipa> c'mon you pulltester
762 2013-09-02 16:10:21 <runeks> Looks like the reason for the poor RPC performance was because it used a new connection for every call. With bitcoinrpc I fetch those 100 blocks in 7.5 seconds.
763 2013-09-02 16:10:31 <gmaxwell> TheLordOfTime: reindex will set you right, and then you can see if it makes it to the point where it fails for everyone else.. your vm is multi-core, I hope?
764 2013-09-02 16:11:08 <sipa> runeks: are you doing multiple calls in parallel?
765 2013-09-02 16:11:20 <runeks> sipa: Nope.
766 2013-09-02 16:11:22 <gmaxwell> runeks: careful with the bitcoinrpc, it runs every JSON 'number' through decimal() which creates performance problems for some rpcs.. not getblock though.
767 2013-09-02 16:11:40 <TheLordOfTime> gmaxwell: unfortunately not, this system is a dual core processor, one core has to remain available for the actual host system, one for the VM
768 2013-09-02 16:11:46 <sipa> runeks: is there huge variance in time per rpc call
769 2013-09-02 16:11:47 <TheLordOfTime> gmaxwell: but it's got the entire 1 core dedicated to it.
770 2013-09-02 16:11:55 <runeks> gmaxwell: Yeah I initially opted out of it because it did that. But I'm just getting the block data (hex) so it shouldn't matter as you say.
771 2013-09-02 16:11:55 <sipa> runeks: or are they all just on average slow
772 2013-09-02 16:11:58 <gmaxwell> TheLordOfTime: oh, well I think thats no chance of reproducing that reported debian unstable glitch.
773 2013-09-02 16:12:10 <TheLordOfTime> gmaxwell: then we're not going to be able to reproduce until i get a new system
774 2013-09-02 16:12:18 <TheLordOfTime> which is quad core, and i'll put 2 cores into the VM
775 2013-09-02 16:12:20 <gmaxwell> runeks: New connections doesn't quite explain it.. since the cli was doing that too.. (actually three of them per block)
776 2013-09-02 16:12:43 <gmaxwell> TheLordOfTime: your vm enviroment won't let you use as many cpus as the host??
777 2013-09-02 16:13:00 <TheLordOfTime> gmaxwell: i can let it but then i get a huge freeze up on both VM and host
778 2013-09-02 16:13:13 <TheLordOfTime> this isn't a high powered system :/
779 2013-09-02 16:13:25 <gmaxwell> ah okay
780 2013-09-02 16:13:41 <gmaxwell> well one good bug reproduced in any case.
781 2013-09-02 16:13:43 <runeks> gmaxwell: Good point. Then I can't explain it.
782 2013-09-02 16:13:59 <runeks> sipa: I'll check the variance... gimme a minut
783 2013-09-02 16:14:23 <sipa> runeks: mainly i want to know, are they all fast, but are there sometimes multi-second lockups
784 2013-09-02 16:14:32 <sipa> or are the cals just really 1s per call
785 2013-09-02 16:14:36 <sipa> *calls
786 2013-09-02 16:14:48 <runeks> sipa: Well it's better now as I mentioned. around 75 ms per call.
787 2013-09-02 16:14:58 <sipa> "now" ?
788 2013-09-02 16:15:00 <sipa> what changed?
789 2013-09-02 16:15:12 <runeks> sipa: bitcoinrpc instead of jsonrpc
790 2013-09-02 16:15:37 <sipa> can you go back to jsonrpc then for a second, and check this?
791 2013-09-02 16:15:55 <sipa> it may be a lead for the weird lockups/deadlocks some people experience with rpc
792 2013-09-02 16:15:59 <runeks> sipa: I can do that yeah.
793 2013-09-02 16:16:13 <sipa> if you experience a slowdown with one client library and not with another, that's a big hint
794 2013-09-02 16:16:17 <runeks> By the way, running ./bitcoind is still 2x faster than bitcoinrpc
795 2013-09-02 16:16:29 <sipa> heh??
796 2013-09-02 16:16:41 <sipa> that's utterly ridicuous
797 2013-09-02 16:18:29 <runeks> Yeah I don't get it :\
798 2013-09-02 16:18:46 <sipa> can i see your code?
799 2013-09-02 16:19:27 <runeks> Max: 0.215998888016
800 2013-09-02 16:19:27 <runeks> Min: 0.00325512886047