1 2012-07-07 00:11:17 <luke-jr> BlueMatt: don't ask me how, but that causes shutdown to segfault on my mainnet wallet: 081d73f (HEAD, refs/bisect/bad) Remove costly uint256/CBigNum::ToString() from SetBestChain.
2 2012-07-07 01:52:50 <luke-jr> FWIW, Eligius's last block hit #bitcoin-watch almost instantly (4 seconds), but took 3 minutes to get to my local client
3 2012-07-07 01:53:31 <gmaxwell> crazy.
4 2012-07-07 01:54:54 <gmaxwell> luke-jr: perhaps we should make the log entries have microsecond timestamps, and start collecting on a bunch of nodes.
5 2012-07-07 01:55:13 <luke-jr> gmaxwell: I'm not sure how accurate that could be
6 2012-07-07 01:55:34 <luke-jr> I mean sure, we can use GetAdjustedTime, but that probably doesn't keep the whole network within microseconds
7 2012-07-07 01:55:58 <gmaxwell> healthy ntp gets subsection. but more importantly your times for all your peer's invs will only have a single unknown offset (your local time)
8 2012-07-07 01:56:03 <luke-jr> (it's easy for me to keep Eligius, my IRC client, and my Bitcoin client in sync)
9 2012-07-07 01:56:04 <gmaxwell> bitcoin time sucks.
10 2012-07-07 01:56:30 <gmaxwell> I don't mean for the whole network. I mean a handful of technical folks (e.g. people in this channel, some pool ops)
11 2012-07-07 01:56:41 <luke-jr> i c
12 2012-07-07 01:56:43 <gmaxwell> See if we can see any pattern to the propagation delays.
13 2012-07-07 01:56:50 <luke-jr> 2012-07-07 03:49:34,773 Waker for BitcoinNode DEBUG Read wakeup
14 2012-07-07 01:57:01 <luke-jr> that was for 00000000000005c702f10bacad8f612266264b392cd6eaad86bead687bc95b82
15 2012-07-07 01:57:18 <luke-jr> [03:49:38] <ljrbot> Blk 00000000000005c702f10bacad8f612266264b392cd6eaad86bead687bc95b82: 1HkJnteXo5b8aFFa9zEipYtwHASD9rYyXM 3.97766304 BTC, 1PyvPBpWUD1NwCuSppr6DCA3TrP9zwF9qR 2.37041814 BTC, 16wvk2GPpuoL3LD1B41VLunZB7oa9SeQyj 2.16759968 BTC, 1LS6JBSyWKp3RdzAWdNHhdCGCGNEcK1Wwa 2.08380267 BTC, 1HWa3wHyHHb1s1KnbGGYWLhHJGc8eSw6M7
16 2012-07-07 01:58:21 <luke-jr> hmm, my debug.log hit 2 GB so is broken
17 2012-07-07 09:25:24 <sipa> gmaxwell: during IBD of 140k blocks, the hash of 2454233 distinct transactions was calculated, and each on average 3.6 times
18 2012-07-07 09:25:46 <sipa> (this was by simply adding a mutable hash field to CTransaction)
19 2012-07-07 09:28:05 <jrmithdobbs> 3.6 times each? damn
20 2012-07-07 09:28:12 <jrmithdobbs> why the rework?
21 2012-07-07 09:33:32 <sipa> and its hash is requested in several places, it seems
22 2012-07-07 09:37:39 <jrmithdobbs> sipa: makes sense
23 2012-07-07 09:42:39 <sipa> but i don't see why it can more than 2
24 2012-07-07 09:45:06 <Kiba> whoops
25 2012-07-07 09:45:11 <Kiba> wrong channel
26 2012-07-07 09:47:45 <jrmithdobbs> sipa: third is a re-recv that stops processing when it knows i[D[D[D[D[D[D[D[D[D[D[D[D[D[D[D[D[D[D[D[D[D[D[D[D[D[D[D[D[D[D[D[D[D[D[Dt already has it maybe?
27 2012-07-07 09:47:51 <jrmithdobbs> whoah
28 2012-07-07 09:48:11 <jrmithdobbs> lets try that again. how the hell is irssi so shitty with delayed input
29 2012-07-07 09:48:35 <sipa> jrmithdobbs: no, got it; it was in gmaxwell's profile image
30 2012-07-07 09:48:36 <jrmithdobbs> sipa: maybe the third+ are re-processing from another receive of the block/txn from a diff peer?
31 2012-07-07 09:48:50 <jrmithdobbs> sipa: ?
32 2012-07-07 09:48:53 <sipa> jrmithdobbs: this is just IBD from a local file on disk
33 2012-07-07 09:48:57 <CodesInChaos> does it really matter performance wise?
34 2012-07-07 09:48:58 <jrmithdobbs> ah
35 2012-07-07 09:49:11 <sipa> CodesInChaos: apparently 40% of CPU is spent on tx hashing
36 2012-07-07 09:49:14 <jrmithdobbs> CodesInChaos: yes. a lot. this is the biggest perf bottleneck in the client and has been for a while
37 2012-07-07 09:49:18 <CodesInChaos> I'd expect ECC to be *much* slower than hashing
38 2012-07-07 09:49:28 <sipa> CodesInChaos: sure, but during IBD signatures aren't checked
39 2012-07-07 10:36:38 <BlueMatt> luke-jr: meh
40 2012-07-07 10:36:53 <BlueMatt> luke-jr: feel free to fix it
41 2012-07-07 10:48:19 <sipa> gmaxwell: caching the tx hashes doesn't seem to make any difference to reduce IBD time ultraprune, though
42 2012-07-07 10:55:42 <CodesInChaos> is that process CPU or IO bound?
43 2012-07-07 10:56:38 <sipa> CPU certainly isn't maxed out here, though there are bursts, most likely
44 2012-07-07 11:39:09 <sipa> gmaxwell: on tmpfs on the same system, 5m19s with hash cashing to 185k
45 2012-07-07 11:39:58 <sipa> caching!
46 2012-07-07 11:40:00 <jrmithdobbs> nice
47 2012-07-07 11:42:07 <sipa> remarkably, reading a file from an ext3 fs, after having the OS cache it, is faster than reading from tmpfs
48 2012-07-07 11:42:32 <jrmithdobbs> sipa: not really, tmpfs is backed by the same cache and the code isn't looked at as much
49 2012-07-07 11:42:51 <sipa> true
50 2012-07-07 11:42:56 <jrmithdobbs> not really remarkable, i mean
51 2012-07-07 11:50:46 <sipa> gmaxwell: without txhash caching: 7m1s for 185 blocks
52 2012-07-07 11:50:49 <sipa> * 185k
53 2012-07-07 11:51:13 <sipa> looks i need to optimize IO first
54 2012-07-07 11:51:25 <jrmithdobbs> sipa: so with this tor update i see one other minor thing
55 2012-07-07 11:52:30 <jrmithdobbs> sipa: there any way to replace the local socket connection (usually 127.0.0.1 or similar) src addr in the peer list for anyone connected via hidden service that is advertising something else?
56 2012-07-07 11:52:39 <jrmithdobbs> in getpeerlist
57 2012-07-07 11:52:55 <sipa> not really
58 2012-07-07 11:53:03 <jrmithdobbs> i think it'd require reworking how that's stored significantly when i looked
59 2012-07-07 11:53:19 <sipa> what would you want to see instead?
60 2012-07-07 11:53:27 <sipa> the addrFrom claimed by the peer?
61 2012-07-07 11:54:23 <jrmithdobbs> if available, ya
62 2012-07-07 11:54:57 <jrmithdobbs> i'm not sure really, thinking about it, seeing the loopback as the addr is just weird
63 2012-07-07 11:57:31 <sipa> it wouldn't be hard to add a field to CNode to retain the peer's addrFrom
64 2012-07-07 11:57:44 <sipa> anyone an idea what 'DOB' would mean?
65 2012-07-07 11:58:03 <jrmithdobbs> date of birth?
66 2012-07-07 11:58:17 <sipa> ...
67 2012-07-07 11:58:27 <orion> 14:52:14 < sipa> go to sync.h, and change the ifdef osx there to bsd
68 2012-07-07 11:58:40 <orion> sipa: There is no src/sync.h in 0.6.3.
69 2012-07-07 11:58:41 <sipa> orion: it may be in util.h still in older code
70 2012-07-07 11:59:03 <sipa> jrmithdobbs: i should have known
71 2012-07-07 12:19:05 <jrmithdobbs> sipa: also https://github.com/bitcoin/bitcoin/pull/1565
72 2012-07-07 12:19:18 <gribble> New news from bitcoinrss: jrmithdobbs opened pull request 1565 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/pull/1565>
73 2012-07-07 12:20:12 <sipa> jrmithdobbs: merged
74 2012-07-07 12:22:02 <jrmithdobbs> apparently my github ssh keys needed to be verified was why it wasn't letting me update earlier this week (it wasn't giving me the message at first, ha)
75 2012-07-07 12:22:45 <sipa> yeah, there was some security breach at github some months ago
76 2012-07-07 12:26:13 <jrmithdobbs> ya i checked at the time, apparently before they added that measure
77 2012-07-07 12:39:49 <gribble> New news from bitcoinrss: Diapolo opened pull request 1566 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/pull/1566>
78 2012-07-07 12:55:19 <MC1984> yo guys
79 2012-07-07 12:56:03 <MC1984> is it normal for bitcoin to be writing and reading tens of gigabytes while dling the blockchain
80 2012-07-07 12:56:56 <MC1984> im getting a bit worried about if its raping my SSD to death
81 2012-07-07 12:57:03 <sipa> for now, yes, unfortunately
82 2012-07-07 12:59:46 <luke-jr> MC1984: SSD can be read unlimited times AFAIK
83 2012-07-07 13:00:36 <MC1984> yeah but in task manager its writing about 2.5GB for 1000 blocks
84 2012-07-07 13:00:50 <MC1984> about 3gb read
85 2012-07-07 13:01:41 <MC1984> so thats prob multiple hundreds of GB of write ops for a full chain download thus far
86 2012-07-07 13:01:51 <jrmithdobbs> MC1984: it's really really bad from ~4/1/11 onwards ;p
87 2012-07-07 13:02:11 <MC1984> the speed isnt too bad now, much better than it used to be
88 2012-07-07 13:02:22 <MC1984> but im concerned about the write ops on an SSD
89 2012-07-07 13:02:38 <MC1984> i wonder if bitcoin will end up pushing the limits of SSD tech at this rate
90 2012-07-07 13:02:49 <jrmithdobbs> no
91 2012-07-07 13:02:58 <CodesInChaos> why would it write more than the total size of the blockchain?
92 2012-07-07 13:03:07 <MC1984> i have no idea
93 2012-07-07 13:03:24 <CodesInChaos> What column in the taskmanager are you talking about?
94 2012-07-07 13:03:38 <MC1984> unless this I/O read/write bytes column included memory ops too
95 2012-07-07 13:03:44 <CodesInChaos> I believe some IO related columns include data you write to sockets, internal pipes etc
96 2012-07-07 13:04:20 <MC1984> i assumed I/O referred to the storage only
97 2012-07-07 13:04:28 <luke-jr> CodesInChaos: because of the stupid database
98 2012-07-07 13:05:33 <CodesInChaos> MC1984: You could run Resource Monitor, I think it'll tell you correctly how much data is written to the disk
99 2012-07-07 13:05:51 <CodesInChaos> (It's a bit buggy though if you're using TrueCrypt)
100 2012-07-07 13:07:36 <CodesInChaos> hmm strange there is really a factor 10 between network activity and disk writes
101 2012-07-07 13:08:45 <jrmithdobbs> yes
102 2012-07-07 13:14:37 <sipa> CodesInChaos: it writes the blockchain once
103 2012-07-07 13:14:38 <MC1984> ok its writing to disk at a steady 1.1mb/second
104 2012-07-07 13:14:51 <sipa> CodesInChaos: but the block index is continuously updated and flushed
105 2012-07-07 13:15:39 <MC1984> and its actually d/ling the chain @ 100kb/sec
106 2012-07-07 13:16:04 <MC1984> so the write ops outstrip the d/ling by 10:1
107 2012-07-07 13:16:25 <jrmithdobbs> 10:08 < jrmithdobbs:#bitcoin-dev> yes
108 2012-07-07 13:16:33 <sipa> it should be a lot less once we switch from BDB to LevelDB
109 2012-07-07 13:17:47 <MC1984> cool
110 2012-07-07 13:18:04 <MC1984> also, it definately got stuck download the chain a while ago
111 2012-07-07 13:18:17 <MC1984> restarted bitcoin and it carried on
112 2012-07-07 13:18:57 <jrmithdobbs> stuck how?
113 2012-07-07 13:19:27 <MC1984> block counter stopped, cpu low, d/l low, disk ops low
114 2012-07-07 13:19:34 <MC1984> thumb up its ass
115 2012-07-07 13:23:57 <jrmithdobbs> dunno, i just started a full sync off HEAD after sipa merged that socks fix (wiped data dir completely) with v4/v6/tor peers and i'm up to ~141804 after ~50 minutes
116 2012-07-07 13:24:31 <jrmithdobbs> it's sat and spun cpu a few times but the logs have never stopped for more than 30-90 seconds on some of those huge horrible blocks from last summer
117 2012-07-07 13:26:42 <jrmithdobbs> you sure you didn't maybe have a network event that dropped your connection and didn't recover right or something?
118 2012-07-07 13:27:01 <jrmithdobbs> eg, dhcp ip change or similar
119 2012-07-07 13:58:00 <gmaxwell> 05:48 < sipa> gmaxwell: caching the tx hashes doesn't seem to make any difference to reduce IBD time ultraprune, though
120 2012-07-07 13:58:36 <gmaxwell> dunno if you notice but gethash, according to the callgrind, was actuall burning a lot of its time in serlization/heap manipulation rather than the actual hash.
121 2012-07-07 14:00:06 <gmaxwell> https://people.xiph.org/~greg/ultraprune_profile.png < follow the chain to gethash() the SHA256 looks like only about 10% (the numbers there are this+children out of the total)
122 2012-07-07 14:05:30 <MC1984> jrmithdobbs you could be right
123 2012-07-07 14:07:42 <luke-jr> gmaxwell: so GetSerializeSize sucks bad?
124 2012-07-07 14:20:05 <sipa> gmaxwell: sure, but caching prevents both from happening repeatedly
125 2012-07-07 14:20:44 <sipa> gmaxwell: anyhow, i've been thinking about a SHA256Writer, which could be used as destination for the serialize framework, and hashes in-place; instead of needing to allocate an output buffer
126 2012-07-07 14:22:06 <sipa> seems strange that something like that isn't already used
127 2012-07-07 14:34:59 <jrmithdobbs> sipa: your dns seed is down again
128 2012-07-07 14:49:22 <jrmithdobbs> bitcoind seems to validate slightly quicker on blockchain download when built with clang syncing over the network, but it's doing something that backs up the runqueue more than it does when built with gcc 4.6
129 2012-07-07 15:31:27 <D34TH> hey guys, can i bounce something off of you real quick
130 2012-07-07 15:31:57 <D34TH> bitcoin-qt.pro:282 would that be why it couldnt properly compile?
131 2012-07-07 15:32:05 <D34TH> on windows
132 2012-07-07 15:34:47 <luke-jr> obviously that depends on your error
133 2012-07-07 15:35:15 <D34TH> buttload of boost errors
134 2012-07-07 15:35:28 <D34TH> because doesnt bitcoin require 1.47
135 2012-07-07 15:35:47 <luke-jr> 1.49 now
136 2012-07-07 15:35:49 <luke-jr> for Windows
137 2012-07-07 15:35:54 <luke-jr> with a custom patch
138 2012-07-07 15:36:02 <D34TH> also, it breaks on 1.50
139 2012-07-07 15:36:13 <D34TH> just doesnt even start
140 2012-07-07 15:49:45 <sipa> jrmithdobbs: thanks, i didnt notice it was down
141 2012-07-07 15:56:52 <jrmithdobbs> 2012-07-07_17:54:03.05073 send version message: version 60001, blocks=181305, us=deezs52qlm2tdbtc.onion:8333, them=0.0.0.0:0, peer=127.0.0.1:52664
142 2012-07-07 15:56:55 <jrmithdobbs> 2012-07-07_17:54:03.05074 receive version message: version 60001, blocks=187734, us=deezs52qlm2tdbtc.onion:8333, them=0.0.0.0:0, peer=127.0.0.1:52664
143 2012-07-07 16:05:31 <phantomcircuit> jrmithdobbs, working?
144 2012-07-07 16:06:02 <jrmithdobbs> ya that client sync'ed over tor completely
145 2012-07-07 16:16:16 <freewil> does running bitcoin over tor make it easier for an attacker to isolate your network
146 2012-07-07 16:16:51 <freewil> ... and run a sybil attack
147 2012-07-07 17:42:57 <gmaxwell> freewil: "it depends"
148 2012-07-07 17:43:36 <gmaxwell> freewil: but bitcoin is resistant to sybil attacks. ::shrugs:: they're mostly just dos attacks unless you're inattentive and they have a lot of computing power.
149 2012-07-07 17:44:36 <MC1984> yep, this little machine is now choking on a million dicks
150 2012-07-07 17:44:45 <MC1984> i mean satoshi dice blocks
151 2012-07-07 17:48:53 <freewil> gmaxwell, alright thanks
152 2012-07-07 17:50:52 <gmaxwell> freewil: by 'it depends' I mean that it moves the vulnerability. Tor becomes an absolute defense against an attack by someone with control over your network but no control outside. But otoh it may make an attack easier for someone who can't control your network but can control exit nodes. If you have addnoded onion peers which aren't attacker controlled you're back to absolute defense.
153 2012-07-07 17:56:32 <freewil> hmm i see
154 2012-07-07 17:57:07 <freewil> does the attacker even have to control exit nodes
155 2012-07-07 17:57:19 <freewil> or could they just be relays
156 2012-07-07 18:00:49 <freewil> ok nevermind
157 2012-07-07 18:01:02 <freewil> i guess the link from the source to the exit node is encrypted
158 2012-07-07 18:02:02 <sipa> yes
159 2012-07-07 18:06:32 <freewil> so could one say that the best solution for someone not overly concerned about anonymity is to run bitcoin using tor and non-tor connections as described in sipa's doc file?
160 2012-07-07 18:07:49 <freewil> ./bitcoin -tor=127.0.0.1:9050 -externalip=57qr3yd1nyntf5k.onion -discover
161 2012-07-07 18:09:06 <sipa> except that's not a valid onion address ;)
162 2012-07-07 18:09:21 <freewil> then why is it in the doc ;)
163 2012-07-07 18:40:17 <sipa> gmaxwell: with a specialized CHashWriter, 6m42s instead of 7m01s for 185k blocks; let me repeat to see how much variation there is
164 2012-07-07 18:48:44 <sipa> gmaxwell: ok, 6m41s this time
165 2012-07-07 18:51:02 <jrmithdobbs> hrm
166 2012-07-07 18:51:21 <jrmithdobbs> so it sync'ed up to blocks from end of may in 3 hours and has spinning getting current since
167 2012-07-07 18:51:29 <jrmithdobbs> s/has/has been/
168 2012-07-07 18:51:48 <jrmithdobbs> (another 3 hours)
169 2012-07-07 18:52:01 <sipa> git head has a checkpoint at 185333
170 2012-07-07 18:52:15 <sipa> so after that point it's obviously significantly slower
171 2012-07-07 18:54:55 <sipa> (that's june 19th)
172 2012-07-07 18:56:46 <sipa> ok, 6m36s now
173 2012-07-07 19:00:26 <jrmithdobbs> sipa: slowed down before then
174 2012-07-07 19:01:24 <sipa> when did SD start?
175 2012-07-07 19:05:16 <jrmithdobbs> oh, actually about the same time it slowed way the fuck down, ha
176 2012-07-07 19:05:32 <jrmithdobbs> that'd explain it, 32bit linux on a p4 prescott(-ish)
177 2012-07-07 19:06:09 <jrmithdobbs> so those tiny shit txns are probably slowing it way way down
178 2012-07-07 19:06:40 <jrmithdobbs> sipa: have you tested how much disabling the debug.log output (completely) during IBD affects things?
179 2012-07-07 19:07:19 <jrmithdobbs> esp recv/accept/reject/newlongest messages though
180 2012-07-07 19:07:43 <sipa> haven't tested that yet, no
181 2012-07-07 19:08:23 <jrmithdobbs> that seems to be the weird system performance affecting thing i noticed when built with clang vs gcc, lock/flushes related to stdout (-printtoconsole)
182 2012-07-07 19:08:43 <jrmithdobbs> eg it performs about the same but built with clang it affects the rest of the systems' responsiveness
183 2012-07-07 19:08:46 <jrmithdobbs> it's weird
184 2012-07-07 19:08:56 <jrmithdobbs> (runqueue is deeper)
185 2012-07-07 19:14:51 <gribble> New news from bitcoinrss: sipa opened pull request 1567 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/pull/1567>
186 2012-07-07 19:16:19 <midnightmagic> i wish tycho had done his asic after all :(
187 2012-07-07 19:16:23 <midnightmagic> oh well
188 2012-07-07 19:17:29 <sipa> why?
189 2012-07-07 19:34:14 <midnightmagic> sipa: Because a single vendor in any form is a bad thing IMO. Competition is crucial.
190 2012-07-07 19:35:21 <sipa> oh, sure
191 2012-07-07 19:35:41 <JFK911> thanks to tivo's insistence about "competition" after they avoided participating in standards development, you pay more for cable service and are hassled with cablecard.
192 2012-07-07 19:35:41 <sipa> but i didn't hear about tycho doing anything asic-related, and i wondered why him specifically
193 2012-07-07 19:36:11 <sipa> explain?
194 2012-07-07 19:36:18 <JFK911> they made this big stink after trying to be a worldwide dvr monopoly
195 2012-07-07 19:36:25 <sipa> (tivo doesn't exist here, and i don't watch tv)
196 2012-07-07 19:37:12 <JFK911> well, we developed standards for conditional access and then tivo complained that they couldn't be a part of it after refusing licensing talks from the companies that wanted to give dvr's to cable customers.
197 2012-07-07 19:38:10 <JFK911> they succeeded in getting a generation of equipment obsoleted, expecting operators would want to buy their equipment.
198 2012-07-07 19:38:14 <JFK911> they didn't.
199 2012-07-07 19:58:54 <sipa> my god
200 2012-07-07 19:59:16 <sipa> transaction hashes are already cached inside a block during verification... in vMerkleTree
201 2012-07-07 20:00:53 <Diapolo> What GCC version is used for official Win builds currently?
202 2012-07-07 20:02:19 <sipa> 4.4, i assume
203 2012-07-07 20:04:37 <Diapolo> I'm playing aroung with 4.7 and tried to compile Qt from source and all the required libs but got a non-working Bitcoin-Qt executable ... it so much freaky to do such things on Windows ^^. Documentation is worse as hell.
204 2012-07-07 21:05:16 <gmaxwell> http://blockexplorer.com/block/000000000000034e8d63e1894a4b86af7db2db129862063540defd5e0b1f3709 < lots of fees in this block, most from three transactions
205 2012-07-07 21:05:49 <gmaxwell> my guess is that the user attempted to make a single transaction and it failed because the result was >100k. I think we don't give a very informative failure for this.
206 2012-07-07 21:06:42 <gmaxwell> I guess he then increased his optional fees, but kept failing (of course). When he got to 0.1 BTC/kb he tried breaking up his transaction. This ~13 BTC in fees.
207 2012-07-07 21:07:57 <gmaxwell> I'm happy to have been paid on it, but not happy that somewhere there is someone who is angry at the fees.
208 2012-07-07 21:09:07 <sipa> hmm, yes, with the many small transaction outputs now roaming the bitcoin tubes, the hard transaction size limit gets reached
209 2012-07-07 21:10:13 <gmaxwell> s/This ~13/This resulted in ~13/
210 2012-07-07 21:52:28 <sipa> gmaxwell: strange, i managed to remove all duplicate tx hashing in ultraprune (verified with asserts), and got the 185k-block speed down to about 6 minutes, but still can't get it as low as the 5m20s i had before by just adding a cached hash value to CTransaction...
211 2012-07-07 23:51:57 <Diablo-D3> I thought we wernt logging this channel?
212 2012-07-07 23:57:25 <luke-jr> you're confusing it with #bitcoin
213 2012-07-07 23:57:44 <Diablo-D3> no, I thought this one wasnt logged either
214 2012-07-07 23:57:57 <Diablo-D3> gmaxwell went on some anti logging cruisade awhile back