1 2012-12-13 00:04:55 <sipa> seems to work!
2 2012-12-13 01:41:36 <Ferroh> 0.5 probability that a block will be found after 6.93 minutes. Does that seem right to you guys?
3 2012-12-13 01:41:39 <Ferroh> http://www.reddit.com/r/Bitcoin/comments/14r91u/the_reason_we_do_see_long_delays_between_blocks/
4 2012-12-13 01:41:40 <Ferroh> http://www.reddit.com/r/Bitcoin/comments/14r91u/the_reason_we_do_see_long_delays_between_blocks/
5 2012-12-13 01:49:29 <fiesh> Ferroh: https://en.wikipedia.org/wiki/Exponential_distribution#Mean.2C_variance.2C_moments_and_median
6 2012-12-13 01:49:51 <fiesh> Ferroh: it's all listed there, and yes, 6.93... = log(2)/10 is right for the median
7 2012-12-13 01:51:25 <fiesh> (which of course means that the computations in the reddit article are wrong, working with a halflife time of 10 minutes)
8 2012-12-13 01:51:44 <fiesh> oh it seems he fixed them
9 2012-12-13 02:04:26 <roconnor> so I was thinking of solving etotheipi's problem as follows. Let PubKey[i] = Hash(ChainCode[i])*PubKey[0] and PrivKey[i] = Hash(ChainCode[i])*PrivKey[0].
10 2012-12-13 02:04:50 <roconnor> one releases a signature of PubKey[0]
11 2012-12-13 02:05:37 <roconnor> Call Hash(ChainCode[i]) := InvoiceID[i]
12 2012-12-13 02:06:06 <roconnor> then release (InvoiceID[i], PubKey[i]) to your client (though PubKey[i] can be derived).
13 2012-12-13 02:06:41 <roconnor> the client can verify that PubKey[i] = InvoiceID[i]*PubKey[0] and that PubKey[0] is properly signed.
14 2012-12-13 02:07:51 <roconnor> this means the client knows that only someone who effectively knows PrivKey[0] could receive funds to PubKey[i].
15 2012-12-13 02:09:01 <roconnor> And the Hash prevent the client from deriving ChainCode[i], and discovering what other transactions belong to the PubKey[0].
16 2012-12-13 02:09:02 <roconnor> And the Hash prevent the client from deriving ChainCode[i], and discovering what other transactions belong to the PubKey[0].
17 2012-12-13 02:10:22 <roconnor> however a client can prove to the world that their transaction went to the owner of PrivKey[0].
18 2012-12-13 02:10:23 <roconnor> however a client can prove to the world that their transaction went to the owner of PrivKey[0].
19 2012-12-13 02:30:29 <gavinandresen> roconnor: https://bitcointalk.org/index.php?topic=130456 is relevant
20 2012-12-13 07:51:51 <m0mchil> sipa: still need windows tester?
21 2012-12-13 08:04:16 <sipa> m0mchil: yes, see http://bitcoin.sipa.be/builds/leveldb17
22 2012-12-13 08:05:14 <m0mchil> sipa: yes, was wondering if this is it... expected to see a 64 bit one too
23 2012-12-13 08:05:43 <sipa> no 64-bit for now
24 2012-12-13 08:08:01 <m0mchil> currently able to sanity test it in VM only, will do performance test later
25 2012-12-13 08:08:02 <m0mchil> currently able to sanity test it in VM only, will do performance test later
26 2012-12-13 08:11:26 <sipa> just getting it to compile in 32-bit already took enough :)
27 2012-12-13 08:11:27 <sipa> just getting it to compile in 32-bit already took enough :)
28 2012-12-13 08:19:06 <jeremias> Bitcoin team at coderwall: https://coderwall.com/team/bitcoin
29 2012-12-13 08:21:32 <jeremias> hmm, now noticed that this team at coderwall doesn't make much sense, it is aimed for companies
30 2012-12-13 08:27:27 <m0mchil> sipa: menu is shown, but not working (File, Settings, Help)... I think it was the same with 244-g3ccb06f
31 2012-12-13 08:27:28 <m0mchil> sipa: menu is shown, but not working (File, Settings, Help)... I think it was the same with 244-g3ccb06f
32 2012-12-13 08:28:03 <sipa> wut?
33 2012-12-13 08:28:32 <sipa> that's the last thing i would expect to fail
34 2012-12-13 08:28:33 <sipa> that's the last thing i would expect to fail
35 2012-12-13 08:29:06 <sipa> qt is just about the only thing that didn'tget upgraded
36 2012-12-13 08:29:29 <m0mchil> yes, seems strange
37 2012-12-13 08:30:45 <m0mchil> also, in options dialog (showing fine trough system tray menu) it keeps reverting to first tab 'Main'
38 2012-12-13 08:31:21 <m0mchil> does someone else reported this on windows 7?
39 2012-12-13 08:31:22 <m0mchil> does someone else reported this on windows 7?
40 2012-12-13 08:31:34 <sipa> 6?
41 2012-12-13 08:32:13 <sipa> i tried it priefly on win7 myself, but not thoroughly
42 2012-12-13 09:06:26 <gribble> Luke-Jr was last seen in #bitcoin-dev 15 hours, 39 minutes, and 12 seconds ago: <Luke-Jr> gmaxwell: of course they aren't. I don't see how that's even possible.
43 2012-12-13 09:06:26 <jine> ;;seen Luke-Jr
44 2012-12-13 09:06:27 <gribble> Luke-Jr was last seen in #bitcoin-dev 15 hours, 39 minutes, and 12 seconds ago: <Luke-Jr> gmaxwell: of course they aren't. I don't see how that's even possible.
45 2012-12-13 09:09:57 <gribble> theymos was last seen in #bitcoin-dev 2 weeks, 1 day, 8 hours, 38 minutes, and 27 seconds ago: <theymos> gmaxwell: Sounds like maybe some strange problem with BBE's server then. Much less interesting than a DoS attack. :( I'll have to watch to see if something like that happens again. Thanks for looking into it.
46 2012-12-13 09:09:57 <jine> ;;seen theymos
47 2012-12-13 12:08:54 <wumpus> BlueMatt: I don't think there are any problems with boost 1.52 an bitcoin, not that I remember at least
48 2012-12-13 12:17:06 <sipa> wumpus: good to know; i managed to compile a windows bitcoin-qt binary with leveldb 1.7, boost 1.52, mingw-w64 g++ 4.6.3
49 2012-12-13 12:17:07 <sipa> wumpus: good to know; i managed to compile a windows bitcoin-qt binary with leveldb 1.7, boost 1.52, mingw-w64 g++ 4.6.3
50 2012-12-13 12:20:51 <sipa> wumpus: https://bitcointalk.org/index.php?topic=129861.msg1397240#msg1397240
51 2012-12-13 12:24:04 <m0mchil> sipa: false alarm about the menus, the problem is only in the VM
52 2012-12-13 12:24:05 <m0mchil> sipa: false alarm about the menus, the problem is only in the VM
53 2012-12-13 12:25:26 <wumpus> ah now I remember, I managed to cross-compile it with the mingw-64 that comes with Ubuntu 10.04 once as well, it worked fine I just had some wine issues with boost interprocess (https://github.com/bitcoin/bitcoin/issues/1719), that was boost 1.51 though
54 2012-12-13 12:26:01 <wumpus> no reason to hold up a releast though I suppose no one except developers are running bitcoin in wine in practice
55 2012-12-13 12:26:26 <sipa> well, i think we want to get rid of boost::interprocess anyway
56 2012-12-13 12:27:48 <wumpus> yeah
57 2012-12-13 12:34:12 <sipa> m0mchil: good to know
58 2012-12-13 12:35:29 <m0mchil> when you think is best to simulate machine crash - during block importing?
59 2012-12-13 12:36:55 <m0mchil> also, i can't answer myself why is the improting of first 193000 block not faster, windows only... it seems there is only one trhead actively crunching, but I don't see IO maxed
60 2012-12-13 12:41:13 <sipa> m0mchil: if you pass a huge -dbcache (like a gigabyte or so), data will be written to leveldb in large batches as well (which take several seconds) and that gets written to debug.log - so a power failure exactly at that point is what you want :)
61 2012-12-13 12:41:36 <m0mchil> nice :)
62 2012-12-13 12:42:34 <m0mchil> will let current test complete and test this
63 2012-12-13 12:42:55 <m0mchil> any idea about first 193000 being processed by a single thread?
64 2012-12-13 12:43:18 <m0mchil> there >are< multiple threads... but only one of them is processing
65 2012-12-13 12:43:19 <m0mchil> there >are< multiple threads... but only one of them is processing
66 2012-12-13 12:45:14 <TD> m0mchil: by design
67 2012-12-13 12:45:21 <TD> m0mchil: the first 193k blocks aren't signature checked
68 2012-12-13 12:45:32 <m0mchil> oh, checkpoints
69 2012-12-13 12:45:33 <TD> it's not easy to parallelize the work that goes on during this period, but it's only for the first part of chain download
70 2012-12-13 12:45:34 <TD> indeed
71 2012-12-13 12:45:35 <TD> indeed
72 2012-12-13 12:53:24 <sipa> m0mchil: the new build has a -nocheckpoints option by the way
73 2012-12-13 12:53:57 <sipa> but even with that you won't see too much parallellism before block 130k or so
74 2012-12-13 12:54:26 <sipa> it's just a very simple job queue for now, and it's not that good at keeping all threads busy if there's not too much work
75 2012-12-13 12:54:27 <sipa> it's just a very simple job queue for now, and it's not that good at keeping all threads busy if there's not too much work
76 2012-12-13 13:17:40 <gavinandresen> sipa: I got about a 2x speedup on my quad-core mac benchmarking yesterday (41 versus 94 minutes to rebuild coins/ up to block 200,111)
77 2012-12-13 13:33:29 <MC1984> woo are you doing multi threaded?
78 2012-12-13 14:05:41 <sipa> gavinandresen: using coins rebuild?
79 2012-12-13 14:05:54 <sipa> MC1984: basic form yes
80 2012-12-13 14:07:10 <sipa> gavinandresen: oh, i should read entire sentences
81 2012-12-13 14:07:11 <sipa> gavinandresen: oh, i should read entire sentences
82 2012-12-13 14:07:17 <sipa> gavinandresen: hmm, nice, but could be better
83 2012-12-13 14:07:18 <sipa> gavinandresen: hmm, nice, but could be better
84 2012-12-13 14:14:45 <m0mchil> sipa: ok, for 283-gf6aadb1-win32
85 2012-12-13 14:15:09 <m0mchil> 88 minutes user time, 5h:46m CPU time, 71% average CPU, peak mem 235 MB, average memory usage was somewhat lower (than 244-g3ccb06f-win32)
86 2012-12-13 14:18:49 <jaromil> lv
87 2012-12-13 14:19:30 <jaromil> mistype nvm, hi everyone :)
88 2012-12-13 14:22:45 <sipa> gavinandresen: do you happen to know how much of that time was to get to 193k?
89 2012-12-13 14:23:52 <gavinandresen> sipa: didn't check. I'm re-running recompiled with clang, I'll keep track of the 0-193k time
90 2012-12-13 14:25:03 <gmaxwell> sipa: p2pool produced an invalid block today. Might be somewhat interesting.
91 2012-12-13 14:25:04 <gmaxwell> sipa: p2pool produced an invalid block today. Might be somewhat interesting.
92 2012-12-13 14:25:46 <sipa> gavinandresen: when benchmarking par/hal code, i started from a synced node, deleted coins/, let it resync to 193k, stop, and then made a snapshot of coins/ and blktree/; so i could just benchmark how long rebuilding 193k-210k took by restoring the snapshot
93 2012-12-13 14:25:53 <sipa> gavinandresen: oh wow, do we know how?
94 2012-12-13 14:25:56 <sipa> eh, gmaxwell
95 2012-12-13 14:29:54 <gmaxwell> 12/13/12 09:46:16 InvalidChainFound: invalid block=0000000000000202a4ba height=212048 work=659104249036541781231 date=12/13/12 09:52:08
96 2012-12-13 14:30:19 <gmaxwell> But it didn't log any reason _why_ it didn't accept it. timestamp and prev looks fine.
97 2012-12-13 15:06:37 <gavinandresen> sipa: ... make that a 3x speedup for 193,000 on up. Which matches CPU usage I'm seeing.
98 2012-12-13 15:15:02 <denisx> gavinandresen: so you do have a macversion of HEAD now?
99 2012-12-13 15:15:17 <denisx> gimme gimme gimme! ;)
100 2012-12-13 15:15:18 <denisx> gimme gimme gimme! ;)
101 2012-12-13 15:43:09 <MC1984> excellent
102 2012-12-13 15:44:16 <MC1984> is it just verification multi threaded
103 2012-12-13 15:44:27 <MC1984> or verify + i/o
104 2012-12-13 15:44:30 <MC1984> or both
105 2012-12-13 15:45:31 <gavinandresen> just ecdsa signature checking
106 2012-12-13 15:46:23 <sipa> but signature checking DOES run in parallel with DB lookups of later transactions in the same block
107 2012-12-13 15:46:41 <sipa> although the latter itself is signle-threaded
108 2012-12-13 15:48:27 <sipa> in cases the inputs required are cached, signature verification is about 50x slower, so it won't matter much
109 2012-12-13 15:50:02 <gavinandresen> sipa: compiling with a new clang and -flto is not significantly faster on my machine.
110 2012-12-13 15:50:03 <gavinandresen> sipa: compiling with a new clang and -flto is not significantly faster on my machine.
111 2012-12-13 15:51:29 <sipa> gavinandresen: no idea about clang - didn't even know that supported flto
112 2012-12-13 15:51:30 <sipa> gavinandresen: no idea about clang - didn't even know that supported flto
113 2012-12-13 17:18:09 <BlueMatt> wumpus: ok, i just know various issues have appeared in some versions and you and Diapolo seem to recall which versions better than me :)
114 2012-12-13 19:03:55 <helo> for ultraprune, will there be a way for a user to manually purge their old blocks?
115 2012-12-13 19:04:17 <helo> rather, "client-assisted" purge of old blocks
116 2012-12-13 19:06:18 <ciscoftw> blockchian.info not reachable from certain networks??? i can access blockchain.info behind some networks and not others, anybody know why this is???
117 2012-12-13 19:06:19 <ciscoftw> blockchian.info not reachable from certain networks??? i can access blockchain.info behind some networks and not others, anybody know why this is???
118 2012-12-13 19:06:20 <sipa> sorry for the confusion, but despite the name, ultraprune is not really about pruning blocks
119 2012-12-13 19:06:21 <sipa> sorry for the confusion, but despite the name, ultraprune is not really about pruning blocks
120 2012-12-13 19:06:48 <sipa> it's the name of a new database layout (which uses a ultra-pruned *copy* of the block chain) and associated validation engine
121 2012-12-13 19:06:49 <sipa> it's the name of a new database layout (which uses a ultra-pruned *copy* of the block chain) and associated validation engine
122 2012-12-13 19:07:21 <sipa> as a nice advantage, it also would allow removing old blocks without problems, but that's not implemented for now
123 2012-12-13 19:27:30 <gmaxwell> It's not only that pruning isn't implemented??? Pruning can't be done until there are considerable p2p enhancements so that nodes could still _find_ blocks.
124 2012-12-13 19:32:32 <sipa> i wonder what would happen if you'd just delete old block files yourself
125 2012-12-13 19:32:54 <sipa> it may just work, with rescans and block servings failing
126 2012-12-13 19:32:55 <sipa> it may just work, with rescans and block servings failing
127 2012-12-13 19:40:20 <gmaxwell> sipa: if people do that in any great number the network will quite rapidly become unusuable...
128 2012-12-13 19:40:23 <gmaxwell> er unusable.
129 2012-12-13 19:40:24 <gmaxwell> er unusable.
130 2012-12-13 19:44:06 <sipa> gmaxwell: i know, i woudln't encourage it; but if it happens, it pribably means we need to officially implement it (and turn of the NODE_NETWORK bit in that case)
131 2012-12-13 19:44:07 <sipa> gmaxwell: i know, i woudln't encourage it; but if it happens, it pribably means we need to officially implement it (and turn of the NODE_NETWORK bit in that case)
132 2012-12-13 19:45:14 <gmaxwell> ugh. Alternatively, until the network part is implemented make it a runtime assert. So the node will just crash the first time it fails.
133 2012-12-13 19:45:25 <gmaxwell> But I do see what you're saying.
134 2012-12-13 19:46:03 <gmaxwell> Are we even smart enough not to pick a non-NODE_NETWORK peer for IBD?
135 2012-12-13 19:51:09 <sipa> gmaxwell: i believe that is checked, yes
136 2012-12-13 19:51:57 <sipa> but imho there is much more need for nodes that need to sync let's say the last week vs those that have to sync from scratch
137 2012-12-13 19:52:23 <sipa> so a network bit saying "i serve the last 1000" blocks (or something) would make sense
138 2012-12-13 19:52:24 <sipa> so a network bit saying "i serve the last 1000" blocks (or something) would make sense
139 2012-12-13 19:52:38 <sipa> as opposed to just nothig/everything
140 2012-12-13 20:06:16 <gmaxwell> right, but thats why taking away NODE_NETWORK would currently be so disruptive.
141 2012-12-13 20:06:41 <gmaxwell> also risky because people don don't know better may just rm all of them and not be able to reorg.
142 2012-12-13 20:06:42 <gmaxwell> also risky because people don don't know better may just rm all of them and not be able to reorg.
143 2012-12-13 20:08:47 <sipa> sure
144 2012-12-13 20:18:18 <sipa> ACTION expects a failed pulltester report for #2106
145 2012-12-13 21:20:04 <gavinandresen> ;;blocks
146 2012-12-13 21:20:05 <gribble> 212106
147 2012-12-13 21:23:13 <sipa> checkpoint addition time!
148 2012-12-13 21:23:14 <sipa> checkpoint addition time!
149 2012-12-13 21:24:19 <gavinandresen> ... and 0.7.2 final release time...
150 2012-12-13 21:24:26 <sipa> sounds good
151 2012-12-13 21:24:51 <sipa> i can build
152 2012-12-13 21:24:52 <sipa> i can build
153 2012-12-13 21:25:11 <gavinandresen> tagging now....
154 2012-12-13 21:26:17 <gavinandresen> * [new tag] v0.7.2 -> v0.7.2
155 2012-12-13 21:26:59 <sipa> $ ./bitcoin-build.sh v0.7.2
156 2012-12-13 21:27:00 <sipa> $ ./bitcoin-build.sh v0.7.2
157 2012-12-13 21:27:23 <helo> :D
158 2012-12-13 21:27:24 <helo> :D
159 2012-12-13 21:33:25 <gavinandresen> building mac/linux/win...
160 2012-12-13 21:33:26 <gavinandresen> building mac/linux/win...
161 2012-12-13 21:38:48 <sipa> 6083dbb68badfcb0706ba584d43d680e33527094ec8fede5300c45db986c61a3 bitcoin-qt.exe
162 2012-12-13 21:38:56 <sipa> 7598da15df63a80a426838491c3b79b42ef3ef4b3bc42cf867804e1e1eb06b69 bitcoin-0.7.2-win32-setup.exe
163 2012-12-13 21:40:51 <gavinandresen> afk for a while, dinner. (mac build is done and uploading to sourceforge, though)
164 2012-12-13 22:08:57 <sipa> gavinandresen: pushed
165 2012-12-13 22:16:48 <denisx> sipa: nice, who found the 100% freebsd problem?
166 2012-12-13 22:18:27 <sipa> Author: Alex <alex>
167 2012-12-13 22:18:28 <sipa> Author: Alex <alex>
168 2012-12-13 22:19:06 <sipa> == no clue
169 2012-12-13 22:19:58 <D34TH> sipa: my money is on Alex
170 2012-12-13 22:19:59 <D34TH> sipa: my money is on Alex
171 2012-12-13 22:21:25 <sipa> 00:09:27 -!- paraipan (Alex B) [~paraipan@gateway/tor-sasl/paraipanakos] has joined #bitcoin-dev
172 2012-12-13 22:21:36 <sipa> paraipan: are you Alex? :)
173 2012-12-13 22:21:46 <paraipan> yes
174 2012-12-13 22:21:51 <paraipan> why you asking?
175 2012-12-13 22:22:12 <denisx> paraipan: what was the 100% cpu bug on freebsd?
176 2012-12-13 22:22:13 <denisx> paraipan: what was the 100% cpu bug on freebsd?
177 2012-12-13 22:22:17 <paraipan> author of what?
178 2012-12-13 22:22:37 <paraipan> i don't think I'm that Alex
179 2012-12-13 22:22:38 <paraipan> lol
180 2012-12-13 22:23:10 <sipa> denisx: i know what the bug was, i don't know who fixed it :)
181 2012-12-13 22:23:39 <sipa> (boost::interprocess brokenness)
182 2012-12-13 22:24:42 <denisx> but the osx problem is still there
183 2012-12-13 22:25:43 <sipa> the slow validation?
184 2012-12-13 22:25:48 <denisx> yes
185 2012-12-13 22:26:23 <denisx> I did a short sample and it is used in mach_msg_trap
186 2012-12-13 22:26:24 <denisx> I did a short sample and it is used in mach_msg_trap
187 2012-12-13 22:27:19 <sipa> http://stackoverflow.com/questions/1488601/how-to-find-out-what-mach-msg-trap-waits-for
188 2012-12-13 22:28:30 <denisx> hmm, maybe its the progressbar itself
189 2012-12-13 22:28:37 <denisx> osx had similar problems in the early days
190 2012-12-13 22:28:38 <denisx> osx had similar problems in the early days
191 2012-12-13 22:29:54 <sipa> is bitcoind also slow?
192 2012-12-13 22:32:24 <sipa> gavin reported reindex for him running in 41 minutes, which sounds reasonable
193 2012-12-13 22:32:25 <sipa> gavin reported reindex for him running in 41 minutes, which sounds reasonable
194 2012-12-13 22:32:43 <denisx> sipa: what do you mean with "bitcoind also" ?
195 2012-12-13 22:32:44 <denisx> sipa: what do you mean with "bitcoind also" ?
196 2012-12-13 22:32:58 <sipa> well you say the problem may be with the progressbar
197 2012-12-13 22:33:22 <sipa> one perfect way to check whether that's the cause is running the same code without progressbar :)
198 2012-12-13 22:33:23 <sipa> one perfect way to check whether that's the cause is running the same code without progressbar :)
199 2012-12-13 22:33:35 <denisx> when the updating of teh blocks is done the progressbar is not there anymore
200 2012-12-13 22:33:42 <denisx> then bitcoind runs fine
201 2012-12-13 22:33:59 <sipa> bitcoind doesn't have a progressbar - bitcoin-qt does
202 2012-12-13 22:34:00 <sipa> bitcoind doesn't have a progressbar - bitcoin-qt does
203 2012-12-13 22:34:34 <denisx> sipa: ah, got it
204 2012-12-13 22:36:06 <denisx> but I dont have bitcoind on osx ;(
205 2012-12-13 22:36:17 <sipa> oh
206 2012-12-13 22:36:34 <sipa> gavinandresen: you don't build bitcoind for osx?
207 2012-12-13 22:42:15 <gavinandresen> sipa : no, just Qt.
208 2012-12-13 22:42:46 <denisx> gavinandresen: I would like to have bitcoind
209 2012-12-13 22:42:48 <gavinandresen> (well, I do build bitcoind for osx all the time, I just haven't bothered to do all the packaging to redistribute it)
210 2012-12-13 22:43:18 <gavinandresen> denisx: do you have Xcode installed?
211 2012-12-13 22:43:24 <denisx> gavinandresen: sure
212 2012-12-13 22:43:43 <gavinandresen> denisx: ok, I'll upload a binary for you...
213 2012-12-13 22:43:44 <gavinandresen> denisx: ok, I'll upload a binary for you...
214 2012-12-13 22:43:54 <denisx> gavinandresen: cool, thanks
215 2012-12-13 22:54:55 <gavinandresen> denisx: http://skypaint.com/bitcoin/bitcoind-0.7.2-macosx
216 2012-12-13 22:57:24 <denisx> gavinandresen: now I need to build all those libs and put it in /opt/local/lib ?
217 2012-12-13 22:58:33 <denisx> no, I have them already in bitcoin-qt...
218 2012-12-13 22:58:34 <denisx> no, I have them already in bitcoin-qt...
219 2012-12-13 22:58:50 <gavinandresen> yeah, that's the packaging that I'm too lazy to do....
220 2012-12-13 22:58:51 <gavinandresen> yeah, that's the packaging that I'm too lazy to do....
221 2012-12-13 22:59:16 <denisx> I see
222 2012-12-13 23:00:15 <gavinandresen> LD_LIBRARY_PATH=/Application/Bitcoin-Qt/Contents/MacOS bitcoind ...etc... or something like that might work
223 2012-12-13 23:01:41 <denisx> no matter what I do: Illegal instruction: 4
224 2012-12-13 23:01:57 <gavinandresen> sipa: builds match, I'll push my sigs and upload to sourceforge
225 2012-12-13 23:04:35 <sipa> no third builder? :(
226 2012-12-13 23:04:45 <sipa> ACTION pokes Luke-Jr, BlueMatt, wumpus
227 2012-12-13 23:07:26 <gavinandresen> ACTION has to go to a school concert now
228 2012-12-13 23:07:27 <gavinandresen> ACTION has to go to a school concert now
229 2012-12-13 23:29:19 <slush> which files needs to be on fast storage to speedup bitcoind? blkindex.dat, blk*.dat and database/ directory?
230 2012-12-13 23:30:46 <sipa> database/ mostly needs fast writability, blkindex.dat shouldn't matter too much if you have lots of cache, and the block files need fast read access
231 2012-12-13 23:30:55 <sipa> in 0.8 however, just coins/ needs to be fast
232 2012-12-13 23:31:08 <slush> sipa: when will be 0.8 released?
233 2012-12-13 23:31:09 <slush> sipa: when will be 0.8 released?
234 2012-12-13 23:31:14 <sipa> when it's ready
235 2012-12-13 23:31:17 <slush> :-)
236 2012-12-13 23:31:29 <slush> I'm considering to move files to ramdisk
237 2012-12-13 23:31:30 <slush> I'm considering to move files to ramdisk
238 2012-12-13 23:32:45 <slush> but maybe the best solution is to don't touch it and wait for 0.8 which will hopefully improve I/O performance
239 2012-12-13 23:33:20 <sipa> you can try whether it already does :)
240 2012-12-13 23:33:59 <sipa> by 'speedup bitcoind', which operation do you mean in particular?
241 2012-12-13 23:34:16 <slush> catching up blocks, in normal pool operation
242 2012-12-13 23:34:30 <slush> with bigger blocks there is sometimes significant latency
243 2012-12-13 23:34:34 <sipa> process/relay incoming blocks, relay transactions, IBD, getblocktemplate, ...
244 2012-12-13 23:34:42 <slush> everything ;)
245 2012-12-13 23:35:26 <slush> pool is running standard 0.7, which is pretty well connected, so every part takes some CPU time
246 2012-12-13 23:35:27 <slush> pool is running standard 0.7, which is pretty well connected, so every part takes some CPU time
247 2012-12-13 23:36:08 <etotheipi_> sipa: is there anything different about how 0.8+ writes to the blk files? I mean, I know how they are renamed and relocated... but I'm seeing errors to do with impartial blocks being written
248 2012-12-13 23:36:20 <sipa> etotheipi_: the block files are pre-allocated
249 2012-12-13 23:36:33 <etotheipi_> oh hell
250 2012-12-13 23:36:49 <sipa> in regions of 16 MiB - every time a new region is needed, 16 MiB of zeroes are written
251 2012-12-13 23:36:50 <sipa> in regions of 16 MiB - every time a new region is needed, 16 MiB of zeroes are written
252 2012-12-13 23:36:51 <etotheipi_> how do you know how much data is there?
253 2012-12-13 23:37:00 <sipa> it's in the block index
254 2012-12-13 23:37:31 <sipa> but in general, if you see 4 0x00's instead of the block magic, you can assume it's the EOF
255 2012-12-13 23:37:54 <sipa> it shouldn't ever produce holes
256 2012-12-13 23:38:17 <etotheipi_> okay
257 2012-12-13 23:38:18 <etotheipi_> okay
258 2012-12-13 23:38:29 <etotheipi_> that's reasonable.... the magic bytes saved me here
259 2012-12-13 23:38:30 <etotheipi_> that's reasonable.... the magic bytes saved me here
260 2012-12-13 23:38:36 <sipa> 0.7 always just appended, even after partial blocks or so
261 2012-12-13 23:38:54 <sipa> 0.8 always writes in the first unused location, which means overwriting a partial block if there'd be one
262 2012-12-13 23:39:02 <etotheipi_> so much for a smooth transition to 0.8 for Armory...
263 2012-12-13 23:39:20 <sipa> well, people complained about excessive fragmentation
264 2012-12-13 23:39:27 <sipa> this is a neat solution, imho
265 2012-12-13 23:39:28 <etotheipi_> ooh, I see
266 2012-12-13 23:39:28 <sipa> this is a neat solution, imho
267 2012-12-13 23:39:39 <etotheipi_> each block ending up at different places on disk?
268 2012-12-13 23:40:06 <sipa> the index just maintains how much bytes are used in each block file
269 2012-12-13 23:40:24 <etotheipi_> well, Armory uses strictly blk000X.dat size (according to OS) to know when new blockdata comes in
270 2012-12-13 23:40:31 <etotheipi_> so obviously, I need to rework that code
271 2012-12-13 23:40:37 <sipa> oh
272 2012-12-13 23:40:49 <sipa> i'd expect you to listen to the P2P port for block messages or so
273 2012-12-13 23:41:09 <slush> sipa: sorry for stupid questions, but what happen to bdb when process crash and database/ is lost?
274 2012-12-13 23:41:10 <slush> sipa: sorry for stupid questions, but what happen to bdb when process crash and database/ is lost?
275 2012-12-13 23:41:12 <etotheipi_> sipa: it's because Armory no longer stores any blockdata itself, it only stores pointers to file locations
276 2012-12-13 23:41:37 <sipa> slush: say a prayer that it still works; if it doesn't, restart from scratch
277 2012-12-13 23:41:39 <etotheipi_> since blk files are append-only, it's worked out quite well until now
278 2012-12-13 23:42:16 <etotheipi_> sipa: it also allows armoryengine to operate without any of the networking code
279 2012-12-13 23:42:16 <sipa> slush: the database/ dir is there exactly for the case that the process crashes by the way
280 2012-12-13 23:42:17 <sipa> slush: the database/ dir is there exactly for the case that the process crashes by the way
281 2012-12-13 23:42:29 <slush> so it's some kind of binary log?
282 2012-12-13 23:42:32 <sipa> yes
283 2012-12-13 23:42:32 <slush> journal
284 2012-12-13 23:42:35 <slush> damn
285 2012-12-13 23:42:54 <sipa> why?
286 2012-12-13 23:43:08 <slush> you wrote that they need fast write access
287 2012-12-13 23:43:16 <slush> so moving them to RAM would make sense
288 2012-12-13 23:43:17 <slush> so moving them to RAM would make sense
289 2012-12-13 23:43:32 <slush> I can keep blk files in disk cache without keeping them in ramdisk...
290 2012-12-13 23:43:38 <sipa> if you shutdown with -detachdb, you don't need database
291 2012-12-13 23:43:39 <sipa> if you shutdown with -detachdb, you don't need database
292 2012-12-13 23:43:53 <slush> but what if server crash...
293 2012-12-13 23:43:54 <slush> but what if server crash...
294 2012-12-13 23:43:58 <sipa> but that will mean rebuild from scratch in case of unexpected crash
295 2012-12-13 23:44:02 <slush> yes
296 2012-12-13 23:44:03 <slush> yes
297 2012-12-13 23:44:24 <sipa> but the log data is cached in memory anyway
298 2012-12-13 23:44:44 <sipa> the reason it is slow, is because there are sycnhronous writes to database/, to guarantee nothing gets lost
299 2012-12-13 23:45:01 <slush> strange that I enabled writeback on disk and it still waits on I/O
300 2012-12-13 23:45:02 <slush> strange that I enabled writeback on disk and it still waits on I/O
301 2012-12-13 23:45:10 <slush> with ram cache unused :-(
302 2012-12-13 23:45:17 <sipa> it waits for the OS to flush to disk
303 2012-12-13 23:45:39 <sipa> if you "speed things up" by having database/ in RAM, you might as well disable the log in the first place
304 2012-12-13 23:45:41 <slush> even with writeback enabled? I didn't expect that flush() will wait
305 2012-12-13 23:45:42 <slush> even with writeback enabled? I didn't expect that flush() will wait
306 2012-12-13 23:45:50 <slush> yes, I see
307 2012-12-13 23:45:51 <slush> yes, I see
308 2012-12-13 23:45:55 <sipa> writeback? on disk?
309 2012-12-13 23:45:58 <slush> yes
310 2012-12-13 23:46:08 <sipa> so it flushes to the disk cache
311 2012-12-13 23:46:15 <sipa> but doesn't wait for the disk to write its cache
312 2012-12-13 23:46:28 <sipa> but as the disk cache is small, it blocks anyway
313 2012-12-13 23:46:38 <slush> then one should expect that it will eats server RAM with cache
314 2012-12-13 23:46:41 <slush> but it doesn't happen
315 2012-12-13 23:46:42 <slush> but it doesn't happen
316 2012-12-13 23:46:59 <sipa> it should, but that won't speedup fsync
317 2012-12-13 23:47:08 <sipa> as it will still wait for things to be written to disk
318 2012-12-13 23:47:16 <slush> hm
319 2012-12-13 23:47:17 <slush> hm
320 2012-12-13 23:47:31 <sipa> cache helps for reading, not for writing
321 2012-12-13 23:47:42 <slush> so storing everything in RAM (and making snapshots every hour or so) looks like the best solution
322 2012-12-13 23:47:46 <slush> but it requires really lot of ram
323 2012-12-13 23:47:47 <slush> but it requires really lot of ram
324 2012-12-13 23:48:02 <sipa> in 0.8, coins/ is only 150 MiB or so
325 2012-12-13 23:48:11 <slush> oh nice
326 2012-12-13 23:48:12 <sipa> and it is cached inside the client anyway
327 2012-12-13 23:48:13 <sipa> and it is cached inside the client anyway
328 2012-12-13 23:48:32 <slush> so 0.8 won't require SSD in RAID anymore? ;)
329 2012-12-13 23:48:43 <sipa> no
330 2012-12-13 23:48:52 <sipa> well... maybe... depending on what you need
331 2012-12-13 23:49:10 <sipa> at times it writes block files to disk before relaying them, you probably still want that writing to be fast
332 2012-12-13 23:49:55 <slush> why it should wait to disk I/O before relaying them?
333 2012-12-13 23:50:06 <sipa> because that happens in the same thread
334 2012-12-13 23:50:07 <sipa> because that happens in the same thread
335 2012-12-13 23:50:12 <slush> but well, storing hundreds of MB in RAM is no problem
336 2012-12-13 23:50:13 <slush> but well, storing hundreds of MB in RAM is no problem
337 2012-12-13 23:50:18 <sipa> i'd like to change that, but it's somewhat of a large refactor
338 2012-12-13 23:50:22 <slush> I see
339 2012-12-13 23:50:23 <slush> I see
340 2012-12-13 23:52:36 <slush> sipa: to safely restore database from crash, I need snapshot of blk files and database/ from the same time, correct?
341 2012-12-13 23:53:22 <sipa> technically, only of blkindex.dat and database/ - the corresponding blk0*.dat files can be snapshotted later
342 2012-12-13 23:53:23 <sipa> technically, only of blkindex.dat and database/ - the corresponding blk0*.dat files can be snapshotted later
343 2012-12-13 23:53:41 <slush> hm, perfect
344 2012-12-13 23:54:07 <sipa> how will you do that?
345 2012-12-13 23:54:08 <sipa> how will you do that?
346 2012-12-13 23:54:13 <slush> LVM in ramdisk
347 2012-12-13 23:54:16 <sipa> lvm snapshot?
348 2012-12-13 23:54:17 <sipa> right
349 2012-12-13 23:54:27 <slush> will store database/ and blkindex, which is less than 1 GB in size
350 2012-12-13 23:54:35 <slush> and blk files on disk
351 2012-12-13 23:54:36 <slush> and blk files on disk
352 2012-12-13 23:55:01 <sipa> that should be safe
353 2012-12-13 23:55:15 <slush> absolutely last stupid question, at least for this time:
354 2012-12-13 23:55:41 <zapsoda> Anyone know what data type "getbalance" returns?
355 2012-12-13 23:56:00 <slush> do I need snapshots of blk files at all? Or can I provide snapshot of blkindex + database/ and put it together with any newer version of blk files?
356 2012-12-13 23:56:01 <slush> do I need snapshots of blk files at all? Or can I provide snapshot of blkindex + database/ and put it together with any newer version of blk files?
357 2012-12-13 23:56:58 <sipa> slush: that's fine
358 2012-12-13 23:56:59 <sipa> slush: that's fine
359 2012-12-13 23:57:05 <sipa> the blk files are append-only