1 2012-07-21 00:01:34 <[Tycho]> gmaxwell: building gcc48 to try it just out of curiosity :)
  2 2012-07-21 00:05:20 <jgarzik> [Tycho]: gets around lack of 'const' in GetBlockHeader(), I suspect
  3 2012-07-21 00:05:33 <jgarzik> er, the other way around
  4 2012-07-21 00:06:10 <[Tycho]> Mostly I'm curious why it works on another machine with almost the same g++ 4.2
  5 2012-07-21 00:06:33 <jgarzik> [Tycho]: is 'another machine' 32bit rather than amd64?
  6 2012-07-21 00:06:53 <[Tycho]> No.
  7 2012-07-21 00:07:10 <[Tycho]> Same amd64, just a bit older version of OS
  8 2012-07-21 00:07:37 <[Tycho]> Also I see "patched" remark in the g++ version of the "new" machine.
  9 2012-07-21 00:07:50 <[Tycho]> (may be it was added when I installed gnat there...)
 10 2012-07-21 00:17:15 <xork> Dev regarding the client or can we discuss services too?
 11 2012-07-21 00:32:08 <Diablo-D3> hey gmaxwell
 12 2012-07-21 00:32:15 <Diablo-D3> I have found my new favorite word
 13 2012-07-21 00:32:18 <Diablo-D3> refuctoring
 14 2012-07-21 00:32:22 <Diablo-D3> http://www.codinghorror.com/blog/2012/07/new-programming-jargon.html
 15 2012-07-21 02:20:10 <gmaxwell> hm. Person I was helping earlier in #bitcoin has a stuck node on macos...
 16 2012-07-21 02:20:27 <gmaxwell> debug.log shows:
 17 2012-07-21 02:20:28 <gmaxwell> received block 00000000000004f77730
 18 2012-07-21 02:20:30 <gmaxwell> REORGANIZE: Connect 1 blocks; 00000000000003f38b33..0000000000000025bc30
 19 2012-07-21 02:20:49 <gmaxwell> and then no more activity (just lots of connect noise)
 20 2012-07-21 02:21:28 <gmaxwell> so it looks like yet another way for nodes to get stuck. :-/
 21 2012-07-21 02:23:11 <leotreasure> hello
 22 2012-07-21 02:23:34 <leotreasure> i'm trying to run bitcoind on a vps but getting error: couldn't connect to server
 23 2012-07-21 02:27:22 <D34TH> is it completly started?
 24 2012-07-21 02:27:33 <D34TH> or is it still downloading the chain?
 25 2012-07-21 02:27:50 <leotreasure> i think it should be finished downloading by now
 26 2012-07-21 02:28:04 <leotreasure> i started it about 12 hours ago
 27 2012-07-21 02:28:38 <leotreasure> although i got an anti abuse email from the vps provider saying i had a lot of disk io operations
 28 2012-07-21 02:28:51 <leotreasure> so i'm not sure if they have stopped it or not
 29 2012-07-21 02:29:13 <leotreasure> i tried ps aux and can't seem to see the bitcoind running
 30 2012-07-21 02:29:25 <luke-jr> leotreasure: tell the VPS provider to stfu? :p
 31 2012-07-21 02:29:27 <gmaxwell> leotreasure: look in the debug.log
 32 2012-07-21 02:29:35 <luke-jr> it's not abuse to use the service you're paying for <.<
 33 2012-07-21 02:29:59 <leotreasure> thanks luke-jr that's what i figured
 34 2012-07-21 02:30:11 <leotreasure> thanks gmaxwell will do this now
 35 2012-07-21 02:31:14 <copumpkin> if you have two chains of the same length, who wins?
 36 2012-07-21 02:31:40 <copumpkin> or rather, what should the client do until one of them wins?
 37 2012-07-21 02:34:54 <gmaxwell> copumpkin: it just believes the first it heard unless something longer comes along.
 38 2012-07-21 02:35:02 <copumpkin> ah, okay
 39 2012-07-21 02:37:47 <leotreasure> gmaxwell: i'm having trouble deciphering this debug.log file
 40 2012-07-21 02:37:52 <doublec> leotreasure: "too much disk io" is normal if you're on a linode. You can change the linode settings so it requires higher io before you get the warning.
 41 2012-07-21 02:37:53 <luke-jr> copumpkin: except if it's at a new difficulty, then it picks the harder one
 42 2012-07-21 02:38:01 <copumpkin> luke-jr: I see
 43 2012-07-21 02:38:34 <leotreasure> thanks doublec i am with vr.org got email warning from a person
 44 2012-07-21 02:38:51 <doublec> leotreasure: if it's not running it may have ben OOMed - how much memory does your vps have?
 45 2012-07-21 02:38:52 <leotreasure> gmaxwell: is the first line the most recent?
 46 2012-07-21 02:39:16 <luke-jr> leotreasure: bottom line is
 47 2012-07-21 02:39:33 <leotreasure> doublec: 256MB
 48 2012-07-21 02:40:20 <phantomcircuit> leotreasure, what kind of vps?
 49 2012-07-21 02:40:24 <phantomcircuit> xen kvm openvz
 50 2012-07-21 02:40:28 <phantomcircuit> lxc?
 51 2012-07-21 02:40:56 <copumpkin> so a reorg is simply when another chain gets longer than the one you're believing?
 52 2012-07-21 02:41:03 <phantomcircuit> copumpkin, yes
 53 2012-07-21 02:41:21 <copumpkin> and if there are two or more competing chains and they keep getting longer than each other, then it'll cause havoc
 54 2012-07-21 02:41:28 <phantomcircuit> pretty much
 55 2012-07-21 02:41:32 <copumpkin> fair enough
 56 2012-07-21 02:41:49 <copumpkin> so the mapping from block hash -> block metadata can be append-only, right?
 57 2012-07-21 02:41:49 <phantomcircuit> it's uncommon enough that it's sort of a shrug worthy event
 58 2012-07-21 02:41:57 <copumpkin> the only thing that needs undo support is the metadata for the chain you're working on
 59 2012-07-21 02:42:21 <phantomcircuit> leotreasure, setup swap
 60 2012-07-21 02:42:27 <phantomcircuit> laugh as you use even more disk io
 61 2012-07-21 02:42:46 <leotreasure> phantomcircuit: :) ty
 62 2012-07-21 02:43:00 <leotreasure> i'm not sure what kind of vps this is or how to find out
 63 2012-07-21 02:43:04 <leotreasure> all i see is Cloud Servers - VR256
 64 2012-07-21 02:43:09 <phantomcircuit> it's xen i read their website
 65 2012-07-21 02:43:23 <leotreasure> ok ty
 66 2012-07-21 02:43:34 <galambo_> i would find a bigger vps server that has enough customers so that they don't have time to watch over your shoulder
 67 2012-07-21 02:43:55 <phantomcircuit> leotreasure, get a momentovps server it'll handle bitcoind fine
 68 2012-07-21 02:44:00 <luke-jr> copumpkin: won't cause havoc for people who wait for 6 confirms :P
 69 2012-07-21 02:44:06 <phantomcircuit> (i run that shit show)
 70 2012-07-21 02:44:20 <phantomcircuit> luke-jr, i believe he was referring to disk io
 71 2012-07-21 02:44:33 <copumpkin> luke-jr: conceivably, there could be two competing chains longer than 6 confirms with powerful weird/dishonest miners, right?
 72 2012-07-21 02:44:57 <leotreasure> thanks phantomcircuit will check it out
 73 2012-07-21 02:45:00 <galambo_> there were two competing chains 53 blocks long once
 74 2012-07-21 02:45:04 <luke-jr> copumpkin: that'd be a like 99% attack
 75 2012-07-21 02:45:15 <copumpkin> well, some weirdo could keep mining a separate chain, right?
 76 2012-07-21 02:45:25 <copumpkin> just for the lulz
 77 2012-07-21 02:45:38 <galambo_> i think thats called an altcoin ;)
 78 2012-07-21 02:45:39 <copumpkin> without any of the resolution logic
 79 2012-07-21 02:45:47 <phantomcircuit> copumpkin, that would be very expensive lulz
 80 2012-07-21 02:46:07 <copumpkin> I guess that'd just be considered a fork, then?
 81 2012-07-21 02:46:16 <copumpkin> if you don't attempt to reconcile with the other chain?
 82 2012-07-21 02:46:18 <phantomcircuit> each block is like
 83 2012-07-21 02:46:24 <phantomcircuit> 500USD in btc
 84 2012-07-21 02:46:29 <copumpkin> oh, I know
 85 2012-07-21 02:46:31 <phantomcircuit> your chain would eventually lose the race
 86 2012-07-21 02:46:37 <phantomcircuit> so each block would cost you 500 USD
 87 2012-07-21 02:46:48 <phantomcircuit> that adds up pretty fast
 88 2012-07-21 02:46:49 <copumpkin> I guess if we kept doing this
 89 2012-07-21 02:47:00 <copumpkin> powerful but stupid nodes mining a forked chain
 90 2012-07-21 02:47:09 <copumpkin> the difficulty would go down on that chain and everyone would just ignore it
 91 2012-07-21 02:47:10 <copumpkin> right?
 92 2012-07-21 02:47:51 <phantomcircuit> yeah
 93 2012-07-21 02:48:08 <phantomcircuit> when the difficulty recalculation happened the second chain would become 100% prunable
 94 2012-07-21 02:48:15 <phantomcircuit> as it would no longer even be valid
 95 2012-07-21 02:48:20 <phantomcircuit> well sort of
 96 2012-07-21 02:48:45 <phantomcircuit> i guess you could be a dick and cause lots of separate chains
 97 2012-07-21 02:48:46 <copumpkin> so difficulty discrepancies in forks are handled by picking the more difficult of them, because it suggests that more people mine on it?
 98 2012-07-21 02:48:53 <phantomcircuit> but seriously that would be horrendously expensive
 99 2012-07-21 02:49:16 <phantomcircuit> the chain with the most difficulty is always chosen
100 2012-07-21 02:49:21 <phantomcircuit> it's the work calculation
101 2012-07-21 02:49:27 <phantomcircuit> which is more or less inverted target
102 2012-07-21 02:49:36 <phantomcircuit> (not really but close enough)
103 2012-07-21 03:42:43 <jgarzik> anyway
104 2012-07-21 03:42:53 <jgarzik> what does a serialized vector<unsigned char> look like?
105 2012-07-21 03:43:00 <jgarzik> CompactSize + chars ?
106 2012-07-21 03:51:38 <luke-jr> ;;bc,blocks
107 2012-07-21 03:51:39 <gribble> 190030
108 2012-07-21 04:19:42 <luke-jr> 188529 is the last block of blk0001.dat, if no orphans are present
109 2012-07-21 05:47:47 <leotreasure> how come getblockcount doesn't return anything?
110 2012-07-21 05:49:23 <leotreasure> & stop doesn't stop
111 2012-07-21 05:49:50 <leotreasure> ctrl z works but is that bad for bitcoind when downloading blockchain?
112 2012-07-21 05:59:22 <leotreasure> plz disregard
113 2012-07-21 06:00:12 <leotreasure> different question: how can i issue commands like getblockcount to bitcoind while it is downloading, and not launch another bitcoind process?
114 2012-07-21 06:06:52 <dooglus> leotreasure: if you issue commands, it won't start another server
115 2012-07-21 06:09:11 <dooglus> leotreasure: ie. if any command line argument doesn't start with '-' or 'bitcoin:', it's considered to be an RPC client invokation
116 2012-07-21 06:27:35 <leotreasure> thanks dooglus
117 2012-07-21 06:28:50 <leotreasure> does that mean in order to getblockcount i need to connect to bitcoind using -rpcuser username and -rpcpassword pw?
118 2012-07-21 06:36:39 <dooglus> leotreasure: no.  you need to set those things in bitcoin.conf
119 2012-07-21 06:36:52 <dooglus> leotreasure: then forget about them
120 2012-07-21 06:37:09 <leotreasure> oh ok thanks!
121 2012-07-21 06:37:27 <dooglus> leotreasure: both the server and client will read the same file
122 2012-07-21 06:38:11 <leotreasure> do you know if it the case that bitcoind is unresponsive to getblockcount command while downloading the blockchain?
123 2012-07-21 06:47:51 <dooglus> leotreasure: while loading the blockchain from disk, it is for sure
124 2012-07-21 06:48:07 <dooglus> leotreasure: while downloading from the network it can be slow, but should respond
125 2012-07-21 06:49:48 <dooglus> leotreasure: I have definitely used 'getblockcount' to monitor the download progress before
126 2012-07-21 06:50:15 <leotreasure> thanks, it's probably taking even longer because i'm running it through ionice
127 2012-07-21 06:50:46 <leotreasure> been waiting at least 10 minutes and still no response
128 2012-07-21 07:03:51 <finway> Hello,devs, i've downloaded blockchain from eu1.bitcoincharts.com/blockchain ,  how should i import and check it ?
129 2012-07-21 07:11:46 <dooglus> finway: http://bitcoin.stackexchange.com/a/4148/659
130 2012-07-21 07:15:05 <finway> thanks, dooglus
131 2012-07-21 07:15:41 <dooglus> finway: when I tried that it took many hours and froze part way through
132 2012-07-21 07:15:53 <dooglus> finway: hopefully you have better luck
133 2012-07-21 07:16:13 <dooglus> alternatively, just use the files you downloaded - unpack into .bitcoin/ and trust...
134 2012-07-21 07:17:59 <finway> I've told sipa can do that, but i don't know how, thanks.
135 2012-07-21 07:18:10 <finway> I've been told
136 2012-07-21 08:51:02 <gribble> New news from bitcoinrss: Diapolo opened pull request 1616 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/pull/1616>
137 2012-07-21 09:06:24 <gribble> New news from bitcoinrss: Diapolo opened pull request 1617 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/pull/1617>
138 2012-07-21 11:58:23 <jgarzik> gavinonaplane: watch out for snakes
139 2012-07-21 11:58:38 <gavinonaplane> damn, now you got me worried
140 2012-07-21 11:59:00 <gavinonaplane> more likely some slugs, I'm flying home from Seattle
141 2012-07-21 12:52:14 <TD> justmoon: epic
142 2012-07-21 12:52:23 <TD> justmoon: i guess i need to do some squashing of commits now. hmm.
143 2012-07-21 12:53:00 <TD> justmoon: any idea how gavin prefers pull reqs?
144 2012-07-21 12:53:08 <TD> justmoon: with all work-in-progress preserved or squashed ?
145 2012-07-21 13:09:35 <gavinonaplane> TD: squashed into logical commits
146 2012-07-21 13:13:43 <TD> you're on a plane!
147 2012-07-21 13:15:43 <TD> gavinonaplane: thanks for adding the SIGHASH flags as an option to signrawtx. that will prove useful in future.
148 2012-07-21 13:22:09 <lianj> is there a SIGHASH_ANYONECANPAY tx on the mainnet?
149 2012-07-21 13:23:42 <TD> lianj: no idea. probably not.
150 2012-07-21 13:24:04 <lianj> TD: hehe ok. just looking for a fixture for tests
151 2012-07-21 13:24:55 <TD> justmoon: i force pushed a rebase to my branch (including your changes)
152 2012-07-21 13:25:36 <TD> justmoon: i'll probably merge all of your commits into previous ones, will try and make sure your name shows up in the relevant places .... not sure how git manages commits that are the product of >1 person though
153 2012-07-21 13:33:12 <justmoon> TD: ok
154 2012-07-21 13:33:54 <justmoon> TD: no idea how gavinandresen prefers his pull requests, I assume shaken not stirred
155 2012-07-21 13:34:06 <TD> heh
156 2012-07-21 13:34:08 <TD> he just told me
157 2012-07-21 13:34:17 <TD> there are quite a few commits now so it'll take quite some rebasing
158 2012-07-21 13:41:24 <jgarzik> mine, mainnet, mine!
159 2012-07-21 13:44:24 <jgarzik> I meant, mine a block with >1 transaction, mainnet.
160 2012-07-21 13:45:20 <TD> https://sites.google.com/site/tmoipv6/lg-mytouch
161 2012-07-21 13:45:22 <TD> cool
162 2012-07-21 13:45:31 <TD> IPv6 for t-mobile usa 3G connections
163 2012-07-21 13:47:16 <luke-jr> TD: yeah, I've had that for over a year now :p
164 2012-07-21 13:47:23 <TD> which network are you with?
165 2012-07-21 13:47:47 <luke-jr> T-Mobile
166 2012-07-21 13:48:25 <TD> which phone do you use?
167 2012-07-21 13:48:32 <TD> their page says only ICS+ androids work.
168 2012-07-21 13:48:36 <TD> but ICS was out in december
169 2012-07-21 13:49:51 <luke-jr> TD: I don't use a phone, I use a Nokia N900 handheld computer
170 2012-07-21 13:49:56 <TD> ah ok
171 2012-07-21 13:49:59 <luke-jr> with Gentoo
172 2012-07-21 14:01:36 <TD> hrmm
173 2012-07-21 14:01:53 <TD> for some reason there's a -lleveldb sneaking in somewhere between the qt.pro -> xcode project conversion
174 2012-07-21 14:12:56 <gavinonaplane> TD: you're welcome RE: the SIGHASH flags. I was tempted to teach createrawtransaction to set lock time (and maybe sequence numbers), but decided that was going too far.
175 2012-07-21 14:13:09 <TD> heh
176 2012-07-21 14:13:24 <TD> those fields will probably end up being set by apps that speak the wire protocol directly
177 2012-07-21 14:13:27 <TD> rather than being API based
178 2012-07-21 14:14:01 <TD> gavinonaplane: do you use the xcode output of qmake on macos or use -spec macx-g++ ?
179 2012-07-21 14:15:18 <gavinonaplane> TD: -spec macx-g++ I think.  Is that the default for the macports Qt ?
180 2012-07-21 14:15:32 <TD> not sure where i got my qt from. it defaults to xcode, which sucks
181 2012-07-21 14:15:38 <TD> i only just discovered -spec macx-++
182 2012-07-21 14:16:59 <gavinonaplane> One of these months I might learn enough Qt to contribute a patch or two
183 2012-07-21 14:17:34 <gavinonaplane> If only we'd implemented the GUI in OpenGL.....
184 2012-07-21 14:17:56 <TD> hah
185 2012-07-21 14:18:04 <TD> spinning coins are cool
186 2012-07-21 14:18:09 <justmoon> omg best idea ever
187 2012-07-21 14:26:37 <TD> justmoon: migration still takes quite a long time :(
188 2012-07-21 14:26:50 <TD> justmoon: some people will be unhappy with the big delay. but i guess it is a one time cost.
189 2012-07-21 14:27:08 <TD> hey devrandom
190 2012-07-21 14:27:41 <justmoon> TD: I think if you're running the original client, you don't mind long delays all that much :P
191 2012-07-21 14:27:48 <TD> yeah
192 2012-07-21 14:27:51 <justmoon> took about 45 minutes for me
193 2012-07-21 14:27:51 <TD> i'd assume so :)
194 2012-07-21 14:28:05 <TD> hmmm
195 2012-07-21 14:28:09 <devrandom> hey TD
196 2012-07-21 14:28:11 <TD> back when i was testing it seemed faster
197 2012-07-21 14:28:22 <jgarzik> TD: one time cost, that won't be seen by new users
198 2012-07-21 14:28:25 <devrandom> mmmm... bloom filters....
199 2012-07-21 14:28:28 <TD> more like 5 minutes. 45 mins is how fast you can rescan the chain with sig checking enabled
200 2012-07-21 14:28:35 <TD> i wonder if there's a regression somewher
201 2012-07-21 14:29:21 <justmoon> hmm you mean on linux it's taking a long time too?
202 2012-07-21 14:32:49 <TD> justmoon: i'm doing it on osx at the moment.
203 2012-07-21 14:32:54 <TD> i wish i had taken timings for the migration
204 2012-07-21 14:33:02 <TD> it just seemed faster back then. but i was testing with a smaller chain
205 2012-07-21 14:33:20 <TD> i'm seeing 0.01% every 800msec or so, i'd guess
206 2012-07-21 14:33:36 <TD> no unexpected hotspots. spends most of its time deserializing things
207 2012-07-21 14:34:05 <TD> maybe it would have been better to have some custom code that just iterated across the bdb rather than rescanned the chain from scratch
208 2012-07-21 14:35:26 <gmaxwell> TD: I expect there are a lot of nodes out there with corrupted databases. It's probably better to take the chance to clean that up, rather than have it incorrectly reported as bugs w/ leveldb code.
209 2012-07-21 14:35:42 <TD> what makes you think there are nodes with corrupted databases?
210 2012-07-21 14:36:00 <gmaxwell> I know for a fact that there are I just dont know how many.
211 2012-07-21 14:36:03 <TD> oh, ok
212 2012-07-21 14:36:34 <gmaxwell> We get one person every few wees showing up in #bitcoin, or #p2pool with a node stuck because its rejecting a perfectly valid block.
213 2012-07-21 14:36:47 <gmaxwell> er, s/wees/weeks/
214 2012-07-21 14:37:17 <TD> hmm
215 2012-07-21 14:37:22 <justmoon> caused by unclean shutdowns during block download?
216 2012-07-21 14:37:48 <luke-jr> TD: http://luke.dashjr.org/programs/bitcoin/files/charts/bestblocks.html
217 2012-07-21 14:37:52 <gmaxwell> justmoon: that _shouldn't_ cause that, but e.g. stuff missing from the index for some reason.  People are running bitcoin on various 'windows quality hardware', and bitcoin is a great detector of that.
218 2012-07-21 14:38:02 <TD> it looks like p2pools luck has stabilized, at least.
219 2012-07-21 14:38:23 <TD> interesting chart
220 2012-07-21 14:38:34 <TD> luke-jr: why are there so many nodes stuck at 189k? or is that an artifact of your crawl
221 2012-07-21 14:38:46 <luke-jr> TD: most likely
222 2012-07-21 14:38:53 <luke-jr> ;;bc,blocks
223 2012-07-21 14:38:55 <gribble> 190112
224 2012-07-21 14:38:58 <luke-jr> we only just passed 190k
225 2012-07-21 14:39:04 <gmaxwell> http://people.xiph.org/~greg/bestblocks.png
226 2012-07-21 14:39:09 <gmaxwell> ^ probably a little more informative.
227 2012-07-21 14:39:55 <gmaxwell> (as you can see, thats a month old or so)
228 2012-07-21 14:40:06 <TD> yeah
229 2012-07-21 14:40:16 <gmaxwell> In any case, I expect things to be even worse among non-listening nodes... as they're more likely to be on marginal desktop hardware.
230 2012-07-21 14:40:16 <TD> i guess the big jump around 183k is satoshidice starting up
231 2012-07-21 14:41:46 <TD> gmaxwell: do you think the luck issue is what was holding back p2pool? now it's fixed i'd hope to see usage rise again
232 2012-07-21 14:41:49 <justmoon> TD: oh btw. during migration I accidentally clicked on the splash screen, which caused it to disappear, then I had no more information about the status or progress of the migration until it was finished
233 2012-07-21 14:42:00 <TD> justmoon: oh yeah
234 2012-07-21 14:42:03 <TD> justmoon: that's weird
235 2012-07-21 14:42:08 <TD> hadn't noticed that
236 2012-07-21 14:42:22 <TD> gmaxwell: though i guess it'll take a while for miners to notice that p2pool is no worse than other pools now
237 2012-07-21 14:42:38 <TD> justmoon: i didn't write any code to do that, so i guess it's the splash screens default behavior
238 2012-07-21 14:42:48 <gmaxwell> TD: I don't know that it was ... hard to say. Most 'latecomer' pools have had a hard time breaking this short of share.
239 2012-07-21 14:43:02 <TD> justmoon: it's kind of a crappy hack to abuse the splash in that way. ideally it'd be a real window with a progress bar
240 2012-07-21 14:43:14 <gmaxwell> I think there is are a fair amount of miners that started at some time, picked a pool, and just have no way of discovering other options.
241 2012-07-21 14:43:17 <justmoon> TD: yep, that's my point exactly ;)
242 2012-07-21 14:43:45 <TD> justmoon: i thought about it, but it seemed like a lot of work for something that'll appear exactly one, to a small number of people
243 2012-07-21 14:44:24 <justmoon> TD: yeah, I had the same thought regarding the performance - seems like it's easier to just bite the bullet and focus on things with more long term benefits
244 2012-07-21 14:44:34 <TD> yeah
245 2012-07-21 14:44:52 <TD> after we get this merged, the next big win is multi-threaded tx verification
246 2012-07-21 14:44:53 <justmoon> maybe an easy fix would be to make the splash screen non-dismissable
247 2012-07-21 14:44:54 <TD> i think
248 2012-07-21 14:44:59 <TD> right
249 2012-07-21 14:45:02 <TD> that'd be how i'd do it :)
250 2012-07-21 14:55:26 <TD> justmoon: did you switch leveldb to use the boost env on non-win32 platforms as well?
251 2012-07-21 14:56:48 <TD> hmm, nope
252 2012-07-21 14:56:49 <TD> weird
253 2012-07-21 14:56:54 <TD> i wonder why it's got so much slower for me
254 2012-07-21 14:57:06 <xorgate> hm bitcoind.exe always crashes on exit
255 2012-07-21 14:57:18 <TD> justmoon: could you send me some of your ACCEPTED ? blocks in ? msec lines?
256 2012-07-21 14:57:27 <TD> i'm seeing really high variance in how quickly it's processing transactions
257 2012-07-21 14:57:55 <justmoon> you want larger blocks or small ones?
258 2012-07-21 14:58:10 <TD> any
259 2012-07-21 14:58:12 <TD> larger ones
260 2012-07-21 14:58:20 <TD> just want to get a feel for what performance you're seeing
261 2012-07-21 14:58:33 <TD> i used to see blocks take only a few msec regardless of how many transactions they had, almost
262 2012-07-21 14:58:45 <TD> now i'm seeing stuff like
263 2012-07-21 14:59:02 <TD> but it's also really variable
264 2012-07-21 14:59:41 <TD> eg 18 txns in 1000+ msec
265 2012-07-21 15:00:02 <TD> i was testing migration times on a hdd linux workstation before. i wonder if mac performance is a lot worse
266 2012-07-21 15:01:59 <justmoon> I just started a fresh migration and I'm gonna let it run for a bit without doing anything else intensive - during the migration this morning I was using Adobe Premiere, so that probably wouldn't be the cleanest of data :P
267 2012-07-21 15:03:49 <TD> ok
268 2012-07-21 15:04:09 <TD> you're on a windows hdd box?
269 2012-07-21 15:05:06 <justmoon> windows 2x SSD (RAID-0)
270 2012-07-21 15:05:18 <TD> oh
271 2012-07-21 15:05:22 <TD> ok
272 2012-07-21 15:05:26 <TD> i'm on ssd too
273 2012-07-21 15:05:31 <TD> leveldb was designed for hard disks, really
274 2012-07-21 15:06:16 <TD> hmm
275 2012-07-21 15:06:31 <TD> ah crap
276 2012-07-21 15:06:43 <TD> i think the problem is that sophos has taken an interest in the sst files
277 2012-07-21 15:08:06 <TD> justmoon: another issue is the crap way i implemented migration. it rewrites blk000?.dat
278 2012-07-21 15:08:12 <TD> for basically no good reason
279 2012-07-21 15:08:17 <TD> so that's a lot of unnecessary disk io right there
280 2012-07-21 15:08:56 <TD> even so. disabling the AV on-access scanning sped things up a lot
281 2012-07-21 15:09:26 <TD> it's doing about 4h of chain per second now.
282 2012-07-21 15:09:43 <justmoon> yeah, I just turned off eset and from a very unscientific look at the timings I think it's 30-50% faster now
283 2012-07-21 15:10:05 <TD> eset?
284 2012-07-21 15:10:13 <justmoon> best antivirus for windows
285 2012-07-21 15:10:18 <TD> ah
286 2012-07-21 15:10:40 <justmoon> technically eset is the company and NOD32 is the program
287 2012-07-21 15:10:53 <TD> right
288 2012-07-21 15:10:56 <TD> i've heard of nod32
289 2012-07-21 15:12:29 <justmoon> ok, so here are some timings with av on: http://pastebin.com/8UfsnxRd
290 2012-07-21 15:13:18 <justmoon> and with av off: http://pastebin.com/60ZnNV4K
291 2012-07-21 15:15:09 <TD> ok that looks kind of reasonable
292 2012-07-21 15:15:53 <TD> it's reliable <1msec per tx though
293 2012-07-21 15:16:20 <TD> i'm still seeing a lot more variability than you
294 2012-07-21 15:21:40 <justmoon> it looks like the migration is cpu-bound for me - bitcoin process is consistently at 25% utilization, i.e. one core
295 2012-07-21 15:25:08 <TD> yeah
296 2012-07-21 15:25:15 <TD> that's what i was seeing as well in the best case
297 2012-07-21 15:25:21 <TD> for some reason on my laptop it's not
298 2012-07-21 15:25:50 <TD> oh well
299 2012-07-21 15:55:24 <TD> justmoon: https://github.com/mikehearn/bitcoin/commits/leveldb
300 2012-07-21 15:55:32 <TD> justmoon: could you take a quick glance over this? i'll submit it to gavin later
301 2012-07-21 15:55:44 <justmoon> TD: okidoki
302 2012-07-21 15:55:59 <TD> it's gonna take several rounds of review, i'd imagine ....
303 2012-07-21 15:57:43 <luke-jr> TD: hmm, GitHub thinks you have some other pullreqs in there :x
304 2012-07-21 15:57:46 <luke-jr> https://github.com/mikehearn/bitcoin/compare/leveldb
305 2012-07-21 15:59:10 <luke-jr> TD: is it possible to build a bitcoind supporting multiple database engines? if not, maybe USE_DBENGINE=bdb/leveldb ?
306 2012-07-21 16:00:24 <TD> that's weird
307 2012-07-21 16:00:29 <TD> why does it show up those extra commits?
308 2012-07-21 16:00:53 <TD> luke-jr: it's not possible with the way the code is written and there is already indeed a USE_LEVELDB build flag
309 2012-07-21 16:01:31 <luke-jr> TD: I mean for the makefiles, not the ifdefs
310 2012-07-21 16:01:51 <TD> yeah, there is. qmake USE_LEVELDB=-
311 2012-07-21 16:02:16 <TD> same for makefile.unix, etc
312 2012-07-21 16:02:28 <TD> it defaults to on
313 2012-07-21 16:02:32 <luke-jr> & I meant using USE_DBENGINE=a/b/c instead of that
314 2012-07-21 16:03:03 <TD> sure. i'm hoping we can scrap the bdb txdb code at some point though. the only reason there's a build flag at all is in case we want to quickly go back to the old system before a release
315 2012-07-21 16:03:24 <luke-jr> the entire leveldb code shouldn't be copied into the bitcoind git repo either :/
316 2012-07-21 16:03:27 <TD> if we end up supporting >2 txdb engines, there'd better be a really good reason for that
317 2012-07-21 16:03:31 <TD> i discussed that with gavin
318 2012-07-21 16:03:46 <TD> we need a slightly branched leveldb, also, the upstream is not packaged and does not support things like "make install"
319 2012-07-21 16:03:53 <luke-jr> -.-
320 2012-07-21 16:04:05 <TD> we need everyone to be on the fastest code, otherwise slow nodes drag everyone else down. so we want this to be the default, really
321 2012-07-21 16:04:23 <luke-jr> fix upstream
322 2012-07-21 16:04:24 <TD> otherwise some builds/distro-packages will just skip it and use bdb. easiest solution is statically linking it in
323 2012-07-21 16:04:39 <luke-jr> distros won't static link it in
324 2012-07-21 16:04:47 <luke-jr> you're just making them just through hoops to bypass it
325 2012-07-21 16:04:54 <gmaxwell> luke-jr: the database engine is part of our distributed algorithim, effectively. It was an error to use a system provided one in the first place.
326 2012-07-21 16:04:55 <TD> it's included in the code, so they have no choice short of major patching, which they should not do
327 2012-07-21 16:05:25 <TD> but yes, distros can and will inflict major brain-damage on whatever code we release. that sort of thing is why we should encourage linux users to use our own binaries or packages
328 2012-07-21 16:05:37 <luke-jr> TD: more likely to major patch than package it with the static link
329 2012-07-21 16:05:41 <TD> upstream doesn't want to include the patches we're using (i checked) because they don't want a boost dependency
330 2012-07-21 16:05:53 <luke-jr> TD: then do it without boost&?
331 2012-07-21 16:06:06 <luke-jr> users should never run binaries not provided by their distro
332 2012-07-21 16:06:16 <TD> i'd say the exact opposite :)
333 2012-07-21 16:06:17 <TD> anyway
334 2012-07-21 16:06:33 <TD> if you want to port leveldb to win32 directly, go right ahead. i've got better things to do, as has justmoon
335 2012-07-21 16:06:51 <TD> (and get it included into upstream, and make upstream packaged, and ensure distributors include those packages, and build with leveldb, etc)
336 2012-07-21 16:06:58 <gmaxwell> luke-jr: In any case it's not a 'static link' we should just build it internally so there is no conflict with the system (as if the system would package leveldb to begin with)
337 2012-07-21 16:07:11 <luke-jr> gmaxwell: same thing
338 2012-07-21 16:07:17 <luke-jr> it's bad practice
339 2012-07-21 16:07:44 <justmoon> TD: porting for the native windows build should be pretty trivial at this point, so I may very well do it
340 2012-07-21 16:07:49 <TD> oh, urgh. i bet i know why these commits appeared. i did a rebase with them in at some point
341 2012-07-21 16:07:53 <luke-jr> FreeSWITCH does the same nonsense, except with like 20+ libraries, so they're just not supported by any distro
342 2012-07-21 16:08:01 <TD> which i guess removed the merge commits.
343 2012-07-21 16:08:08 <TD> i wonder if it matters
344 2012-07-21 16:08:29 <TD> luke-jr: boohoo. i used to work on Wine, as an upstream developer. distro packaging was a net loss for us and our users, every time
345 2012-07-21 16:08:44 <TD> luke-jr: actually we just refused to bug fix or support any user who ran distro-provided packages because they were so broken
346 2012-07-21 16:08:52 <justmoon> TD: if you want me to test/review it with boost 1.49 it'll have to wait until tomorrow
347 2012-07-21 16:08:52 <TD> i suspect freeswitch are quite happy with their state of affairs
348 2012-07-21 16:09:11 <TD> justmoon: does the boost version matter? the current code seems to work
349 2012-07-21 16:09:35 <luke-jr> TD: doesn't mean it's good for the users
350 2012-07-21 16:09:59 <TD> users were happier when using the upstream provided packages, in our case. distributors tended to introduce subtle bugs as part of "improving" the software
351 2012-07-21 16:10:03 <TD> also see: debian openssl
352 2012-07-21 16:10:04 <gribble> New news from bitcoinrss: postfan opened issue 1618 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/issues/1618>
353 2012-07-21 16:10:08 <gmaxwell> luke-jr: thats for the best wrt freeswitch.
354 2012-07-21 16:10:26 <luke-jr> TD: that's a problem with distros "improving" software; but forcing them to fix bugs ENCOURAGES distro meddling
355 2012-07-21 16:10:46 <luke-jr> and yes, static linking/embedding IS a bug
356 2012-07-21 16:10:51 <gmaxwell> (its better than no one use freeswitch)
357 2012-07-21 16:11:33 <luke-jr> gmaxwell: ?
358 2012-07-21 16:11:34 <TD> including source isn't a bug. it's just a policy decision. in the event that leveldb does a new upstream release, we can just include it and retest
359 2012-07-21 16:11:52 <gmaxwell> luke-jr: Again, the _exact_ behavior of the DB is of critical importance to the distributed bitcoin algorithim. Some modification upstream considers minor like capping the maximum number of records updated in a transaction would potentially destroy bitcoin.
360 2012-07-21 16:12:01 <TD> luke-jr: ok, mis-included commits fixed
361 2012-07-21 16:12:13 <luke-jr> gmaxwell: if it is, that is a bug. db behaviour cannot be part of the protoco;l
362 2012-07-21 16:12:31 <gmaxwell> luke-jr: It's necessarily part of the distributed algorithim.
363 2012-07-21 16:13:06 <gmaxwell> Bitcoin's blockchain is a distributed database... that all nodes have to agree on. So yes the behavior of that database really matters.
364 2012-07-21 16:13:16 <luke-jr> it isn't. the example you gave is clearly a bug, as its behaviour is non-deterministic
365 2012-07-21 16:13:24 <luke-jr> a reorg may overflow the limit in some nodes, but not others
366 2012-07-21 16:13:39 <gmaxwell> As far as freeswitch goes, it's OT, I'll tell you the funny story in private. :)
367 2012-07-21 16:13:59 <gmaxwell> luke-jr: sure because upstream has changed behavior... and bitcoin on some nodes and not others are linking to a different version with different behavior.
368 2012-07-21 16:14:14 <luke-jr> no, because different nodes have different views of the network at different times
369 2012-07-21 16:14:16 <gmaxwell> They might have widely announced this change. But that doesnt help us.
370 2012-07-21 16:14:30 <justmoon> TD: it looks like the update to boost 1.49 is already merged in bitcoin master, so you just need to rebase correctly and the difference should disappear, I'll test with 1.49 on win tomorrow
371 2012-07-21 16:14:44 <luke-jr> if a reorg only has to deal with 1 block, it may succeed, whereas a node which had 2 blocks deep in the old chain might fail
372 2012-07-21 16:14:56 <TD> justmoon: i pushed a rebase just now, so i think it looks good now
373 2012-07-21 16:15:15 <gmaxwell> luke-jr: yes, right. There can be no maximum. But if upstream introduces one, it's game over. Which is why the behavior of the database is part of the algorithim, because it can't be permitted to introduce algorithim incompatible limits.
374 2012-07-21 16:15:50 <TD> luke-jr: look at gregorys point a different way. there can be all kinds of bugs in the database layer. if some people get linked against a version that is bug free but dramatically slower for some reason, it's still gonna hose some segment of the network
375 2012-07-21 16:16:19 <TD> some dependencies are just really critical and we need to ensure we control when they're upgraded so they can be tested appropriately
376 2012-07-21 16:16:47 <luke-jr> TD: this is going back to centralization of client
377 2012-07-21 16:17:13 <luke-jr> the network needs to survive regardless of what software any given node runs
378 2012-07-21 16:17:41 <TD> you'll be glad to hear that bluematt did a ton of work to turn bitcoinj into a fully validating implementation, btw :)
379 2012-07-21 16:17:53 <TD> so who knows. maybe implementation diversity isn't far off
380 2012-07-21 16:17:57 <TD> though it's still kinda scary
381 2012-07-21 16:18:00 <TD> for now though, this codebase is it
382 2012-07-21 16:18:52 <gmaxwell> luke-jr: I'm not saying that everything has to be the same. I'm saying that it all has to be tested and sanity checked for our application.
383 2012-07-21 16:19:23 <luke-jr> TD: it shouldn't be merged so long as it requires a static leveldb IMO
384 2012-07-21 16:19:36 <gmaxwell> The authors of bitcoin software have to take direct responsiblity for making sure that their underlying code doesn't impose any behavioral requirements which would be incompatible with the distributed algorithim. And you can't reliable do that if its changing out from under you.
385 2012-07-21 16:19:39 <TD> gavin can make that call
386 2012-07-21 16:19:43 <TD> btw, amusing/irritating example of how a distributor got their packages blacklisted from upstream developer support in the Wine project
387 2012-07-21 16:19:57 <TD> some genius decided to break the wine package which was one single package upstream, into several smaller ones
388 2012-07-21 16:20:14 <TD> so they decided regedit was a utility, and put it into wine-utils or something like that.
389 2012-07-21 16:20:32 <luke-jr> that sounds logical
390 2012-07-21 16:20:38 <TD> so users stopped getting it installed by default.
391 2012-07-21 16:21:13 <gmaxwell> But regedit is a core part of the win32 system, and you can't run many installers without it.
392 2012-07-21 16:21:16 <TD> FAIL. it has a command line interface and some app installers expect it to be there. they effectively shell out to it, as part of the install process. because it's always present on windows, these apps did not check if the operation succeeded. it cannot fail
393 2012-07-21 16:21:28 <TD> what's more, some apps expect registry entries to be there (because they can't be  installed without them)
394 2012-07-21 16:21:33 <TD> yep, gmaxwell has it
395 2012-07-21 16:21:45 <TD> and the app that got reported to us as "fails on wine" corrupted its memory if some registry entries weren't there
396 2012-07-21 16:21:51 <gmaxwell> ha.. yea, so you'll actually get windows apps mysterously failing. Sucks.
397 2012-07-21 16:22:09 <TD> i burned a couple of evenings trying to reproduce the reported crash until the reported mentioned he was on debian
398 2012-07-21 16:22:12 <luke-jr> sounds like it doesn't support Windows 3.11!
399 2012-07-21 16:22:57 <gmaxwell> Perhaps, but wine's purpose is largely to run the diverse set of vulgar windows applications out there. "The app is bugged" is not much of a useful answer.
400 2012-07-21 16:23:06 <TD> this was a few years ago
401 2012-07-21 16:23:16 <TD> of course it didn't support windows 3.1, though regedit is actually present on that version too
402 2012-07-21 16:23:19 <TD> (it did something a bit different)
403 2012-07-21 16:23:43 <gmaxwell> Likewise, when the bitcoin chain forms two substantial forks "node's on the other half are bugged" isn't much of a consolation to the people suffering, and getting robbed, from the enormous mess left behind.
404 2012-07-21 16:24:05 <luke-jr> well, I would say the correct answer in that case, is that WindowsAppX should be a package too, with dependencies on WINE and regedit ;)
405 2012-07-21 16:24:23 <luke-jr> actually, the WINE pkg should probably be a metapkg depending on regedit+other parts
406 2012-07-21 16:24:58 <TD> sure, if he had ensured that "apt-get install wine" did exactly the same thing as upstream, in a more complicated way, no problem. but of course whenever you get people introducing pointless complexity, you get bugs
407 2012-07-21 16:25:04 <gmaxwell> In any case, if we're really going to mandate the included leveldb we should insert a call into the code that checks the leveldb version and has a big nasty comment with it on how you kill puppies if you build the software with anything else and don't personally validate that it won't fail.
408 2012-07-21 16:25:43 <TD> yeah. i don't know if leveldb has a function that reports its version, actually, but i can check
409 2012-07-21 16:25:50 <luke-jr> either bdb support should remain, system leveldb should be supported, or we're forcing distros to hack things
410 2012-07-21 16:25:52 <TD> maybe there's something in the header files
411 2012-07-21 16:26:06 <TD> no distro is "forced" to hack things. that's a policy choice they make
412 2012-07-21 16:26:09 <TD> it's a dumb one
413 2012-07-21 16:26:15 <luke-jr> if we really need a leveldb fork, that should be a separate git repo we can embed into the main one
414 2012-07-21 16:26:29 <TD> anyway, it's also important linux users upgrade to follow new whitelisted scripts and so on. distros have a terrible track record of this
415 2012-07-21 16:26:32 <luke-jr> TD: willfully creating a bug (static linked fork) forces distros to fix it
416 2012-07-21 16:26:33 <gmaxwell> ^ thats probably a reasonable thing to do.
417 2012-07-21 16:26:47 <gmaxwell> luke-jr: we static link wallet.cpp. :)
418 2012-07-21 16:26:51 <TD> i'd actually prefer to detect somehow if the app had been built on linux build servers and then refuse to start :)
419 2012-07-21 16:27:02 <luke-jr> gmaxwell: that's actually part of the project code
420 2012-07-21 16:27:12 <gmaxwell> luke-jr: so would leveldb be in this case.
421 2012-07-21 16:27:30 <luke-jr> gmaxwell: no.
422 2012-07-21 16:27:40 <gmaxwell> TD: yea, a version whitelisting sounds good.. if upstream doesn't have a get_version() then surely they'd take a patch to add one.
423 2012-07-21 16:28:13 <gmaxwell> luke-jr: would you be made happier if we also renamed it to bitlevel_db ? :)
424 2012-07-21 16:28:24 <luke-jr> gmaxwell: no. -.-
425 2012-07-21 16:28:38 <luke-jr> gmaxwell: there is no need for a fork
426 2012-07-21 16:30:40 <luke-jr> whereas forcing a static fork of leveldb creates a real reason for someone to fork bitcoind (to fix it)
427 2012-07-21 16:33:19 <galambo_> you could always wait for the final product and testing before making a decision
428 2012-07-21 16:34:41 <galambo_> i dont quite understand the extent of this change though. would leveldb replace oracle db entirely
429 2012-07-21 16:35:24 <TD> ok, there's a leveldb::kMajorVersion and leveldb::kMinorVersion
430 2012-07-21 16:35:30 <TD> so not a function but constants, just as good
431 2012-07-21 16:35:31 <gmaxwell> TD: Whats the leveldb upstream position on file format compatiblity?
432 2012-07-21 16:36:04 <TD> i am not expecting leveldb to evolve much further, honestly. it's "done" in the sense that it has met its needs.
433 2012-07-21 16:36:13 <gmaxwell> Right.
434 2012-07-21 16:36:21 <TD> if i compare to bigtable, there's not much delta now they did bloom filtering
435 2012-07-21 16:36:29 <TD> so i don't really know what their format compatibility policy is
436 2012-07-21 16:36:59 <luke-jr> & so it's unmaintained too? -.-
437 2012-07-21 16:37:01 <gmaxwell> It's just a disk backed hashtable with transaction updates. Whoptie do.  And exactly what we need.
438 2012-07-21 16:38:11 <TD> SetBestChain: new best=0000000000000506cbbf  height=190107  work=395968689014181958764  date=07/21/2012 16:06:48
439 2012-07-21 16:38:21 <TD> hmph. still not as fast as we need it to be.
440 2012-07-21 16:38:29 <TD> oh well
441 2012-07-21 16:38:49 <gmaxwell> TD: was that node hot-cache on the memory pool?
442 2012-07-21 16:38:55 <TD> probably not
443 2012-07-21 16:39:05 <TD> i'm seeing lots of orphan tx messages after it
444 2012-07-21 16:39:23 <luke-jr> -logtimestamp?
445 2012-07-21 16:39:30 <gmaxwell> o_O
446 2012-07-21 16:39:35 <gmaxwell> Yea.. although
447 2012-07-21 16:39:46 <luke-jr> -logtimestamps*
448 2012-07-21 16:39:48 <TD> ah
449 2012-07-21 16:39:49 <TD> thanks
450 2012-07-21 16:39:52 <TD> didn't know about that
451 2012-07-21 16:40:28 <gmaxwell> TD: they dont have timestamps just to reduce their value for someone trying to deanonymize nodes to go around demanding logs from people.
452 2012-07-21 16:40:35 <TD> yeah, i remember
453 2012-07-21 16:41:27 <luke-jr> 349 MB saved by pruning index&
454 2012-07-21 16:41:39 <luke-jr> (per datadir)
455 2012-07-21 16:42:46 <gmaxwell> Ultraprune is a lot more interesting, as it reduces the working set to something like 100mb now.
456 2012-07-21 16:43:05 <luke-jr> gmaxwell: what more does it get rid of?
457 2012-07-21 16:43:23 <luke-jr> gmaxwell: I'm just trying to make a one-shot "base datadir" to copy everywhere (and hardlink blk0001.dat)
458 2012-07-21 16:43:29 <gmaxwell> luke-jr: it seperate block storage from the data needed for validation.
459 2012-07-21 16:43:41 <luke-jr> O.o
460 2012-07-21 16:43:47 <luke-jr> that's & already separate?
461 2012-07-21 16:44:05 <gmaxwell> so basically the blocks are just for archival. Then there is a 100mbyte txout database, which _only_ indexes txouts.
462 2012-07-21 16:44:24 <luke-jr> i c
463 2012-07-21 16:44:33 <gmaxwell> So the only time it ever reads the blocks is when feeding a new node, or during a reorg.
464 2012-07-21 16:44:51 <luke-jr> I guess I'd notice if my own private nodes break, so I can take this risk
465 2012-07-21 16:45:07 <gmaxwell> The blocks are also seperated into many files, so sharing them would be easier.
466 2012-07-21 16:45:44 <luke-jr> -rw-r--r-- 1 luke-jr luke-jr 2097361271 Jul 21 14:48 blk0001.dat
467 2012-07-21 16:45:46 <luke-jr> -rw-r--r-- 1 luke-jr luke-jr  158613504 Jul 21 18:36 blkindex.dat
468 2012-07-21 16:50:32 <gribble> New news from bitcoinrss: mikehearn opened pull request 1619 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/pull/1619>
469 2012-07-21 16:53:15 <TD> luke-jr: btw thanks for implementing fee dependency calculations
470 2012-07-21 16:54:09 <luke-jr> TD: Gavin seems to be reinventing that change (probably for the better), so hopefully he'll add that feature back in before merging it
471 2012-07-21 16:56:15 <gmaxwell> hm. leveldb seems to be light on tests... we should probably have at least a buildtime test of a large reorg.
472 2012-07-21 16:57:22 <TD> yeah, some tests of the txdb code would be good in general (doesn't have to be ldb specific)
473 2012-07-21 16:58:31 <luke-jr> some of my pullreqs have some
474 2012-07-21 16:58:37 <luke-jr> but they do focus on other stuff
475 2012-07-21 16:59:07 <luke-jr> (in particular, it sets up the db env in memory and builds a 100+ block chain)
476 2012-07-21 16:59:21 <luke-jr> (speaking of which, does LevelDB work with memory-only db environmentsG)
477 2012-07-21 16:59:52 <copumpkin> does anyone know of a document (maybe threads on the forum?) discussing all sorts of reorg scenarios and how things would work during them?
478 2012-07-21 17:00:04 <gmaxwell> I suppose that test should actually run on disk in any case the rationale being that the db enviroment may have different bugs for those cases.
479 2012-07-21 17:00:14 <copumpkin> I have this unit test which looks handy: https://bitcointalk.org/index.php?topic=46370.0
480 2012-07-21 17:00:24 <copumpkin> but was curious about other higher-level discussions of the behavior
481 2012-07-21 17:00:32 <copumpkin> I'd like to implement it correctly, obviously :)
482 2012-07-21 17:00:38 <TD> luke-jr: hm, not really. i guess you can start a transaction and never finish it. that would amount to the same thing
483 2012-07-21 17:00:53 <TD> (well, a batch)
484 2012-07-21 17:00:59 <TD> though it does exercise very different codepaths
485 2012-07-21 17:01:00 <gmaxwell> copumpkin: I mean, the high level behavior is that you must end up in an identical state to the dead fork never having existed.
486 2012-07-21 17:01:01 <luke-jr> test_bitcoin shouldn't touch my datadir IMO
487 2012-07-21 17:01:19 <luke-jr> I suppose a temp dir would be workable
488 2012-07-21 17:01:28 <TD> yeah. tests usually have a temp dir of their own
489 2012-07-21 17:01:34 <gmaxwell> luke-jr: yea, a temp dir, obviously.
490 2012-07-21 17:02:25 <TD> didn't sipa refactor things so re-orgs no longer create gigantic db transactions?
491 2012-07-21 17:02:27 <luke-jr> my tests are for the block creation and smart time code mainly, so what is ideal for db testing wasn't the focus
492 2012-07-21 17:02:34 <luke-jr> TD: in some cases they need to
493 2012-07-21 17:02:50 <TD> or rather, he changed things so the code doesn't expect an uncommitted db txn to be reflected in read operations
494 2012-07-21 17:03:15 <luke-jr> TD: all the reverse-a-blocks need to be in the same txn still
495 2012-07-21 17:03:18 <TD> there is some inefficient O(N) code in there that allows you to read back from an uncommitted batch, but i think sipa removed the need for it. at least in the current code.
496 2012-07-21 17:03:22 <luke-jr> and the same number of blocks forward on the new chain
497 2012-07-21 17:03:32 <TD> but yeah, good point about the huge re-orgs
498 2012-07-21 17:03:34 <copumpkin> gmaxwell: I mean the conditions under which you'd get branches in the first place, especially ones starting farther back
499 2012-07-21 17:04:25 <gmaxwell> copumpkin: the only condition which should create them is block races.
500 2012-07-21 17:04:38 <copumpkin> even in the presence of evil nodes?
501 2012-07-21 17:04:41 <gmaxwell> Broken nodes do too, however.
502 2012-07-21 17:05:07 <copumpkin> for example, what kind of rules are there on relaying really weird older chains?
503 2012-07-21 17:05:25 <gmaxwell> copumpkin: You only relay blocks from your current best chain.
504 2012-07-21 17:05:31 <gmaxwell> You will not relay a disconnected block.
505 2012-07-21 17:06:01 <copumpkin> so unless a broken/malicious node relays you a chain branched off way back directly, you probably won't see them
506 2012-07-21 17:06:52 <gmaxwell> Correct. And you won't do anything with them when you do see them accept keep them around... you'll only do things with them if they become part of the longest.
507 2012-07-21 17:09:30 <luke-jr> hrm, blk0001.dat doesn't compress very well :<
508 2012-07-21 17:09:54 <gmaxwell> You expected otherwise?
509 2012-07-21 17:09:57 <copumpkin> gmaxwell: I see, I think
510 2012-07-21 17:10:54 <copumpkin> a lot of this logic doesn't seem too hard to specify formally in a language like agda. The difficulty proofs of breaking hashes and stuff I wouldn't want to touch, but protocol rules and the like are nice and discrete and could be encoded fairly easily
511 2012-07-21 17:12:17 <copumpkin> at least at first glance :)
512 2012-07-21 17:12:28 <copumpkin> it's fairly easy to encode a stack language like that
513 2012-07-21 17:12:49 <copumpkin> if only I had more time for this stuff :)
514 2012-07-21 17:14:44 <luke-jr> gmaxwell: yes
515 2012-07-21 17:14:50 <luke-jr> most of blk0001.dat should be redundant
516 2012-07-21 17:15:17 <copumpkin> you mean all the repeated hashes as references?
517 2012-07-21 17:15:52 <copumpkin> it doesn't seem like there'd be that much repetition, but you could compress hash references into simply file offset references once you've verified them
518 2012-07-21 17:16:10 <gmaxwell> luke-jr: it's not like your compressor knows how and where to runs sha256. :)
519 2012-07-21 17:20:54 <luke-jr> gmaxwell: still, multiuse addresses, numbers, etc&
520 2012-07-21 17:31:27 <xorgate> suppose i wish to make a program that uses bitcoind rpc interface to get a list of received txs, and send them back. how would i figure out what address to send to, i know the tx id
521 2012-07-21 17:32:25 <gmaxwell> xorgate: It's really inadvisable to try to do that.
522 2012-07-21 17:32:40 <luke-jr> xorgate: impossible with bitcoind
523 2012-07-21 17:32:46 <gmaxwell> There is no 'from' address in bitcoin, the closest you can get is the prior destination.
524 2012-07-21 17:32:54 <luke-jr> xorgate: and it's possible that it wasn't even sent from a valid address
525 2012-07-21 17:33:14 <xorgate> how does satoshidice do this/work around this ?
526 2012-07-21 17:33:20 <gmaxwell> But the prior destination may not be representable as an address, the sender may have no way of recovering the coins anymore, or they might go to another party entirely.
527 2012-07-21 17:33:22 <luke-jr> xorgate: it doesn't work right
528 2012-07-21 17:34:22 <copumpkin> it works for most transactions it receives, but it's kind of an incorrect model
529 2012-07-21 17:34:31 <copumpkin> to ever try to send coins "back to the sender"
530 2012-07-21 17:35:21 <luke-jr> sigh, 1.3 GB
531 2012-07-21 17:35:31 <luke-jr> I was going to publish this, but now I'm not sure I want to :P
532 2012-07-21 17:35:42 <copumpkin> oh, a signed blk001.dat?
533 2012-07-21 17:35:47 <copumpkin> +0
534 2012-07-21 17:36:01 <luke-jr> copumpkin: I could sign it, but the main reason is because it's the minimal blk0001.dat that can just be hardlinked everywhere
535 2012-07-21 17:36:23 <gmaxwell> its trivial enough to make your own.
536 2012-07-21 17:36:30 <xorgate> oof. did i miss anything? last was 'it doesn't work right'
537 2012-07-21 17:36:32 <copumpkin> xorgate: we were telling you that the basic model of "send back to sender" doesn't make sense in bitcoin the protocol, but works in most cases right now
538 2012-07-21 17:36:38 <gmaxwell> and non-trivial enough to install it in a way that can be hardlinked.
539 2012-07-21 17:36:54 <luke-jr> gmaxwell: it's not really trivial to make IMO
540 2012-07-21 17:37:15 <gmaxwell> luke-jr: you just -connect to another isolated node that has the full chain, no?
541 2012-07-21 17:37:30 <luke-jr> gmaxwell: hmm, I wonder if that might work
542 2012-07-21 17:37:37 <luke-jr> I wrote a Python script to build it <.<
543 2012-07-21 17:37:55 <gmaxwell> it should work.
544 2012-07-21 17:38:04 <luke-jr> so long as the other node is isolated..
545 2012-07-21 17:38:18 <luke-jr> if it receives a block while you're doing it, it'll send that first and go out of order :/
546 2012-07-21 17:38:25 <gmaxwell> right.
547 2012-07-21 17:38:43 <luke-jr> plus, you'd need to do some magic to get it to stop at the right block
548 2012-07-21 17:38:59 <luke-jr> at least blkindex is only 61 MB xz'd
549 2012-07-21 17:41:50 <luke-jr> gmaxwell: wizkid057 claims to have converted the file to hex plaintext and compressed it under 200 MB :|
550 2012-07-21 17:42:01 <wizkid057> not hex
551 2012-07-21 17:42:03 <luke-jr> oh?
552 2012-07-21 17:42:27 <luke-jr> then?
553 2012-07-21 17:42:29 <wizkid057> the result of getblock for every non-orphan
554 2012-07-21 17:42:41 <wizkid057> in plaintext
555 2012-07-21 17:42:58 <wizkid057> then bzip2'd
556 2012-07-21 17:43:02 <luke-jr> 7aac5826b91b4f87a2e9534e0e38e8d64ed21aff8a4eb8ff8dde4e726e67fe1a  blk0001.dat
557 2012-07-21 17:43:10 <luke-jr> wizkid057: oh, I bet that's missing a lot of info
558 2012-07-21 17:43:18 <wizkid057> its every block
559 2012-07-21 17:43:22 <wizkid057> how could it be missing anything?
560 2012-07-21 17:44:51 <luke-jr> wizkid057: scripts?
561 2012-07-21 17:45:16 <wizkid057> http://pool.wizkid057.com/block190127.txt
562 2012-07-21 17:45:46 <wizkid057> i never bothered to look at tx, but, i'd assume thats the transactions in some form...
563 2012-07-21 17:45:49 <copumpkin> yeah, that doesn't contain the full transactions
564 2012-07-21 17:46:00 <wizkid057> well that'd explain things then :P
565 2012-07-21 17:46:16 <copumpkin> :)
566 2012-07-21 17:46:29 <wizkid057> so wait, whats tx then in this context? lol
567 2012-07-21 17:46:53 <copumpkin> the hash of the transaction data, I think
568 2012-07-21 17:46:56 <luke-jr> wizkid057: it's just the txid
569 2012-07-21 17:47:02 <wizkid057> ah
570 2012-07-21 17:47:06 <wizkid057> so need a lot more data then
571 2012-07-21 17:47:15 <copumpkin> for example, you take the top guy on that list
572 2012-07-21 17:47:18 <copumpkin> and feed it into http://blockchain.info/tx-index/12988316/6985f5f9f02bf2276834e2eef18347059ff144d8048e075e4b4146f415a0c90d
573 2012-07-21 17:47:24 <luke-jr> wizkid057: there USED to be a way to get the full data, but some people insisted on removing it :<
574 2012-07-21 17:47:26 <copumpkin> it's 143 bytes, that particular one
575 2012-07-21 17:47:32 <wizkid057> :-\n4079428
576 2012-07-21 17:48:00 <luke-jr> now you get to write a script to iterate over each transaction and call getrawtransaction on it
577 2012-07-21 17:48:14 <copumpkin> at that point, you might as well consume the blk0001.dat yourself
578 2012-07-21 17:48:21 <wizkid057> lol
579 2012-07-21 17:48:25 <copumpkin> it isn't that much work
580 2012-07-21 17:48:43 <wizkid057> well, i actually was just dumping the getblock data to do some other statistics
581 2012-07-21 17:49:09 <wizkid057> saw a block of hex for "tx" and just assumed that was the transactions in raw form, never bothered to check them
582 2012-07-21 17:49:15 <wizkid057> since I didnt need them
583 2012-07-21 17:49:16 <wizkid057> hehe
584 2012-07-21 17:49:25 <copumpkin> so does bitcoind trust its on-disk store? or does it verify hashes when it starts up?
585 2012-07-21 17:49:47 <copumpkin> I noticed it does take a while to start up :)
586 2012-07-21 17:49:54 <copumpkin> well, bitcoin-qt anyway
587 2012-07-21 17:51:02 <wizkid057> i used to be pretty badass with compression of structured data a while back using some custom mangling
588 2012-07-21 17:51:15 <luke-jr> copumpkin: it mostly trusts it
589 2012-07-21 17:51:24 <luke-jr> copumpkin: recent versions check the last 1000 blocks or somehting
590 2012-07-21 17:51:27 <copumpkin> I see
591 2012-07-21 17:51:36 <wizkid057> luke-jr: how small can you get your stripped blk0001.dat?
592 2012-07-21 17:51:42 <luke-jr> wizkid057: 1.3 GB
593 2012-07-21 17:52:16 <wizkid057> would probably need a custom compression algo to get it smaller
594 2012-07-21 17:52:23 <luke-jr> -rw-r--r-- 1 luke-jr luke-jr 1356937600 Jul 21 14:48 blk0001.dat.xz
595 2012-07-21 17:52:25 <luke-jr> -rw-r--r-- 1 luke-jr luke-jr   63706028 Jul 21 18:36 blkindex.dat.xz
596 2012-07-21 17:53:38 <wizkid057> i mean, I could probably, with decent effort, make something specific to the blockchain to drop it down substantially at the cost of a lot of CPU to unpack it
597 2012-07-21 17:54:07 <luke-jr> probably
598 2012-07-21 17:54:15 <gmaxwell> Sipa has already done this... but it just wasn't that impressive.
599 2012-07-21 17:54:27 <luke-jr> gmaxwell: did he eliminate all hashes?
600 2012-07-21 17:54:31 <wizkid057> well, you have to think, you dont need to store the hashes
601 2012-07-21 17:54:34 <wizkid057> yeah, lol
602 2012-07-21 17:54:57 <wizkid057> as long as an ordering is maintained you could drop them
603 2012-07-21 17:54:59 <gmaxwell> and compressed the signatures... there is a bunch of stuff that can be done, but, meh.
604 2012-07-21 17:55:02 <luke-jr> for a single solid file, you could just store varint byte offsets
605 2012-07-21 17:55:40 <luke-jr> but yeah, I guess the sigs are pretty uncompressable :/
606 2012-07-21 17:55:46 <gmaxwell> forget byte offsets, you'd store optimally coded unsigned integers indexing the set of qualified transactions.
607 2012-07-21 17:55:56 <luke-jr> gmaxwell: true
608 2012-07-21 17:56:16 <gmaxwell> If you want to be really awesome you can limit the last txin index to the set of transactions which would be big enough to satisify the explicit outputs.
609 2012-07-21 17:56:22 <wizkid057> dont even really need to store it optimally if you can drop the bulk of the random data
610 2012-07-21 17:56:29 <wizkid057> generic compression algos arent bad at repeated bits
611 2012-07-21 17:56:34 <gmaxwell> (and by awesome I mean hundreds of lines of code to save like .. 50 bytes)
612 2012-07-21 17:57:25 <wizkid057> how many transactions are in the chain anyway?
613 2012-07-21 17:57:53 <luke-jr> 12988251 according to blockchain.info I guess
614 2012-07-21 17:58:08 <luke-jr> but that might include some not in the chain
615 2012-07-21 18:01:12 <wizkid057> hmm... so judging by that you could save at least a few hundred MB by dropping the uncompressable hashes
616 2012-07-21 18:02:20 <wizkid057> 13M hashes wouldnt take long, even on an older machine, to regenerate
617 2012-07-21 18:10:33 <cosurgi> what is the expected date when block reward drops to 25 BTC?
618 2012-07-21 18:10:43 <wizkid057> sometime in november i think
619 2012-07-21 18:10:50 <cosurgi> thx :)
620 2012-07-21 18:10:54 <gmaxwell> ;;bc,halfreward
621 2012-07-21 18:10:54 <gribble> Estimated time of bitcoin block reward halving: Thu Dec  6 12:10:00 2012 | Time remaining: 19 weeks, 5 days, 0 hours, and 0 seconds
622 2012-07-21 18:11:01 <wizkid057> oh december now
623 2012-07-21 18:11:19 <cosurgi> :)
624 2012-07-21 18:11:19 <gmaxwell> wizkid057: thats not new.
625 2012-07-21 18:11:29 <cosurgi> ;;bc,quarterreward
626 2012-07-21 18:11:30 <gribble> Error: "bc,quarterreward" is not a valid command.
627 2012-07-21 18:11:32 <cosurgi> ;-)
628 2012-07-21 18:11:34 <wizkid057> didnt know that command existed...
629 2012-07-21 18:11:42 <cosurgi> me neither
630 2012-07-21 18:11:50 <wizkid057> i did the math myself before and came up with end of november
631 2012-07-21 18:13:15 <cosurgi> I remember Anderson talking about a feature that multiple signatures will be needed to transfer money. A feature used for protecting banks from capturing wallets. He said - each key stored in different location. And all wallets wopuld need to be captured by evildoers to steal money.
632 2012-07-21 18:13:29 <cosurgi> Do you remember that he was talking about that?
633 2012-07-21 18:13:51 <cosurgi> I'm curious if it has been already implemented? I think that I first read about it 3 months ago or so.
634 2012-07-21 18:14:43 <cosurgi> if I spent some time searching I might even find the link for his post.
635 2012-07-21 18:17:12 <TD> cosurgi: yes
636 2012-07-21 18:17:20 <TD> cosurgi: parts of it are implemented
637 2012-07-21 18:21:47 <jgarzik> w00t
638 2012-07-21 18:22:03 <jgarzik> loadblock fails, but the very first initial block download succeeded!
639 2012-07-21 18:22:18 <jgarzik> (grafting a block db into half-a-node)
640 2012-07-21 18:31:27 <jgarzik> chain imports sure are fast, when you don't verify connectedness or signatures
641 2012-07-21 19:12:32 <graingert> it would be nice to use the system keychain for the password
642 2012-07-21 19:48:49 <jgarzik> -rw-rw-r-- 1 jgarzik jgarzik 2369881499 Jul 21 17:33 blocks.dat
643 2012-07-21 19:49:07 <jgarzik> gdbm database, key uint256, value serialized CBlock
644 2012-07-21 19:49:21 <TD> hmm
645 2012-07-21 19:49:31 <TD> does anyone know why the wallet contains supporting transactions?
646 2012-07-21 19:49:36 <TD> satoshi must have had a use case for this in mind
647 2012-07-21 19:50:42 <TD> oh, i see. it's used for relaying
648 2012-07-21 22:08:19 <jgarzik> hrm
649 2012-07-21 22:08:32 <jgarzik> do we already have duplicate TX hashes in the block chain?
650 2012-07-21 22:08:35 <jgarzik> d5d27987d2a3dfc724e359870c6644b40e497bdc0589a033220fe15429d88599
651 2012-07-21 22:11:06 <tcatm> I have that tx in block 91842 (a4d0a...).
652 2012-07-21 22:11:19 <copumpkin> 91812
653 2012-07-21 22:11:50 <copumpkin> hmm
654 2012-07-21 22:12:08 <tcatm> It's a coinbase, though. There were some broken miners creating duplicate transactions by mistake.
655 2012-07-21 22:35:07 <jgarzik> tcatm: it will be interesting when someone wants to spend...
656 2012-07-21 22:37:45 <jgarzik> trying to spend will cause a lookup on txid... which returns what?  most recent tx with that txid?  earliest tx with that txid?
657 2012-07-21 22:41:31 <copumpkin> if I were to guess, I'd guess the most recent :P
658 2012-07-21 23:21:23 <luke-jr> jgarzik: most recent.
659 2012-07-21 23:56:54 <cheako> theymos: Is there a way I can get the block number for 30days ago?  http://blockexplorer.com/b/200002  It seams like just guessing block numbers is a no-no.
660 2012-07-21 23:58:37 <cheako> I'd like to know how many new coins were added since block number X(first block on 2012-06-23 UTC) till 190148(this morning UTC)
661 2012-07-21 23:59:17 <cheako> wb Joric