1 2012-05-22 00:35:30 <gmaxwell> DEFS=-DMAC_OSX -DMSG_NOSIGNAL=0 -DUSE_IPV6
  2 2012-05-22 00:40:34 <jgarzik> 100 orphan tx's on the wall...
  3 2012-05-22 00:40:40 <luke-jr> /home/luke-jr/src/cross-osx/ii/bin/i686-apple-darwin9-ld: truncated or malformed archive: /home/luke-jr/src/cross-osx/ii/lib/libboost_filesystem-gcc40-mt-s-1_47.a at offset 162760 (member extends past the end of the file, can't load from it)
  4 2012-05-22 00:40:42 <luke-jr> hmm
  5 2012-05-22 00:41:07 <jgarzik> 110 orphans
  6 2012-05-22 00:43:34 <gmaxwell> jgarzik: only seen another 368 orphans since early this morning at my node.
  7 2012-05-22 00:44:08 <Diablo-D3> those orphans bother me
  8 2012-05-22 00:44:12 <Diablo-D3> what if its an attack to steal bitcoins
  9 2012-05-22 00:45:36 <gmaxwell> did finway possess your soul while we weren't looking?
 10 2012-05-22 00:46:07 <jgarzik> 152 orphans
 11 2012-05-22 00:47:03 <Diablo-D3> gmaxwell: ?
 12 2012-05-22 00:47:49 <gmaxwell> There is no reason to think its 'an attack to steal bitcoins' .. weird things go on all the time  and I don't even think the load I'm seing now is weird.
 13 2012-05-22 00:47:55 <gmaxwell> (last night OTOH was weird)
 14 2012-05-22 00:48:12 <luke-jr> probably someone trying to exploit the recent DDoS :p
 15 2012-05-22 00:49:36 <gmaxwell> luke-jr: at least the few I captured and decoded were dice transactions.
 16 2012-05-22 00:49:42 <luke-jr> oh
 17 2012-05-22 00:49:55 <luke-jr> someone did mention earlier dice being faster, not waiting for blocks
 18 2012-05-22 00:49:58 <luke-jr> maybe related?
 19 2012-05-22 00:50:01 <gmaxwell> could have just been timing.
 20 2012-05-22 00:50:15 <gmaxwell> luke-jr: it deals exculsively with unconfirmed transactions
 21 2012-05-22 00:50:26 <luke-jr> hm
 22 2012-05-22 00:50:43 <Diablo-D3> gmaxwell: I dunno, Im extra paranoid.,
 23 2012-05-22 01:19:56 <luke-jr> gmaxwell: any idea why linking has undefined symbols: ___stack_chk_guard ___stack_chk_fail
 24 2012-05-22 01:19:57 <luke-jr> ?
 25 2012-05-22 01:22:28 <gmaxwell> you're building with -fstack-protector but not linking libssp .. on linux that linking is hidden, but it's not always well hidden elsewhere.
 26 2012-05-22 01:22:48 <gmaxwell> (E.g. it's not hiddwn with mingw builds, but we don't use -fstack-protector with mingw for some reason)
 27 2012-05-22 01:26:07 <luke-jr> does Mac have it?
 28 2012-05-22 01:27:06 <gmaxwell> it should be built by gcc.
 29 2012-05-22 01:27:14 <gmaxwell> (it's effectively part of libgcc)
 30 2012-05-22 01:27:46 <luke-jr> 4.0?
 31 2012-05-22 01:29:12 <gmaxwell> You've now exhausted my knoweldge on the subject???just turn off the stack protector for now.
 32 2012-05-22 01:30:04 <luke-jr> XD
 33 2012-05-22 01:30:12 <luke-jr> bitcoind seems to be downloading the blockchain on Mac
 34 2012-05-22 01:30:25 <luke-jr> now to get it gitian finished
 35 2012-05-22 01:37:47 <jgarzik> orphans seem to have stopped just after I wrote the above.  checked just now... 152 before, 156 now.
 36 2012-05-22 01:37:58 <jgarzik> so, a burst of orphans
 37 2012-05-22 01:57:05 <Diablo-D3> https://imgur.com/y7Hm9
 38 2012-05-22 01:59:02 <copumpkin> (because I prove it correct first)
 39 2012-05-22 02:36:19 <gribble> New news from bitcoinrss: fanquake opened pull request 1374 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/pull/1374>
 40 2012-05-22 05:30:05 <SomeoneWeird> https://github.com/bitcoinjs/bitcoinjs-server/issues/73
 41 2012-05-22 05:38:21 <jgarzik> T-minus seven minutes to SpaceX launch
 42 2012-05-22 05:41:14 <Diablo-D3> t minus 4 minutes until launch: http://www.nasa.gov/multimedia/nasatv/index.html
 43 2012-05-22 05:48:15 <Joric> heard it was established by a former paypal founder... let's finally launch paypal into space!
 44 2012-05-22 05:57:37 <gribble> New news from bitcoinrss: rebroad opened pull request 1375 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/pull/1375>
 45 2012-05-22 06:13:01 <nathan7> well dammit
 46 2012-05-22 06:13:04 <nathan7> I missed it
 47 2012-05-22 06:18:06 <Joric> it's flying! http://www.nasa.gov/multimedia/nasatv/index.html
 48 2012-05-22 06:20:57 <mtve> hi!    where can i get v0.5.5?
 49 2012-05-22 06:29:03 <Eliel> interesting, it's not linked to from the main bitcoin.org page...
 50 2012-05-22 06:29:07 <Eliel> I can't locate it either
 51 2012-05-22 06:29:35 <mtve> havn't found it either in github nor in sf/svn
 52 2012-05-22 06:29:43 <Eliel> luke-jr: where did you hide it?
 53 2012-05-22 06:30:49 <Eliel> ah, google got me this https://bitcointalk.org/index.php?topic=79651.0
 54 2012-05-22 06:34:43 <mtve> great, thanks!  (now gitorious seems down to me, but that's another story)
 55 2012-05-22 07:16:29 <mtve> nevertheless, 0.6.2.2 fixes high cpu problem on freebsd too, will stay with it.  thanks again.
 56 2012-05-22 08:56:52 <gribble> New news from bitcoinrss: rebroad opened pull request 1376 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/pull/1376>
 57 2012-05-22 09:01:56 <gribble> New news from bitcoinrss: rebroad opened pull request 1377 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/pull/1377>
 58 2012-05-22 09:12:08 <gribble> New news from bitcoinrss: rebroad opened pull request 1378 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/pull/1378>
 59 2012-05-22 10:28:41 <drizztbsd> hi
 60 2012-05-22 10:29:10 <drizztbsd> luke-jr: damn pocoo (closed), do you still have your kvm script to do mining under kvm?
 61 2012-05-22 10:48:07 <luke-jr> drizztbsd: http://codepad.org/mmI1rwLa
 62 2012-05-22 10:58:36 <imsaguy2> wumpus, give me a few days.  I have to let it get behind so it will show the progress bar/text.
 63 2012-05-22 10:58:50 <imsaguy2> or I guess I could just clear my blockchain and redownload
 64 2012-05-22 11:36:15 <drizztbsd> luke-jr: so are you using another card for the host?
 65 2012-05-22 11:40:35 <luke-jr> drizztbsd: I'm using my APU's GPU
 66 2012-05-22 11:40:43 <drizztbsd> right :)
 67 2012-05-22 11:40:54 <drizztbsd> I could use my internal nvidia
 68 2012-05-22 11:41:27 <luke-jr> ewww :p
 69 2012-05-22 12:47:16 <diki> slush:I ran into a bit of a slump with Abe. Since the blockchain is now 1.4 gigs or more, Abe fails to run dying on mmap.mmap. Asking around I was told it is probably because mmap cannot address more space cause my python is 32bit. Yeah ok, but 64-bit python didn't have some modules required for running abe(or these modules failed to install under it).
 70 2012-05-22 12:47:36 <diki> Do you know of a fix?
 71 2012-05-22 12:55:04 <sol56> anyone know why ArmoryQt.py segfaults?
 72 2012-05-22 12:57:00 <slush> diki: Yes,it's the most likely because 32bit. As far as it needs mmap - no, i don't know the fix :(
 73 2012-05-22 12:57:14 <sol56> well im running 64bit debian
 74 2012-05-22 12:57:22 <slush> but 32bit python?
 75 2012-05-22 12:57:47 <sol56> dunno
 76 2012-05-22 12:57:50 <sol56> i dont think so
 77 2012-05-22 13:00:59 <slush> sol56: python -c 'import struct;print( 8 * struct.calcsize("P"))'
 78 2012-05-22 13:01:05 <slush> will print 32 or 64...
 79 2012-05-22 13:01:19 <sol56> root@Sucker:~# python -c 'import struct;print( 8 * struct.calcsize("P"))' 64 root@Sucker:~#
 80 2012-05-22 13:01:42 <sol56> is 64
 81 2012-05-22 13:03:23 <slush> sol56: I'm reading about mmap and there's no obvious reason why it should fail
 82 2012-05-22 13:03:46 <slush> can you obtain how much memory is it trying to allocate?
 83 2012-05-22 13:03:46 <sol56> its something about maping the blockchain
 84 2012-05-22 13:05:40 <sol56> COMMAND         %MEM python          16.6
 85 2012-05-22 13:05:55 <sol56> slush: so thats around when it crashes
 86 2012-05-22 13:06:02 <sol56> i have 16gb of RAM
 87 2012-05-22 13:07:40 <drizztbsd> diki: 32bit => 2gb
 88 2012-05-22 13:10:01 <diki> drizztbsd:yes I was aware of the limitations of a 32-bit OSs' process memory addressability
 89 2012-05-22 13:11:47 <drizztbsd> blockchain is 1.4gb :P
 90 2012-05-22 13:11:58 <sol56> yeah welll
 91 2012-05-22 13:12:01 <sol56> what the heck...
 92 2012-05-22 13:12:14 <diki> drizztbsd:but it still fails
 93 2012-05-22 13:13:07 <diki> It worked before
 94 2012-05-22 13:13:31 <diki> But I had to shutdown my PC, as I booted it up and tried to start abe(while the blkchain was still the same size) it failed
 95 2012-05-22 13:17:06 <diki> Fixed it
 96 2012-05-22 13:17:18 <diki> I downloaded this app that applies the Large Memory Aware flag
 97 2012-05-22 13:17:28 <diki> did it on the python process, and it works now
 98 2012-05-22 13:18:08 <jgarzik> mndrix: your SIGHUP patch was merged
 99 2012-05-22 13:18:34 <sol56> slush any ideas on the mmap fix?
100 2012-05-22 13:19:04 <slush> hm, no :-(
101 2012-05-22 13:19:09 <slush> I have to go, bye!
102 2012-05-22 13:19:09 <sol56> :(
103 2012-05-22 13:19:12 <sol56> cya
104 2012-05-22 13:19:40 <diki> sol56:I found one
105 2012-05-22 13:19:45 <diki> if you are on windows that is..
106 2012-05-22 13:19:51 <sol56> im running 64bit linux
107 2012-05-22 13:19:52 <sol56> so now
108 2012-05-22 13:19:53 <sol56> no*
109 2012-05-22 13:19:55 <sol56> lol
110 2012-05-22 13:20:44 <diki> Oh man..I really hope I won't have to re-import the blockchain in mysql
111 2012-05-22 13:21:08 <diki> usually when I quit abe I have to manually fix the blk offset in mysql
112 2012-05-22 13:21:24 <diki> and apply some queries to let me continue importing without gettign the duplicate error
113 2012-05-22 13:25:00 <diki> sometimes, very rarely
114 2012-05-22 13:25:07 <sol56> the irritating thing
115 2012-05-22 13:25:12 <diki> you would mess it up so bad with missing satoshi_seconds_destroyed
116 2012-05-22 13:25:16 <sol56> is its just Armory
117 2012-05-22 13:25:25 <sol56> if i delete the blockchain
118 2012-05-22 13:25:26 <diki> that you are forced to re-import
119 2012-05-22 13:25:29 <sol56> it starts
120 2012-05-22 13:25:32 <sol56> yeah
121 2012-05-22 13:30:04 <diki> All is working well, but I have 10-15k blocks to import left
122 2012-05-22 13:30:08 <diki> and those will take a while
123 2012-05-22 13:30:20 <diki> at least a few days
124 2012-05-22 13:58:00 <wumpus> imsaguy2: I guess it looks the same as http://bitcoin.sipa.be/bitcoinqt.png ?
125 2012-05-22 14:00:02 <imsaguy2> pretty close
126 2012-05-22 14:00:15 <imsaguy2> if the blue was solid, it'd be ok
127 2012-05-22 14:09:41 <wumpus> imsaguy2: but it is also the windows base theme, not windows xp or vista?
128 2012-05-22 14:10:40 <imsaguy2> if you run windows with the themes service stopped (or you pick it), it runs in a 'classic' theme
129 2012-05-22 14:10:46 <imsaguy2> almost windows 2kish
130 2012-05-22 14:10:56 <imsaguy2> its like the base theme that everything else customizes
131 2012-05-22 14:11:22 <wumpus> right
132 2012-05-22 14:28:42 <nicholas74> question about wallets and private keys: is there essentially a difference, from a technical / protocol point of view, between one wallet with two private keys (and two corresponding public addresses) and two wallets with one private key (and one public address) each?
133 2012-05-22 14:29:01 <sipa> from the point of the network protocol: no
134 2012-05-22 14:29:14 <sipa> wallets are a concept that only exists at the user's end
135 2012-05-22 14:29:46 <nicholas74> ok.. so if I have one wallet with 2 keys & addresses, and someone pays me 10 btc at address1 and 10 btc at address2, my client says I have 20 btc
136 2012-05-22 14:29:57 <sipa> yes
137 2012-05-22 14:30:08 <nicholas74> now if I pay 15 btc, from what private key(s) will this be subtracted?
138 2012-05-22 14:30:27 <kinlo> both
139 2012-05-22 14:30:30 <nicholas74> and does this in fact consist of two payments, rather than one?
140 2012-05-22 14:30:34 <diki> I tried extracting the bitcoin address generator from vanitygen but I still havent made it to work
141 2012-05-22 14:30:39 <sipa> nicholas74: that's the #1 fallacy when trying to understand bitcoin
142 2012-05-22 14:30:41 <diki> so would there be a more cleaner version in C?
143 2012-05-22 14:30:44 <kinlo> no, it will be one payment with 2 inputs and 2 outputs
144 2012-05-22 14:30:56 <sipa> nicholas74: there is no "payment from a key" or "balance per address"
145 2012-05-22 14:30:57 <kinlo> 2 times a 10 btc input, 1 15 btc output, one 5 btc output
146 2012-05-22 14:30:58 <gribble> New news from bitcoinrss: laanwj opened pull request 1379 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/pull/1379>
147 2012-05-22 14:31:08 <kinlo> and the 5 btc output will be to a new address inside your wallet
148 2012-05-22 14:31:15 <sipa> nicholas74: the payment transaction you create will explicitly consume outputs of previous transaction(s)
149 2012-05-22 14:31:16 <kinlo> but the gui hides that, it will just show "15 sent"
150 2012-05-22 14:31:21 <gmaxwell> nicholas74: see the illustration near the top of http://bitcoin.org/bitcoin.pdf
151 2012-05-22 14:31:38 <gmaxwell> Bitcoin is a system of coins (transactions), not balances.
152 2012-05-22 14:31:48 <nicholas74> ok, i understand,
153 2012-05-22 14:32:16 <nicholas74> so at any moment in time, I can deduct my amount of btc (at least what I perceive as "my balance") from all transactions in history
154 2012-05-22 14:32:27 <nicholas74> where "my btc" means "btc for which I own the private keys"
155 2012-05-22 14:32:32 <nicholas74> is that correct?
156 2012-05-22 14:32:35 <kinlo> you will always spent full transactions
157 2012-05-22 14:33:01 <kinlo> and you can ofcourse only spend transactions that haven't been spent beore
158 2012-05-22 14:33:05 <kinlo> before*
159 2012-05-22 14:33:24 <kinlo> but yes, "my btc" means the coins you have the private keys for
160 2012-05-22 14:33:40 <kinlo> and those you have the private keys for are those you can spend
161 2012-05-22 14:34:04 <nicholas74> so if I have privkey1 with 10 btc and privkey2 with 10 btc (right now my client will say "your wallet contains 20 btc"),
162 2012-05-22 14:34:10 <nicholas74> and I pay 15 btc to someone else,
163 2012-05-22 14:34:17 <kinlo> you will create 1 transaction
164 2012-05-22 14:34:37 <kinlo> with 2 inputs (transactions, signed with privkey1 and privkey2)
165 2012-05-22 14:34:39 <kinlo> and 2 outputs
166 2012-05-22 14:34:44 <nicholas74> I end up with.. privkey3 which "has" 5 btc ?
167 2012-05-22 14:34:48 <kinlo> 1 new transaction, for you, with 5 btc
168 2012-05-22 14:34:49 <kinlo> yep
169 2012-05-22 14:34:52 <sipa> nicholas74: you will create a transaction with 2 inputs (the two 10btc coins), and two outputs (5 BTC and 15 BTC, each with a different destination, one the real payment, one change sent back to yourself)
170 2012-05-22 14:34:53 <kinlo> and 1 for the target
171 2012-05-22 14:34:55 <nicholas74> ("has" as in corresponds to 5 btc according to the network)
172 2012-05-22 14:35:10 <nicholas74> right, ok
173 2012-05-22 14:35:23 <kinlo> nicholas74: correct, your wallet contains no coins, the blockchain contains the money if you want to look at it that way
174 2012-05-22 14:35:39 <kinlo> your wallet only contains private keys so you can proove you own them
175 2012-05-22 14:35:39 <nicholas74> yeah ok
176 2012-05-22 14:35:42 <gmaxwell> nicholas74: the better mental model is one of atomic coins that have a size and a owner printed on them.  When you make transactions you send some set of coins off to the network (the forge) to be remade into new coins with new values and owners.
177 2012-05-22 14:36:06 <nicholas74> gmaxwell: awesome explanation, got it
178 2012-05-22 14:36:26 <nicholas74> and whenever new coins get forged (from molten old ones), a new private key is generated for each new coin?
179 2012-05-22 14:36:33 <Diablo-D3> http://www.geek.com/articles/chips/via-launch-a-49-android-pc-20120522/
180 2012-05-22 14:36:53 <gmaxwell> Thats our behavior for the change from transactions it's not mandated by the protocol but it provides greater privacy to do that.
181 2012-05-22 14:37:11 <nicholas74> ok neat
182 2012-05-22 14:37:18 <gmaxwell> You're allowed to reuse addresses.. but then someone snooping can tell those addresses are owned by the same party.
183 2012-05-22 14:37:21 <sipa> kinlo, nicholas74: it's not entirely true that a wallet does not contain coins, by the way; you need the coins to create transactions, and those are kept in the wallet; they get synchronized from the block chain, though
184 2012-05-22 14:37:53 <nicholas74> yeah if I delete my wallet file, but still have my private key, I could regenerate the wallet, right?
185 2012-05-22 14:38:12 <sipa> most parts of it
186 2012-05-22 14:38:31 <sipa> but if you had an unconfirmed transaction in your wallet, there is a risk the network will forget it
187 2012-05-22 14:38:49 <sipa> and the blockchain does not know about accounts, comments, labels, ...
188 2012-05-22 14:39:21 <gavinandresen> sipa gmaxwell : I want to pull https://github.com/bitcoin/bitcoin/pull/1349  now so it gets lots of testing....
189 2012-05-22 14:39:27 <nicholas74> by the way, I was actually under the impression that if I lose my wallet, but still have my private key, I can import the key (at least with MultiBit which I'm using right now) and restore my wallet (except for notes or address books or other personal stuff, but at least the money that it "contained" (well not really but u catch my drift))
190 2012-05-22 14:40:04 <sipa> nicholas74: yes, that's theoretically possible
191 2012-05-22 14:41:00 <nicholas74> ok, but suppose I export my private key(s).. then I make a payment (which gets confirmed) and then I lose my wallet file. I still have the private keys but they're from *before* I made that transaction
192 2012-05-22 14:41:45 <nicholas74> I thought I could still restore my wallet (or at least the btc) with that, but now as I understand the transaction actually generated a new privkey for me (for my remaining btc)
193 2012-05-22 14:41:58 <gmaxwell> gavinandresen: I don't oppose though might we want to address the compressed key performance before release? if we're concerned about DOS attacks the attacker will of course pick the most expensive form to send.
194 2012-05-22 14:42:23 <gmaxwell> nicholas74: the reference client precomputes keys 100 ahead of your actual use.
195 2012-05-22 14:42:35 <gmaxwell> nicholas74: so long as you didn't use up 100 keys between backups you're safe.
196 2012-05-22 14:42:40 <sipa> gavinandresen: i wonder if there isn't another solution: keep a set of script+signature-verified transaction hashes in memory
197 2012-05-22 14:42:52 <sipa> gavinandresen: so move it to the tx layer instead of the key layer
198 2012-05-22 14:43:04 <nicholas74> uhm.. these 100 precomputed privkeys are stored in my wallet?
199 2012-05-22 14:43:08 <gavinandresen> gmaxwell: I don't think there's an issue with compressed key performance, the only reason it looks like there is is because with the sig cache verification is so much faster....
200 2012-05-22 14:43:08 <sipa> nicholas74: yes
201 2012-05-22 14:43:12 <gmaxwell> sipa: well, I could then flood you with script that force lots of checksigs.. but which are pointlessly different.
202 2012-05-22 14:43:17 <gmaxwell> sipa: thus defatin the cache.
203 2012-05-22 14:43:20 <nicholas74> but if I export my key, it only stores one key (well at least MultiBit does)
204 2012-05-22 14:43:54 <gmaxwell> gavinandresen: yea, but  why isn't the compressed key handling behind the cache at least?
205 2012-05-22 14:43:54 <nicholas74> so what you said with being safe as long as I didn't make 100 transactions since my last backup, only holds for wallet files, not for private keys which I exported manually, right?
206 2012-05-22 14:44:04 <gmaxwell> nicholas74: correct.
207 2012-05-22 14:44:10 <nicholas74> ok
208 2012-05-22 14:44:14 <nicholas74> thx
209 2012-05-22 14:44:19 <sipa> gmaxwell: the decompression is inside openssl
210 2012-05-22 14:44:21 <gmaxwell> nicholas74: which is why exporting private keys is not promoted as a backup method! :)
211 2012-05-22 14:44:53 <nicholas74> hehe understood :)
212 2012-05-22 14:44:54 <wumpus> you'd need to export *all* private keys in the wallet
213 2012-05-22 14:44:55 <gmaxwell> sipa: I know it is .. actually I thought it was happening in the validation so it would be cached too. :-/
214 2012-05-22 14:44:56 <nicholas74> good thing I found out before actually using it :)
215 2012-05-22 14:44:57 <sipa> wait, this doesn't make sense
216 2012-05-22 14:45:21 <sipa> gavinandresen: your cache stored pubkey/sig/msghash, no?
217 2012-05-22 14:45:25 <gmaxwell> (I'm unhappy that this means that it's more parts that I don't understand.)
218 2012-05-22 14:45:53 <sipa> gavinandresen: i don't see where key decompression is needed, for validation?
219 2012-05-22 14:46:22 <gavinandresen> cache stores <uint256 sighash, std::vector sig, std::vector pubkey>
220 2012-05-22 14:46:28 <sipa> gavinandresen: oh, i see
221 2012-05-22 14:46:32 <gavinandresen> the decompression happens getting the std::vector pubkey
222 2012-05-22 14:46:45 <sipa> gavinandresen: you construct the CKey, and then inside CKey you request GetPubKey() again
223 2012-05-22 14:46:56 <gavinandresen> If I could store an EC_KEY* in the cache that'd be ok....
224 2012-05-22 14:46:58 <sipa> i think you can avoid constructing the CKey
225 2012-05-22 14:47:02 <diki> nevermind I got it to work
226 2012-05-22 14:47:08 <nicholas74> however.. I noticed the electrum client, it says it can generate or recover a wallet from a seed (which seems to be one single key: http://ecdsa.org/electrum/seed.html )
227 2012-05-22 14:47:09 <sipa> as you construct the CKey from the pubkey
228 2012-05-22 14:47:49 <sipa> gavinandresen: another way would be to change CKey to delay the conversion from pubkey to EC_KEY
229 2012-05-22 14:48:09 <sipa> i have to go; i'll be back in a few hours
230 2012-05-22 14:48:40 <sipa> nicholas74: that's a so-called deterministic wallet, where the used keys are in fact not fully random
231 2012-05-22 14:49:49 <olp> bitcoin-qt doesn't do deterministic wallets anymore?
232 2012-05-22 14:49:53 <wumpus> right, by using a deterministic random generator you can re-generate all the keys from one master key
233 2012-05-22 14:49:56 <wumpus> it never did
234 2012-05-22 14:50:05 <nicholas74> this construction with paying 15 btc (from a total of 20 which consisted of 2x10), leaving 5, resulting in a new single private key which corresponds to my remaining 5 btc.. exactly who or what does that? is that purely client behavior, or is this by design in the protocol?
235 2012-05-22 14:50:24 <olp> cool, then that's good
236 2012-05-22 14:50:30 <gmaxwell> 09:36 < gmaxwell> Thats our behavior for the change from transactions it's not mandated by the protocol but it provides greater privacy to do that.
237 2012-05-22 14:50:33 <gmaxwell> 09:37 < gmaxwell> You're allowed to reuse addresses.. but then someone snooping can tell those addresses are owned by the same party.
238 2012-05-22 14:51:16 <nicholas74> ok
239 2012-05-22 14:51:32 <gmaxwell> by the protocol you must send your change back to an address you own, unless you want to give it away to the miner... but it doesn't have to be a new address, thats just a really good idea.
240 2012-05-22 14:51:49 <nicholas74> aight, i see
241 2012-05-22 14:52:03 <phantomcircuit> gmaxwell, depends on what your goal is
242 2012-05-22 14:52:04 <nicholas74> but with this ever renewing private key whenever I do a transaction, wouldn't that result in a new public address as well?
243 2012-05-22 14:53:09 <gmaxwell> nicholas74: ... No. You don't just have one address. You should be using a new address with every transaction.
244 2012-05-22 14:53:13 <nicholas74> so if I have privkey1 with it's corresponding public address1, which I published on my charity website, then I make a payment (so I now have privkey2 (+public address2) with less btc) but then someone else makes a payment to address1)
245 2012-05-22 14:53:29 <gmaxwell> phantomcircuit: it's pretty much always good.  if you're concerned e.g. about the backup implications use determinsticaly generated change addresses.
246 2012-05-22 14:53:43 <gmaxwell> phantomcircuit: if you want to prove to someone you own that change still, sign a message with it.
247 2012-05-22 14:53:46 <nicholas74> oh ok.. so "publishing" my address to receive btc is not good behavior?
248 2012-05-22 14:54:00 <phantomcircuit> gmaxwell, i understand that just saying sometimes there are reasons not to
249 2012-05-22 14:54:05 <gmaxwell> nicholas74: it's good to avoid doing that if you can, you can't always.
250 2012-05-22 14:54:19 <wumpus> nicholas74: that's perfectly valid, the client wil retain all the older keys and you can still receive btc on them
251 2012-05-22 14:54:30 <nicholas74> aha ok
252 2012-05-22 14:54:40 <nicholas74> ooh right that makes sense now
253 2012-05-22 14:54:59 <nicholas74> so the wallet actually will contain more than 100 private keys even,
254 2012-05-22 14:55:07 <sipa> gavinandresen: in script.cpp, CheckSig() takes a public key and changes it to a CKey to run VerifySignature on; if you'd move the caching mechanism there, you could avoid the ckey construction (which is slow for compressed one)
255 2012-05-22 14:55:35 <wumpus> yes
256 2012-05-22 14:55:36 <nicholas74> or well, it will contain as many private keys as I have made transactions (since it generates a new one upon every payment I make) + the 100 in advance ?
257 2012-05-22 14:55:37 <gmaxwell> nicholas74: sure it can conain many and usually does.. it precomputes 100 _ahead_ and stores infinite behind.
258 2012-05-22 14:55:46 <nicholas74> ok
259 2012-05-22 14:55:47 <nicholas74> clear
260 2012-05-22 14:55:50 <nicholas74> thank you sir :)
261 2012-05-22 14:55:53 <gmaxwell> nicholas74: not every transaction has change, but thats a reasonable assumption.
262 2012-05-22 14:56:05 <gmaxwell> e.g. if I pay you 10 btc using a 10 btc input.. no change.
263 2012-05-22 14:56:14 <nicholas74> right
264 2012-05-22 14:56:56 <sipa> gavinandresen: i think that's the easiest solution; if you prefer keeping it at the key level, you could store the pubkey as a field inside CKey, and don't construct the EC_KEY object until necessary
265 2012-05-22 14:57:09 <nicholas74> I was wondering if privacy was an issue with bitcoin, cause essentially anyone can track any payment in the system
266 2012-05-22 14:57:43 <gmaxwell> see section 10 of that pdf I linked you to eariler.
267 2012-05-22 14:57:54 <gavinandresen> sipa: did you see my comments in the commit?  if we want to get serious about speeding up transaction verification then all of this micro-optimization is probably no the right way to do it
268 2012-05-22 14:58:02 <nicholas74> so if someone knows *one* address that I have (or that I ever used in the past), they can follow any bitcoins that were paid (or change leftovers) from that address to others, and potentially deduct 'how much money I have'
269 2012-05-22 14:58:03 <gmaxwell> (actually you really should read the whole thing, it's only about 8 pages long)
270 2012-05-22 14:58:05 <sipa> gmaxwell: i'm not convinced by your example earlier, by the way, if an attackers sends tons of different signatures for you to check, how is that different from the caching mechanism implemented by gavin now?
271 2012-05-22 14:58:26 <gmaxwell> sipa: takes compuation to compute the signatures.
272 2012-05-22 14:58:30 <nicholas74> gmaxwell: ok, reading..
273 2012-05-22 14:58:37 <gmaxwell> (in particular valid ones)
274 2012-05-22 14:58:48 <sipa> gmaxwell: not following why that is different still
275 2012-05-22 14:59:17 <gavinandresen> if an attacker has to expend more resources to mount the attack than it takes to process the attack then they lose
276 2012-05-22 14:59:52 <gavinandresen> (DDos attacks where the attacker is stealing somebody else's CPU/bandwidth/whatever are different, of course)
277 2012-05-22 15:00:02 <sipa> gavinandresen: yes, i saw it, but i think the cost of CKey construction may be significant
278 2012-05-22 15:00:16 <gmaxwell> nicholas74: they know one address. but they don't know about others thats why everyone should always use new addresses for every transaction.  They also can't always tell which of multiple outputs was change.
279 2012-05-22 15:01:19 <gavinandresen> sipa: I'll benchmark moving the cache to CheckSig after lunch, that's an easy change
280 2012-05-22 15:01:58 <sipa> gavinandresen: i did benchmark your cache by the way, it slowed initial syncup down slightly (maybe statistical variance), but after that, around 40% of incoming signature checks resulted in cache hits
281 2012-05-22 15:02:52 <gavinandresen> sipa: nice, thanks.
282 2012-05-22 15:09:33 <luke-jr> any ideas for how I might test #1240 more? :p
283 2012-05-22 15:10:11 <gmaxwell> luke-jr: write unit tests that form a memory pool with varrious corner cases and check that the resulting blocks are valid.
284 2012-05-22 15:10:25 <gmaxwell> (E.g. memory pools with transactions that would push blocks over the protocol imposed limits)
285 2012-05-22 15:17:48 <luke-jr> http://pastebin.com/qSLwN7eR <-- any test case I'm missing?
286 2012-05-22 15:18:03 <luke-jr> or any that shouldn't be possible?
287 2012-05-22 15:21:04 <gmaxwell> In particular on your > limit test you should be testing at lest the limit exactly and one over at least if possible. Sigops limit should also test the case where the last txn is a multisig txn.
288 2012-05-22 15:21:36 <gmaxwell> its up to your taste testing the 'impossible' cases like duplicates in the mempool. More tests are generally better.
289 2012-05-22 15:22:00 <gmaxwell> (even if the impossible case really is impossible those tests might find some other bug someday)
290 2012-05-22 15:28:17 <guruvan> is there a correct way to use more than one wallet with a single copy of the rest of the datadir?    (so I don't need multiple blockchain copies)
291 2012-05-22 15:33:53 <jgarzik> guruvan: shut down, swap wallets, restart with -rescan
292 2012-05-22 15:40:19 <guruvan> hmm. thanks. I will try this (exactly) again...though...I thought that's what I'd done several times. - I started as I got database errors on upgrade to 0.6.2, backed up, wiped the datadir, got fresh blocchain, and swapped wallets.  still dbase errors, swap back (to the fresh made, unused wallet) and still errors.  - I will try again.
293 2012-05-22 15:43:24 <denisx> is it intentional that the synchronizing progressbar has the whole range of all blocks und not only the range from what I have to the actual block?
294 2012-05-22 15:49:08 <gavinandresen> denisx: yes, that is intentional.
295 2012-05-22 15:49:37 <gavinandresen> denisx: ... and you'll make wumpus grumpy if you start arguing it should be some other way
296 2012-05-22 15:49:55 <denisx> but someone only loading blocks for seven missing days can't see a progress because it is too small
297 2012-05-22 15:53:08 <denisx> is wumpus the only one who wants it that way?
298 2012-05-22 15:53:47 <gavinandresen> denisx: it was discussed to death, and consensus is that's the best way to do it.  You'll make us all grumpy if you want to discuss it to death some more
299 2012-05-22 15:53:54 <jgarzik> hehehe
300 2012-05-22 15:54:52 <denisx> another case of programmers designing a UI the wrong way...
301 2012-05-22 15:54:54 <denisx> ;(
302 2012-05-22 15:55:02 <denisx> I rest my case
303 2012-05-22 15:57:29 <jgarzik> I <heart> gitk
304 2012-05-22 16:04:50 <gavinandresen> sipa: when you're back: https://github.com/bitcoin/bitcoin/pull/1380  moving the cache is a very good idea
305 2012-05-22 16:04:52 <wumpus> no matter what way it is, it's always exactly the wrong way to some people, and of course that's the most important issue in the world
306 2012-05-22 16:05:45 <phantomcircuit> im sweating so much that there are drops of sweat on my desk
307 2012-05-22 16:05:50 <phantomcircuit> it's not even hot
308 2012-05-22 16:07:30 <jgarzik> gavinandresen: can you encapsulate when you move?  it would take two seconds
309 2012-05-22 16:07:46 <luke-jr> gmaxwell: both before and after #1240, bitcoind refuses to go all the way up to the limit&
310 2012-05-22 16:08:07 <gavinandresen> jgarzik: fine
311 2012-05-22 16:08:09 <gribble> New news from bitcoinrss: gavinandresen opened pull request 1380 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/pull/1380>
312 2012-05-22 16:09:04 <luke-jr> denisx: it was the other way for a few versions, and we got a ton of complaints "Bitcoin-Qt starts over when I restart during download!"
313 2012-05-22 16:09:25 <wumpus> I wonder what 'the right way' of designing an UI would be, instantly chainging everything people request, then waiting for the next load of people to complain, then flipping it again, and all over again
314 2012-05-22 16:09:38 <jgarzik> gavinandresen: you introduced a new warning,
315 2012-05-22 16:09:40 <jgarzik> key.cpp: In function void SetValidSigCache(uint256, const std::vector<unsigned char>&, const std::vector<unsigned char>&):
316 2012-05-22 16:10:22 <wumpus> sigh...
317 2012-05-22 16:10:53 <wumpus> yeah, let's solve an unsolvable dispute... by doing it yet another way!
318 2012-05-22 16:11:00 <diki> jgarzik:well all make mistakes.
319 2012-05-22 16:11:16 <wumpus> or simply lobbing a nuke at it
320 2012-05-22 16:11:20 <denisx> luke-jr: that point I understand
321 2012-05-22 16:11:57 <luke-jr> wumpus: well, if there's no progress bar, there's no argument over how it looks/values
322 2012-05-22 16:12:10 <wumpus> true, true
323 2012-05-22 16:17:18 <gavinandresen> jgarzik: encapsulated and fixed the signed/unsigned warning and pushed.  That took 8 minutes, not 2....
324 2012-05-22 16:20:37 <jgarzik> gavinandresen: appreciated nonetheless ;)
325 2012-05-22 16:21:50 <gavinandresen> I've got a feeling I might regret using boost::tuple, though.
326 2012-05-22 16:23:17 <luke-jr> gavinandresen: should CreateNewBlock tests be in miner_tests, or is that just for the builtin CPU miner?
327 2012-05-22 16:23:55 <gavinandresen> miner_tests seems like a good place
328 2012-05-22 16:23:58 <luke-jr> k
329 2012-05-22 16:25:52 <jgarzik> hum
330 2012-05-22 16:26:18 <jgarzik> I wonder what is C++'s equivalent of __FUNCTION__ / __func__
331 2012-05-22 16:27:30 <luke-jr> jgarzik: __func__ was introduced in C++11 IIRC
332 2012-05-22 16:28:20 <gribble> New news from bitcoinrss: jgarzik opened pull request 1381 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/pull/1381>
333 2012-05-22 16:28:35 <jgarzik> ok, C++ wants __PRETTY_FUNCTION__ to return a string of both class name and method name
334 2012-05-22 16:28:40 <jgarzik> __func__ just returns method name
335 2012-05-22 16:46:11 <jgarzik> received a small burst of 20 orphans
336 2012-05-22 16:46:18 <jgarzik> received a small burst of 20 orphan tx's
337 2012-05-22 16:51:18 <luke-jr> gavinandresen: is there any existing way to handle use of CTxDB within test_bitcoin? :/
338 2012-05-22 16:52:38 <gavinandresen> luke-jr: I don't think so. support for unit testing in CDB is sorely needed
339 2012-05-22 16:59:29 <jgarzik> gavinandresen: your commit 77b99cf7 caused a doubling of orphan messages:
340 2012-05-22 16:59:42 <jgarzik> 05/22/12 18:59:17 storing orphan tx cf5fb7678f (mapsz 40)
341 2012-05-22 17:00:16 <gavinandresen> jgarzik: oops
342 2012-05-22 17:01:18 <jgarzik> "git blame src/main.cpp" to the rescue ;-)
343 2012-05-22 17:02:43 <gavinandresen> gotta pick up kids... back later.  Removing the stored orphan tx version of the message looks like the right thing to do
344 2012-05-22 17:03:57 <jgarzik> gavinandresen: I would remove the other one, "storing"
345 2012-05-22 17:04:15 <jgarzik> gavinandresen: resulting in AddOrphanTx always printing something useful, regardless of circumstance
346 2012-05-22 17:04:35 <gavinandresen> jgarzik: ok
347 2012-05-22 17:04:51 <gavinandresen> (need a solution for overly verbose unit tests, though....)
348 2012-05-22 17:05:10 <jgarzik> gavinandresen: I'll take care of this nit, since you're headed out the door
349 2012-05-22 17:27:07 <jgarzik> luke-jr: I'd say we want to backport #1381
350 2012-05-22 17:27:19 <jgarzik> well "we" meaning "you" ;-)
351 2012-05-22 17:28:53 <luke-jr> jgarzik: I'll note that for when it gets merged
352 2012-05-22 17:32:01 <jgarzik> darn you, valid tx's
353 2012-05-22 17:32:10 <jgarzik> where is that crazy person sending orphans, when you need one?
354 2012-05-22 17:49:07 <luke-jr> http://codepad.org/O9fKH3cq <-- does this look reasonable, to emulate the entire db environment in memory?
355 2012-05-22 17:56:21 <jgarzik> luke-jr: you probably want to base on top of CDBEnv
356 2012-05-22 17:56:24 <gavinandresen> luke-jr: "fMockDb" might be better than fDummyDb. I can't comment on making BDB store everything in memory, never done that (but seems to me to be the right approach)
357 2012-05-22 17:59:28 <jgarzik> gavinandresen sipa gmaxwell: trolling for ACKs on #1293 ...  if it makes ACK'ing easier, I can postpone these two, which require additional brain cells & review time: "remove spurious TxnAbort()" and "remove nested transactions"
358 2012-05-22 17:59:35 <luke-jr> jgarzik: there is no such thing? O.o
359 2012-05-22 17:59:41 <jgarzik> absent those, it should be a straight encapsulation
360 2012-05-22 17:59:55 <jgarzik> luke-jr: see #1293
361 2012-05-22 18:04:03 <luke-jr> jgarzik: that does sound like it'd simplify things
362 2012-05-22 18:48:12 <sipa> jgarzik: all commits in #1293 look good to me
363 2012-05-22 18:48:25 <sipa> i'll do some tests
364 2012-05-22 19:01:50 <gribble> New news from bitcoinrss: rebroad opened pull request 1382 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/pull/1382>
365 2012-05-22 19:52:16 <phoenixisland> after upgrading to bitcoin 6.2 my computer crashes near the end of the sychronization process. can anyone help?
366 2012-05-22 19:53:41 <jgarzik> phoenixisland: your entire computer crashes?
367 2012-05-22 19:55:32 <phoenixisland> yes, says boot disc error, install boot disc and hit enter
368 2012-05-22 19:56:54 <jgarzik> phoenixisland: it sounds like you have a virus unrelated to bitcoin, or hardware problems
369 2012-05-22 19:57:19 <phoenixisland> it only crashes when i run bitcoin since upgrading
370 2012-05-22 20:00:11 <jgarzik> phoenixisland: recommend doing something like this, to check your hard drive for hardware problems: http://windows.microsoft.com/en-us/windows-vista/check-your-hard-disk-for-errors
371 2012-05-22 20:00:31 <jgarzik> phoenixisland: also boot fresh, and update a virus scan
372 2012-05-22 20:00:40 <jgarzik> er, update & run virus scan
373 2012-05-22 20:00:55 <phoenixisland> when i restart bitcoin after restarting computer then bitcoin can' find the blkindex.dat at which point i delete everything in appdata folder/bitcoin except wallet.dat and it starts
374 2012-05-22 20:01:06 <jgarzik> phoenixisland: did you download bitcoin from SourceForge.net directly, or some other site?
375 2012-05-22 20:01:13 <phoenixisland> directly
376 2012-05-22 20:01:47 <phoenixisland> ok, will try the hardisc check
377 2012-05-22 20:16:21 <agath> Hello! I am lurking since several months. I was wondering one thing, I don't know if this is the correct channel...
378 2012-05-22 20:16:35 <agath> wouldn't it be useful to allow "payment request" transactions?
379 2012-05-22 20:17:07 <sipa> agath: very useful
380 2012-05-22 20:17:22 <agath> if I want to sell a service, and I expect a payment from a customer, let's say 1.8373842 BTC, it's impractical for the user to send the exact amount with the current functionality...
381 2012-05-22 20:17:36 <sipa> but there's no need to have them be like bitcoin transaction
382 2012-05-22 20:18:04 <agath> I think it would be better to allow the customer to send their bitcoin address where they want the payment request, and when the client receives it, it asks the authorization for sending the exact amount.
383 2012-05-22 20:18:08 <sipa> instead, they could be encoded in a URL, or in a payment-request file, or negotiated through some third-party protocol
384 2012-05-22 20:18:26 <sipa> a bitcoin address is not a transport endpoint
385 2012-05-22 20:18:31 <sipa> you cannot send something to it
386 2012-05-22 20:18:35 <agath> yes it could be, but it would allow for malicious URLs if not handled correctly..
387 2012-05-22 20:18:41 <gmaxwell> Then handle it correctly.
388 2012-05-22 20:18:59 <gmaxwell> (moreover, anything not a URL would have even more malicious potential)
389 2012-05-22 20:19:37 <gmaxwell> E.g. I could watch for payment requests and send redundant ones and people would pay the wrong one.
390 2012-05-22 20:19:53 <agath> hmm yes, true..
391 2012-05-22 20:21:57 <luke-jr> "it's impractical for the user to send the exact amount with the current functionality&" <-- why?
392 2012-05-22 20:22:44 <agath> because what would it happen if the customer inserts the wrong amount? you should manage a "balance" and send back the exceeding credit or generating another request for the missing amount...
393 2012-05-22 20:22:57 <agath> if you feed him the correct amount, it's less prone to errors
394 2012-05-22 20:23:09 <luke-jr> OK, so use a URI
395 2012-05-22 20:23:12 <sipa> you can put the amount in a bitcoin: URI
396 2012-05-22 20:29:21 <agath> ok that could solve the problem!
397 2012-05-22 20:37:27 <BlueMatt> speaking of URIs, whats up with win32 uris on master?
398 2012-05-22 20:37:41 <sipa> no idea, not been following that mess for a while
399 2012-05-22 20:51:23 <sipa> RedEmerald: now i realize there's a section in the fallback node list on the wiki that explains how to connect to onion addresses via tor address mapping
400 2012-05-22 20:51:41 <RedEmerald> sipa: yup yup. i wrote some of it
401 2012-05-22 20:52:14 <sipa> RedEmerald: anyway, i'm working on built-in tor hidden support in bitcoin, so you can say -connect=<onionaddr>, and have the client also exchange these addresses via peer exchange
402 2012-05-22 20:52:26 <RedEmerald> yeah. I'm running next-test with it
403 2012-05-22 20:52:37 <sipa> ok, nice :)
404 2012-05-22 20:52:45 <gmaxwell> RedEmerald: do you have any feedback or bugreports?
405 2012-05-22 20:53:05 <RedEmerald> i'm going to test sending/receiving transactions from it after work
406 2012-05-22 20:53:10 <RedEmerald> but its been working fine
407 2012-05-22 20:53:32 <RedEmerald> 24 connections right now
408 2012-05-22 20:53:37 <RedEmerald> I'm running with -discover, too
409 2012-05-22 20:53:45 <gmaxwell> gavinandresen: make any progress on the testnet reboot?  We've got a bunch of stuff that could use some testnet testing love.
410 2012-05-22 20:54:26 <RedEmerald> p2hwc26zdsrqxiix.onion btw
411 2012-05-22 20:54:29 <RedEmerald> if you need one for testing
412 2012-05-22 21:07:58 <luke-jr> jgarzik: #1381 backport complete fwiw
413 2012-05-22 21:16:07 <luke-jr> 8.8% listening nodes secure against CVE-2012-2459
414 2012-05-22 21:20:14 <dinox> gavinandresen: ping
415 2012-05-22 21:23:22 <RedEmerald> sipa: have you done any testing with transparent for proxies? i usually use torbox, but that means i won't be setting -proxy
416 2012-05-22 21:23:47 <sipa> how does torbox work?
417 2012-05-22 21:24:30 <sipa> just intercept outgoing ip connections, and route them over tor?
418 2012-05-22 21:24:46 <RedEmerald> you have a gateway that runs Tor, and a workstation that uses that Tor
419 2012-05-22 21:25:08 <RedEmerald> that way if a program on the workstation is exploited or leaks it's IP, it doesn't get the real external IP
420 2012-05-22 21:25:29 <RedEmerald> https://trac.torproject.org/projects/tor/wiki/doc/TorBOX
421 2012-05-22 21:26:12 <gmaxwell> RedEmerald: it's really much better to use proxy with bitcoin. We'll never be able to connect to onion hosts via a transparent proxy.
422 2012-05-22 21:26:26 <RedEmerald> thats what i figured
423 2012-05-22 21:28:21 <sipa> since torbox seems to intercept dns resolving as well, i wonder if it could in fact connect to hidden services
424 2012-05-22 21:28:27 <sipa> no idea how it would work, though
425 2012-05-22 21:28:38 <RedEmerald> i'll test and get back to you :)
426 2012-05-22 21:29:04 <sipa> i put my permanent node's onion address on the hidden service page, by the way
427 2012-05-22 21:29:12 <luke-jr> sipa: AAAA?
428 2012-05-22 21:29:38 <sipa> luke-jr: onioncat you mean?
429 2012-05-22 21:29:51 <sipa> could be
430 2012-05-22 21:29:57 <luke-jr> sipa: yes, TorBOX can/should resolve onion addresses to AAAA records
431 2012-05-22 21:29:59 <luke-jr> and route those
432 2012-05-22 21:35:30 <bonks> is there a simple way to retrieve bitcoind's getinfo result from python?
433 2012-05-22 21:36:29 <sipa> bonks: use any JSON-RPC module
434 2012-05-22 21:36:38 <sipa> i'm sure there are many
435 2012-05-22 21:37:31 <bonks> ok, im a python noob
436 2012-05-22 21:39:31 <Diablo-D3> hey bonks
437 2012-05-22 21:39:45 <bonks> hi
438 2012-05-22 21:41:48 <luke-jr> sipa: only one reasonable, actually: jgarzik's bitcoinrpc
439 2012-05-22 21:48:38 <BlueMatt> jgarzik: have you tried splitting txindex into eg txindex[0-F].dat by first bits of tx (or does bdb have an option to do that for you), seems like that could give a bit of a perf bump over just a btree?
440 2012-05-22 21:49:59 <jgarzik> BlueMatt: with the bits in a hash effectively random, I don't see how that would help performance.  I'm probably missing something?  Have not tested that configuration, no.
441 2012-05-22 21:50:19 <bonks> luke-jr: how do I install jgarzik's bitcoinrpc? no instructions
442 2012-05-22 21:50:19 <jgarzik> hashes don't have locality of keys
443 2012-05-22 21:52:05 <BlueMatt> jgarzik: effectively converting the first split in the btree to a 16-tree would cut down on overall btree size quite a bit, when we are searching that tree so rapidly, it could have an effect
444 2012-05-22 21:52:07 <BlueMatt> is my though
445 2012-05-22 21:52:08 <BlueMatt> t
446 2012-05-22 21:52:49 <BlueMatt> s/rapidly/constantly/
447 2012-05-22 21:53:42 <RedEmerald> bah. all the packages are downloading over Tor. going to be a few minutes before i can test bitcoind with torbox
448 2012-05-22 21:58:30 <jgarzik> BlueMatt: how many leaves does BDB pack into a page right now?
449 2012-05-22 21:59:47 <jgarzik> the first few levels of btree are always in cache anyways
450 2012-05-22 22:00:23 <jgarzik> sigh...  would love kyotocabinet's transactional-hash-table-on-disk
451 2012-05-22 22:00:24 <sipa> BlueMatt: so you're basically changing one level of in-file branching to one level of filesystem branching
452 2012-05-22 22:00:30 <BlueMatt> for most of us, the whole thing is in cache...
453 2012-05-22 22:00:37 <jgarzik> sipa: yep
454 2012-05-22 22:00:47 <sipa> i doubt that would help
455 2012-05-22 22:00:49 <BlueMatt> yea
456 2012-05-22 22:00:49 <luke-jr> bonks: I do: emerge bitcoinrpc
457 2012-05-22 22:01:01 <sipa> jgarzik: by all means, try it ;)
458 2012-05-22 22:01:13 <BlueMatt> sipa: probably not, I just wondered if it had been tried
459 2012-05-22 22:01:24 <jgarzik> sipa: a little bit more abstraction and I'm there :)
460 2012-05-22 22:01:36 <BlueMatt> but then there have been more surprising results that have big performance impacts
461 2012-05-22 22:01:43 <jgarzik> sipa: all this database-management code is scattered around main.cpp and wallet*.cpp
462 2012-05-22 22:01:46 <jgarzik> CloseDb etc.
463 2012-05-22 22:01:53 <sipa> i know
464 2012-05-22 22:02:23 <jgarzik> BlueMatt: you're welcome to test it... should be easy with my #blockindex branch
465 2012-05-22 22:02:32 <sipa> but if that kyotocabinet system only uses a single file, there's no need for a cdbenv
466 2012-05-22 22:02:52 <sipa> you could just create a cdb substitute that uses kyoto
467 2012-05-22 22:03:03 <jgarzik> sipa: single file per table IIUC, so a bit more limited than BDB
468 2012-05-22 22:03:30 <jgarzik> sipa: yeah, since we only use one database per file, CDB replacement is feasible
469 2012-05-22 22:03:36 <sipa> good enough, we've been using a single table for everything anyway
470 2012-05-22 22:03:40 <jgarzik> yep
471 2012-05-22 22:03:43 <sipa> *ducks*
472 2012-05-22 22:03:44 <BlueMatt> jgarzik: yea, I might sometime, I just wondered if you had, and hoped it would take you 10 lines to code + start off a benchmark instead of me trying to read up on #blockindex
473 2012-05-22 22:04:03 <jgarzik> sipa: s/table/file/ ITYM
474 2012-05-22 22:04:28 <jgarzik> both #master and #blockindex use a single table per file
475 2012-05-22 22:05:06 <jgarzik> which works with KC
476 2012-05-22 22:07:47 <jgarzik> BlueMatt: the main technical hurdle with that idea remains the same with either #master or #blockindex really:  CDB assumes a single filename, and your scheme would need to change that.
477 2012-05-22 22:08:14 <jgarzik> code past that hurdle, and I could easily test with #blockindex from a patch based on #master
478 2012-05-22 22:08:31 <BlueMatt> jgarzik: ah, I thought #blockindex had abstracted further already, nvm
479 2012-05-22 22:09:04 <jgarzik> BlueMatt: #blockindex does CTxDB -> { CTxDB, CBlockIdxDB, CMetaDB }, nothing more
480 2012-05-22 22:09:12 <BlueMatt> oh, ok
481 2012-05-22 22:09:41 <jgarzik> #blockindex relies heavily on CDB, and only changes one minor CDB detail (setting DB_HASH rather than DB_BTREE)
482 2012-05-22 22:10:04 <BlueMatt> did DB_HASH have much of a perf impact?
483 2012-05-22 22:10:25 <jgarzik> BlueMatt: right now we smash together different datasets into a single huge key/value index, much like a filesystem.
484 2012-05-22 22:10:33 <jgarzik> BlueMatt: yeah, it got slower and bigger ;p
485 2012-05-22 22:10:38 <BlueMatt> oh...
486 2012-05-22 22:11:01 <jgarzik> BlueMatt: but it is untuned.  bdb really wants you to tune the hash fill factor (how heavily you will hash table leaf pages)
487 2012-05-22 22:11:15 <jgarzik> *how heavily you will pack
488 2012-05-22 22:11:24 <jgarzik> anyway, baby bedtime
489 2012-05-22 22:11:45 <sipa> i suspect node startup time to be a lot faster with blockindex
490 2012-05-22 22:12:12 <sipa> as only blkhash.dat is necessary at startup, and is small
491 2012-05-22 22:12:14 <BlueMatt> yea, I wondered, seems like O(1) theoretical should give us a bump, but if its not tuned right, hash tables suck
492 2012-05-22 22:13:11 <sipa> btree's disadvantage is higher index lookup frequency
493 2012-05-22 22:13:36 <sipa> but if the entire index fits in memory, that doesn't matter
494 2012-05-22 22:14:06 <BlueMatt> how is the in-memory cache for DB_HASH done?
495 2012-05-22 22:14:14 <BlueMatt> or does it just cache part of the hash table?
496 2012-05-22 22:14:14 <sipa> hmm?
497 2012-05-22 22:14:25 <sipa> i suppose
498 2012-05-22 22:14:46 <BlueMatt> or, I guess my question is, is the in-memory cache the same between btree and hash?
499 2012-05-22 22:19:32 <RedEmerald> woohoo
500 2012-05-22 22:19:49 <RedEmerald> bitcoind -connect=p2hwc26zdsrqxiix.onion -proxy=192.168.0.1:9100
501 2012-05-22 22:19:54 <RedEmerald> downloading blocks now :)
502 2012-05-22 22:42:26 <jgarzik> back
503 2012-05-22 22:42:45 <jgarzik> sipa: we touch txindex during verification, I thought
504 2012-05-22 22:42:54 <jgarzik> sipa: in addition to LoadBlockIndex at startup
505 2012-05-22 22:44:15 <jgarzik> BlueMatt: check out the two pastebin links at the end of https://github.com/bitcoin/bitcoin/pull/1303
506 2012-05-22 22:44:28 <luke-jr> jgarzik: you merged without fixing the bug breaking Bitcoin-Qt? -.-
507 2012-05-22 22:44:30 <BlueMatt> jgarzik: ahhh
508 2012-05-22 22:44:31 <jgarzik> BlueMatt: that shows db_stat output for #blockindex (DB_HASH) and #master
509 2012-05-22 22:44:47 <jgarzik> BlueMatt: including tree/etc. details
510 2012-05-22 22:45:32 <jgarzik> luke-jr: please be more specific
511 2012-05-22 22:45:42 <jgarzik> luke-jr: I break all kinds of shit all the time!
512 2012-05-22 22:45:53 <luke-jr> jgarzik: fDetachDb not defined etc
513 2012-05-22 22:46:03 <jgarzik> hrm
514 2012-05-22 22:46:04 <luke-jr> jgarzik: I recall it being fixed, but apparently didn't make the merge
515 2012-05-22 22:46:34 <jgarzik> luke-jr: ah, the fix got trapped inside #blockindex
516 2012-05-22 22:48:19 <luke-jr> aha
517 2012-05-22 22:52:53 <diki> le app > 250mb memory allocated
518 2012-05-22 22:52:53 <luke-jr> "blocks" : 44,
519 2012-05-22 22:52:58 <diki> my app that is...
520 2012-05-22 22:56:15 <sipa> jgarzik: hmm
521 2012-05-22 23:09:52 <jgarzik> looks like kyotocabinet does not do cross-file transactions, IIUC
522 2012-05-22 23:10:13 <jgarzik> could still try it with blkindex.dat
523 2012-05-22 23:12:38 <jgarzik> anyway, I'll finish looking at #blockindex before kyotocabinet...  those CTxIndex entries are all tiny, so I do not understand why the index is so huge, or why there is so many overflow pages, or why there is so much free space shown in db_stat
524 2012-05-22 23:12:59 <jgarzik> there is clearly one or more jgarzik problems in there, that we should not blame on BDB
525 2012-05-22 23:25:05 <jgarzik> that's odd
526 2012-05-22 23:25:09 <jgarzik> memset(datKey.get_data(), 0, datKey.get_size());
527 2012-05-22 23:25:24 <jgarzik> checking for NULL after a potential memory reference via memset(3)
528 2012-05-22 23:25:45 <jgarzik> one hopes datKey.get_size() always returns zero for that case, I suppose
529 2012-05-22 23:26:54 <luke-jr> jgarzik: dayKey != datValue ?
530 2012-05-22 23:27:51 <jgarzik> luke-jr: yeah, duh :)
531 2012-05-22 23:28:12 <jgarzik> in any case, we look at datFoo before checking 'ret
532 2012-05-22 23:28:16 <jgarzik> 'ret'