1 2012-06-02 00:54:31 <gribble> New news from bitcoinrss: luke-jr opened pull request 1409 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/pull/1409>
  2 2012-06-02 10:51:33 <pooolop> hi, i have a question abount merged mining + pooshpol + bitcoind 0.6.2.  is there any path to bitcoin 0.6.2 ? to get it all together work ?
  3 2012-06-02 11:33:25 <Eliel> pooolop: pooshpol? do you mean pushpoold?
  4 2012-06-02 11:53:30 <[7]> hm, doesn't seem like having bitcoin on an ssd is a good idea
  5 2012-06-02 11:53:54 <[7]> it makes bitcoin-qt on my laptop start up in minutes instead of seconds, but causes a damn lot of ssd wear
  6 2012-06-02 11:54:06 <[7]> seems like it's more than the rest of the system combined
  7 2012-06-02 11:54:26 <BlueMatt> that doesnt make sense
  8 2012-06-02 11:54:29 <Eliel> [7]: what kind of ssd?
  9 2012-06-02 11:54:43 <BlueMatt> nothing is gonna start slower on any kind of sane ssd than a rotation hdd
 10 2012-06-02 11:54:53 <BlueMatt> unless its one of those shitty ssd thats is unbearably slow
 11 2012-06-02 11:55:01 <[7]> er, i said that the wrong way round
 12 2012-06-02 11:55:14 <[7]> it starts in seconds instead of minutes, but causes awful lots of wear
 13 2012-06-02 12:05:07 <[7]> in the last 13h it caused a whopping 6GB of writes
 14 2012-06-02 12:05:31 <[7]> SSDs MTBFs are specified on 20GB/day IIUC
 15 2012-06-02 12:06:42 <[7]> my ssd has a lifetime write capacity of 1250TB
 16 2012-06-02 12:07:16 <sipa> 6Gb during chain syncup, or just stable usage?
 17 2012-06-02 12:07:37 <[7]> well, starting up (after a downtime of maybe 5 minutes) and then just idling
 18 2012-06-02 12:07:47 <[7]> so normal long-term behavior on a laptop
 19 2012-06-02 12:08:06 <gribble> New news from bitcoinrss: Diapolo opened issue 1410 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/issues/1410>
 20 2012-06-02 12:08:37 <[7]> 48MB during the last 6:45 minutes
 21 2012-06-02 12:08:52 <[7]> just idling in the background
 22 2012-06-02 12:09:16 <sipa> well, bitcoin dumps its address database (typically 1-2 megabytes) to a file every two minutes
 23 2012-06-02 12:09:20 <[7]> oddly enough it has only read 460MB while writing 6GB
 24 2012-06-02 12:09:30 <[7]> I suspect that this has something to do with massive fsyncs
 25 2012-06-02 12:09:38 <sipa> yes, certainly
 26 2012-06-02 12:10:51 <[7]> anything I could try to help nail down the cause?
 27 2012-06-02 12:10:56 <[7]> run it with eatmydata for a day?
 28 2012-06-02 12:12:01 <[7]> looks like this is one more reason to decouple blockchain storage from the wallet and GUI
 29 2012-06-02 12:12:14 <[7]> I do care both about disk space and writes on my laptop
 30 2012-06-02 12:12:22 <[7]> but I don't at all on my server, which is running a bitcoind anyway
 31 2012-06-02 12:12:32 <[7]> but I'd like to have bitcoin-qt on my laptop
 32 2012-06-02 12:12:58 <[7]> I don't care whether the wallet is on the server or on the laptop, but for some people the wallet would ideally be on a dedicated secure device
 33 2012-06-02 12:13:11 <[7]> so these three components should probably be decoupled somehow
 34 2012-06-02 12:13:50 <[7]> no way I'll let bitcoin cut my ssd life into half
 35 2012-06-02 12:15:02 <[7]> in case you're curious how much traffic it caused on your box, try this:
 36 2012-06-02 12:15:03 <[7]> $ cat /proc/$(pidof bitcoin-qt)/io
 37 2012-06-02 12:16:43 <[7]> so... any ideas how I could reduce that traffic, except for remote-X11ing it onto my server? :)
 38 2012-06-02 12:17:53 <[7]> my addr.dat is 6.5MB btw
 39 2012-06-02 12:17:58 <[7]> so that could be the main cause
 40 2012-06-02 12:18:09 <[7]> what exactly is stored in that file?
 41 2012-06-02 12:18:22 <[7]> what happens if I just delete it? will it regenerate it properly?
 42 2012-06-02 12:18:36 <[7]> i.e. could I move that to a ramdisk?
 43 2012-06-02 12:21:32 <BlueMatt> [7]: is there a way for you to track io usage per file?
 44 2012-06-02 12:21:48 <[7]> I don't know of any, but I'd have some other use for that as well
 45 2012-06-02 12:22:03 <BlueMatt> I think lsof output has it in there somehow?
 46 2012-06-02 12:24:33 <BlueMatt> though I have nfc if it counts just data written, or data physically written which could be significantly different due to all the fsyncs
 47 2012-06-02 12:25:30 <sipa> [7]: you can safely delete addr.dat
 48 2012-06-02 12:25:50 <sipa> [7]: 0.7.0 will stop using addr.dat and use peers.dat instead, with a more compact format
 49 2012-06-02 12:25:55 <BlueMatt> ln -s /dev/null addr.dat
 50 2012-06-02 12:26:11 <BlueMatt> how mad would bitcoind get with that?
 51 2012-06-02 12:26:32 <sipa> i guess for peers.dat it wouldn't really be a problem - bdb may complain :)
 52 2012-06-02 12:27:01 <BlueMatt> well, ln addr.dat to a tmpfs
 53 2012-06-02 12:28:02 <[7]> BlueMatt: lsof doesn't show traffic, just the current position in that file, right?
 54 2012-06-02 12:28:40 <BlueMatt> [7]: mmm, might not, sorry, I must be dreaming
 55 2012-06-02 12:29:37 <[7]> hm, bitcoin isn't happy *at all* with addr.dat symlinked to some nonexistant file on /tmp
 56 2012-06-02 12:30:04 <[7]> even linked to a zero-byte file:
 57 2012-06-02 12:30:09 <[7]> A fatal error occured. Bitcoin can no longer continue safely and will quit.
 58 2012-06-02 12:30:33 <BlueMatt> linked to a 0-byte file, probably not, linked to a working addr.dat on tmpfs, should work
 59 2012-06-02 12:32:16 <[7]> removing it altogether seems to work though
 60 2012-06-02 12:32:45 <[7]> after starting and stopping it, that file is like 500K
 61 2012-06-02 12:32:56 <BlueMatt> will it not get re-written all the time if you just remove it, though?
 62 2012-06-02 12:33:19 <[7]> I wouldn't care if it would rewrite it to a ramdisk all the time :P
 63 2012-06-02 12:33:30 <[7]> so i just need to find a way how to move that to /tmp
 64 2012-06-02 12:34:53 <[7]> will it go mad if i occasionally reset addr.dat to an older state?
 65 2012-06-02 12:35:10 <[7]> i.e. copy a template addr.dat to the ramdisk on every boot and symlink to that
 66 2012-06-02 12:35:25 <[7]> i.e. does it need to be in sync with the other files in .bitcoin?
 67 2012-06-02 12:35:52 <sipa> yes
 68 2012-06-02 12:36:01 <sipa> in particular, those in the database subdir
 69 2012-06-02 12:36:04 <BlueMatt> run with -dbdetach
 70 2012-06-02 12:36:12 <[7]> hm, damn
 71 2012-06-02 12:36:15 <BlueMatt> and remove database/* (assuming clean shutdown)
 72 2012-06-02 12:36:21 <[7]> BlueMatt: what does that flag do?
 73 2012-06-02 12:36:30 <BlueMatt> or...backup database and then remove
 74 2012-06-02 12:36:39 <[7]> i.e. rename database?
 75 2012-06-02 12:36:40 <BlueMatt> oh, thats only master, I think
 76 2012-06-02 12:36:49 <BlueMatt> yea
 77 2012-06-02 12:36:59 <BlueMatt> its default before master, or was it in 0.6.2, sipa?
 78 2012-06-02 12:37:27 <[7]> what does bitcoin store where btw?
 79 2012-06-02 12:37:35 <BlueMatt> its all in ~/.bitcoin
 80 2012-06-02 12:37:44 <[7]> how much effort is it to regenerate .bitcoin/database?
 81 2012-06-02 12:37:45 <BlueMatt> blkNNNN.dat are raw blocks
 82 2012-06-02 12:37:57 <[7]> what is blkindex.dat, and can it be regenerated from blkNNN.dat?
 83 2012-06-02 12:37:59 <sipa> -detachdb is since 0.6.0
 84 2012-06-02 12:38:10 <BlueMatt> its (usually) safe to remove .bitcoin/databse if you run with -detachdb and had a clean shutdown
 85 2012-06-02 12:38:29 <[7]> so what is stored in that directory?
 86 2012-06-02 12:38:31 <gribble> New news from bitcoinrss: TheBlueMatt opened pull request 1411 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/pull/1411>
 87 2012-06-02 12:38:32 <BlueMatt> but better to move it instead in case something goes wrong
 88 2012-06-02 12:38:43 <BlueMatt> bdb log files
 89 2012-06-02 12:38:51 <[7]> BlueMatt: well, worst case would be having to redownload the blockchain, right?
 90 2012-06-02 12:39:01 <BlueMatt> worse case would be corrupting your wallet
 91 2012-06-02 12:39:49 <[7]> yeah, but IIUC restoring an old wallet state (within 100 address generations) and removing the rest of .bitcoin should take care of pretty much any kind of failure, right?
 92 2012-06-02 12:40:07 <BlueMatt> yea, you may lose accounts, though
 93 2012-06-02 12:40:24 <[7]> accounts?
 94 2012-06-02 12:40:30 <BlueMatt> s/may/will/
 95 2012-06-02 12:40:47 <BlueMatt> eg address->human-readable name mappings
 96 2012-06-02 12:40:54 <[7]> where are those stored?
 97 2012-06-02 12:43:09 <sipa> you lose your address book, account balances, and transaction comments
 98 2012-06-02 12:43:12 <BlueMatt> wallet
 99 2012-06-02 12:43:19 <sipa> you will not lose money
100 2012-06-02 12:43:22 <[7]> ah, so just everything since the last wallet backup
101 2012-06-02 12:43:25 <BlueMatt> you'll lose everything but money
102 2012-06-02 12:43:34 <BlueMatt> essentially
103 2012-06-02 12:43:38 <[7]> ok, fine
104 2012-06-02 12:43:58 <[7]> i did remove addr.dat and database now, and it doesn't start anymore
105 2012-06-02 12:44:11 <[7]> EXCEPTION: 11DbException
106 2012-06-02 12:44:15 <[7]> i.e. invalid argument
107 2012-06-02 12:47:20 <[7]> putting database back doesn't help either
108 2012-06-02 12:47:30 <[7]> did it screw up by blksomething.dat files?
109 2012-06-02 12:48:00 <sipa> don't mess with the separate database files unless you started with -detachdb the previous run
110 2012-06-02 12:48:18 <sipa> copying files in/out or deleting them will corrupt the directory if you, otherwise
111 2012-06-02 12:48:19 <[7]> ah, the *previous* run... that's valuable information
112 2012-06-02 12:48:23 <sipa> (except for the wallet)
113 2012-06-02 12:48:38 <gribble> New news from bitcoinrss: TheBlueMatt reopened pull request 1411 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/pull/1411>
114 2012-06-02 12:48:39 <sipa> yes, that function disconnects the db files from their environment at shutdown
115 2012-06-02 12:48:42 <[7]> what is blk0001.dat btw? some proprietary format or BDB?
116 2012-06-02 12:48:55 <sipa> just a binary concatenation of all bitcoin blocks
117 2012-06-02 12:49:07 <[7]> i.e. it can't really have screwed that up?
118 2012-06-02 12:49:16 <[7]> so blkindex.dat is the victim?
119 2012-06-02 12:49:24 <sipa> no idea what you did
120 2012-06-02 12:49:34 <abracadabra> anybody have a problem adding addresses to bitcoin client?
121 2012-06-02 12:49:42 <sipa> abracadabra: define adding?
122 2012-06-02 12:49:43 <abracadabra> says i can't add because it's already there
123 2012-06-02 12:49:50 <[7]> sipa: basically removing the database dir and addr.dat and firing up bitcoin-qt
124 2012-06-02 12:49:50 <abracadabra> i'm adding a sending address
125 2012-06-02 12:50:04 <abracadabra> i check the address book, but it's not there
126 2012-06-02 12:50:06 <sipa> [7]: yes, blkindex.dat is the problem then
127 2012-06-02 12:50:10 <abracadabra> restarted the client.. same problem
128 2012-06-02 12:50:13 <sipa> [7]: i'm afraid you'll have to resync
129 2012-06-02 12:50:16 <abracadabra> i sorted address book on label and on address, it's not listed
130 2012-06-02 12:50:29 <sipa> abracadabra: is it an address of your own?
131 2012-06-02 12:50:36 <abracadabra> no
132 2012-06-02 12:50:48 <abracadabra> it's an adddress for someone else that i want to send btc to
133 2012-06-02 12:50:49 <[7]> sipa: well, there shouldn't be any need to re-download if the blk0001.dat file is there, right? i mean, it does have the blocks!
134 2012-06-02 12:51:10 <sipa> [7]: 0.7.0 has a -loadblock function to import an existing blk0001.dat file
135 2012-06-02 12:51:25 <sipa> well, will have
136 2012-06-02 12:51:39 <[7]> should copying over a blkindex.dat from another bitcoin install help?
137 2012-06-02 12:51:43 <sipa> no
138 2012-06-02 12:51:48 <[7]> blk0001.dat should be identical, shouldn't it?
139 2012-06-02 12:51:51 <sipa> no
140 2012-06-02 12:52:04 <sipa> it contains the blocks in the order you downloaded them
141 2012-06-02 12:52:11 <sipa> not in the order of the current best blockchain
142 2012-06-02 12:52:17 <[7]> but copying everything except the wallet from the other instance should work i guess?
143 2012-06-02 12:52:22 <sipa> yes
144 2012-06-02 12:52:39 <sipa> if you include the database/ directory
145 2012-06-02 12:52:55 <sipa> or first run the other instance with -detachdb
146 2012-06-02 12:56:15 <[7]> hm, what I'm actually looking for might be described as a "private e-wallet"
147 2012-06-02 12:56:30 <[7]> I don't really care whether it's a bitcoin-qt gui or a web application
148 2012-06-02 12:56:35 <[7]> is there some such thing?
149 2012-06-02 12:56:47 <sipa> define 'private' ?
150 2012-06-02 12:56:56 <[7]> hosted on my own box
151 2012-06-02 12:57:18 <[7]> basically a web gui for bitcoind
152 2012-06-02 12:57:31 <sipa> hmm, maybe
153 2012-06-02 12:58:07 <BlueMatt> tcatm has one, not sure how long its been since its been updated
154 2012-06-02 12:58:30 <BlueMatt> a year
155 2012-06-02 12:58:31 <BlueMatt> https://github.com/tcatm/bitcoin-js-remote
156 2012-06-02 12:58:49 <BlueMatt> would probably need updating, but I used to use it all the time
157 2012-06-02 12:59:02 <BlueMatt> theres also Spesmilo, but that also hasnt been updated in probably about the same amount of time
158 2012-06-02 12:59:22 <[7]> I'd prefer to stick with the official "satoshi" client
159 2012-06-02 12:59:32 <[7]> so that tcatm thing looks great :)
160 2012-06-02 12:59:40 <BlueMatt> spesmilo uses bitcoind as its backend
161 2012-06-02 12:59:47 <BlueMatt> as does bitcoin-js-remote
162 2012-06-02 13:00:03 <BlueMatt> major difference is just one is gui one is web
163 2012-06-02 13:00:41 <BlueMatt> but Id think both would need updating to use the features in modern bitcoind's, but...maybe not
164 2012-06-02 13:00:44 <[7]> so spesmilo can deal with a remote bitcoind i assume?
165 2012-06-02 13:00:56 <BlueMatt> it has to
166 2012-06-02 13:01:00 <BlueMatt> well, it can use local bitcoind
167 2012-06-02 13:01:45 <[7]> so, from your experience, which one of those two do you think is better?
168 2012-06-02 13:01:55 <[7]> i.e. more feature complete, more stable, more up to date, ...
169 2012-06-02 13:01:58 <BlueMatt> its been a while since Ive used either
170 2012-06-02 13:01:59 <[7]> easier to set up
171 2012-06-02 13:02:09 <BlueMatt> though, I probably used the latest version of both
172 2012-06-02 13:02:23 <BlueMatt> spesmilo wasnt perfectly stable, but was much easier to set up
173 2012-06-02 13:02:42 <sipa> any specific reason why you don't want bitcoin-qt ?
174 2012-06-02 13:02:46 <BlueMatt> iirc
175 2012-06-02 13:03:14 <[7]> sipa: because it requires the blockchain to be stored locally
176 2012-06-02 13:03:24 <sipa> you can run it over X forwarding :p
177 2012-06-02 13:03:31 <sipa> but ok, point taken
178 2012-06-02 13:03:56 <[7]> i have a bitcoind running for p2pool anyway, so no point in storing the blockchain twice
179 2012-06-02 13:04:22 <BlueMatt> give it a few months and there may be a blockchaind ;)
180 2012-06-02 13:04:51 <BlueMatt> (and manually backs-up wallet)
181 2012-06-02 13:05:01 <[7]> in a few months (which usually turns out to be actually a few years) this beast will have worn out my SSD :P
182 2012-06-02 13:05:22 <BlueMatt> heh, ok, probably
183 2012-06-02 13:28:15 <[7]> can bitcoind deal with an encrypted wallet?
184 2012-06-02 13:28:54 <BlueMatt> yea
185 2012-06-02 13:29:10 <BlueMatt> neither spesmilo nor bitcoin-js-remote can, Im sure, but bitcoind can
186 2012-06-02 13:29:18 <[7]> how would i specify the password? through the rpc protocol?
187 2012-06-02 13:30:01 <[7]> and how can i make bitcoind decrypt the wallet for now?
188 2012-06-02 13:31:29 <sipa> it only decrypts keys in it when it needs them
189 2012-06-02 13:31:39 <sipa> permanently decrypting to a file isn't implemented
190 2012-06-02 13:32:27 <slush> hi
191 2012-06-02 13:32:51 <slush> I'm writing simple tool which connects to bitcoin p2p network, but I have one issue
192 2012-06-02 13:33:27 <slush> some nodes, without any obvious reason, are sending me parts of their block chains, but I didn't requested it :-)
193 2012-06-02 13:33:34 <slush> Is there any way how to prevent this?
194 2012-06-02 13:34:14 <slush> I need to be bandwidth-conservative as much as possible...
195 2012-06-02 13:34:15 <sipa> really?
196 2012-06-02 13:34:24 <slush> yes, I'm sending only version packet
197 2012-06-02 13:34:31 <sipa> they send you block packets?
198 2012-06-02 13:34:35 <slush> yes
199 2012-06-02 13:34:46 <sipa> what version do they report?
200 2012-06-02 13:35:56 <slush> well, version and verack
201 2012-06-02 13:39:16 <slush> sipa: moment
202 2012-06-02 13:44:15 <gmaxwell> slush: I manually did that to some nodes that were reporting old chains, trying to unstick them.
203 2012-06-02 13:44:27 <gmaxwell> but what you're describing there sounds like an attack.
204 2012-06-02 13:46:21 <slush> well, it is probably my fault, I responded to "inv" packet with getdata, even on old hashes
205 2012-06-02 13:46:49 <slush> but I'm wondering why some nodes are sending inv packets with block hashes 2000 block old...
206 2012-06-02 13:47:01 <[7]> is there a way to make bitcoin-qt (or that JS gui) show which address some coins got generated to?
207 2012-06-02 13:47:12 <[7]> I'd like to be able to distinguish e.g. p2pool from eligius
208 2012-06-02 14:10:26 <slush> as an example, 129.241.137.181 is sending me "inv" for all blocks since #179439
209 2012-06-02 14:28:33 <slush> gmaxwell: "reporting old chains" = starting height in version packet?
210 2012-06-02 14:29:07 <gmaxwell> slush: right. But I'm not actively doing this, I went and tried it against a bunch a few weeks ago.
211 2012-06-02 14:32:58 <slush> gmaxwell: it looks like some other nodes are doing this as well. I'm reporting "0" in my starting height, because my tool doesn't know current blockchain
212 2012-06-02 14:33:03 <BlueMatt> anyone else notice that Alan Reiner's email "Full Clients in the future - Blockchain management" is essentially just describing how the satoshi client does it?
213 2012-06-02 14:35:31 <Eliel> well, it's pretty obvious way to do it :)
214 2012-06-02 14:38:54 <BlueMatt> true...
215 2012-06-02 14:41:55 <sipa> he seems to assume you want to have pointers to all transactions in ram
216 2012-06-02 14:42:18 <sipa> etotheipi_: right?
217 2012-06-02 14:43:18 <etotheipi_> okay... so how does the satoshi client do it?
218 2012-06-02 14:43:33 <etotheipi_> when the satoshi client is presented with a tx hash
219 2012-06-02 14:43:38 <etotheipi_> how does it find the tx?
220 2012-06-02 14:44:33 <etotheipi_> how does it even know if it has the tx?
221 2012-06-02 14:45:37 <BlueMatt> txindex.dat is the bdb db that stores CTxIndexes
222 2012-06-02 14:45:52 <BlueMatt> a CTxIndex is mostly info on where the tx was spent and where you can find it on disk
223 2012-06-02 14:47:27 <etotheipi_> you mean blkindex.dat?
224 2012-06-02 14:47:35 <BlueMatt> oh, yea sorry
225 2012-06-02 14:48:04 <etotheipi_> okay, well it's mildly concerning to me that blkindex.dat is 1/3 the size of the blockchain itself
226 2012-06-02 14:48:10 <etotheipi_> though I presume it has more info than just that
227 2012-06-02 14:48:25 <BlueMatt> its probably around 95% just that
228 2012-06-02 14:48:38 <BlueMatt> but its bdb, there is a lot of free space in the file
229 2012-06-02 14:48:55 <BlueMatt> also note that pruning it removes >1/2 of its size https://github.com/bitcoin/bitcoin/pull/1405
230 2012-06-02 14:49:58 <etotheipi_> so what is actually held in RAM?  when the blockchain has 1e9 transactions in it...
231 2012-06-02 14:50:34 <BlueMatt> we only hold CBlockIndexes in ram, ie pointers to the block on disk and the block headers
232 2012-06-02 14:52:26 <etotheipi_> oh, so BDB is handling that nastiness for you
233 2012-06-02 14:52:33 <BlueMatt> yep
234 2012-06-02 14:52:40 <etotheipi_> (nastiness, as in maintaining the txindex on disk without consuming ram)
235 2012-06-02 14:52:44 <BlueMatt> yep
236 2012-06-02 14:53:17 <gmaxwell> if not for all the complicated things like that we'd surely not use it. :)
237 2012-06-02 14:53:42 <Eliel> even more complex :D
238 2012-06-02 14:53:54 <BlueMatt> probably fairly similar, though more dealing with sql instead of neat C++ apis
239 2012-06-02 14:54:16 <BlueMatt> but...that would all be in CDB anyway, so probably almost identical in how the code works
240 2012-06-02 14:55:31 <gmaxwell> Eliel: all this needs is a key=value store (a hash table). Arguably bdb is itself huge overkill, except for a number of details (like etotheipi_'s concern) that make it not overkill.
241 2012-06-02 14:56:31 <Eliel> I see
242 2012-06-02 14:57:11 <etotheipi_> well it does seem to be remarkably space inefficient
243 2012-06-02 14:57:30 <etotheipi_> but otherwise handles this the way we want it to
244 2012-06-02 14:58:10 <Eliel> what would be the drawbacks of using CDB instead of BDB?
245 2012-06-02 14:58:22 <BlueMatt> CDB is just a wrapper around BDB
246 2012-06-02 14:58:27 <gmaxwell> etotheipi_: well, it's carrying indexes which more than double the size of it.
247 2012-06-02 14:58:50 <BlueMatt> It uses more space to get more efficiency in looking up and inserting records, and since block download is largely limited by the speed of txindex.dat, Id say its well worth any faster db we can find
248 2012-06-02 15:00:04 <Eliel> BlueMatt: libcdb1 package doesn't depend on anything but libc
249 2012-06-02 15:00:20 <BlueMatt> we arent using libcdb, we are using our own CDB class
250 2012-06-02 15:00:31 <Eliel> BlueMatt: I meant to ask about libcdb :)
251 2012-06-02 15:00:53 <Eliel> not the bitcoin CDB class
252 2012-06-02 15:01:06 <Eliel> http://cr.yp.to/cdb.html
253 2012-06-02 15:01:16 <BlueMatt> yea, there are thousands of dbs out there we could have chose, bdb is well known, well-tested and probably the most-used db in the world, so...satoshi picked that
254 2012-06-02 15:01:43 <BlueMatt> s/chose/chosen/
255 2012-06-02 15:04:07 <BlueMatt> Eliel: that said, I know jgarzik has been looking at replacements for bdb, mostly as a curiosity, I dont think its actually gonna happen just because there would need to be huge advantages to move everyone using bitcoind to create a whole new db, but...
256 2012-06-02 15:05:45 <Eliel> BlueMatt: I know, but it's relevant for etotheipi_ :)
257 2012-06-02 15:05:55 <Eliel> armory doesn't have to do that
258 2012-06-02 15:06:31 <dw> bdb also has nitty details like multple writers worked out a long time ago
259 2012-06-02 15:07:52 <Eliel> looks like cdb is a hashtable based file format.
260 2012-06-02 15:08:05 <dw> yep, and crappy incremental update support
261 2012-06-02 15:08:09 <sipa> etotheipi_: blkindex.dat contains for each txout which other tx spent it, where a single bit would suffice
262 2012-06-02 15:08:29 <sipa> though it makes detecting corruption easier
263 2012-06-02 15:08:40 <Eliel> ok... cdb has a 4GB size limit. No go.
264 2012-06-02 15:09:49 <dw> you can't really incrementally update a cdb file without wasting tonnes of space, and risking a damaged file on power failure
265 2012-06-02 15:10:00 <dw> so its a crappy format for anything that requires frequent mutation
266 2012-06-02 15:10:16 <sipa> jeff was looking at kyotocabinet
267 2012-06-02 15:10:51 <dw> kyoto's documentation sucks, its api doesn't appear to be thread safe, and the author only seems to care about performance :)
268 2012-06-02 15:11:10 <dw> it has some GetLastError() type call that is process global
269 2012-06-02 15:11:17 <BlueMatt> luckily for us, working around all of those are easy
270 2012-06-02 15:11:35 <BlueMatt> though better multi-threaded perf is nice for us
271 2012-06-02 15:18:30 <etotheipi_> well I admit, I have zero experience with database engines
272 2012-06-02 15:19:06 <etotheipi_> in hindsight, it's kind of a silly question... of course bitcoin is not the first application requiring such a solution
273 2012-06-02 15:19:16 <sipa> haha
274 2012-06-02 15:19:50 <jrmithdobbs> etotheipi_: ;p
275 2012-06-02 15:20:03 <sipa> by the way, by blockchain compression, do you mean pruning?
276 2012-06-02 15:20:05 <jrmithdobbs> ps: see, i'm not always a complete dick
277 2012-06-02 15:21:04 <jrmithdobbs> dw: you can update cdb without risk on power failure if you use it how it was designed to be used
278 2012-06-02 15:21:23 <jrmithdobbs> dw: but yes it is hugely wasteful of disk and mem i/o to do so atomically
279 2012-06-02 15:22:03 <etotheipi_> sipa, I mean unspent-tx-out-list-trees
280 2012-06-02 15:22:13 <dw> jrmithdobbs: yep, that's not update tho, thats more like rewrite :P
281 2012-06-02 15:22:28 <sipa> etotheipi_: how does that reduce any storage?
282 2012-06-02 15:22:30 <dw> also relies on filesystem to ensure atomic rename
283 2012-06-02 15:22:53 <jrmithdobbs> dw: well what do you expect from something designed to be able to do lookups of account existence and ip blacklists ;p
284 2012-06-02 15:22:59 <jrmithdobbs> dw: works pretty well for those things
285 2012-06-02 15:23:33 <sipa> etotheipi_: or you just mean: how to distribute just the essential information to verifying nodes that do not want to be a full node?
286 2012-06-02 15:23:35 <etotheipi_> if you have 10 million users sending 1 simple tx per day, your storage reqts increases by 10million tx per day... but with unspent txout-trees it should be constant
287 2012-06-02 15:24:21 <etotheipi_> of course it's never that simple... but there's a lot of space savings to be had, but I haven't worked out the compute complexity of maintaining such a tree yet
288 2012-06-02 15:24:27 <Eliel> etotheipi_: you're thinking more about the RAM footprint than disc foorprint here, right?
289 2012-06-02 15:24:33 <etotheipi_> no, disk foot print
290 2012-06-02 15:25:18 <etotheipi_> so that almost-full-nodes can store the essentials of the whole chain without storing the whole chain
291 2012-06-02 15:25:20 <sipa> etotheipi_: patricia tree on the tx hash, that maps to a bitvector?
292 2012-06-02 15:25:45 <gmaxwell> he's talking about open transaction trees, the flip chain stuff.
293 2012-06-02 15:25:55 <sipa> yes, i'm aware now
294 2012-06-02 15:26:40 <etotheipi_> unfortunately, now is not a good time for me to discuss it, but the tree-structure idea is here:  https://bitcointalk.org/index.php?topic=52859.msg885838#msg885838
295 2012-06-02 15:27:08 <etotheipi_> which would give lightweight nodes the ability to retrieve&verify their unspent-output-lists with only a few kB/addr
296 2012-06-02 15:27:34 <etotheipi_> and then another post brought up the idea that the hashes of those trees could be maintained in a separate blockchain with merged mining
297 2012-06-02 15:27:42 <etotheipi_> which actually makes this possible, as opposed to a pipe dream
298 2012-06-02 15:28:01 <gmaxwell> I'm not fond of address ordering the tree because I don't see how to keep it balanced.
299 2012-06-02 15:28:37 <sipa> also, for a normal usecase, a wallet has no keys in it from the past when it is generated
300 2012-06-02 15:28:53 <sipa> sure it's extremely useful for rescanning
301 2012-06-02 15:29:05 <gmaxwell> Though my fondness of that would increase once I was convinced it was possible to avoid an attacker creating linear update cost.
302 2012-06-02 15:29:10 <sipa> but i think we should rather find ways to avoid rescanning altogether
303 2012-06-02 15:29:24 <etotheipi_> gmaxwell: I don't see where there is a balance issue
304 2012-06-02 15:29:44 <etotheipi_> sipa: well if such a tree is maintained... you can rescan your whole wallet extremely quickly
305 2012-06-02 15:30:01 <sipa> etotheipi_: exactly what i said :)
306 2012-06-02 15:30:02 <etotheipi_> as I said, a few kB per address... and it's verifiable against the headers
307 2012-06-02 15:31:07 <sipa> i don't disagree that it is perfect for doing fast rescans, but i don't think we should be aiming for technologies that require rescanning at all (except maybe in recover-from-disaster scenarios)
308 2012-06-02 15:31:41 <etotheipi_> sipa: I don't understand... the trees can be updated incrementally
309 2012-06-02 15:31:53 <etotheipi_> someone, somewhere, at some time has to do a full scan to create the tree
310 2012-06-02 15:32:02 <etotheipi_> well, multiple people...
311 2012-06-02 15:32:34 <gmaxwell> I'm much more interested in things that enable cheap block validation. But as you point out, your address tree thing could actually just be mergemined.
312 2012-06-02 15:33:20 <sipa> only the root of the address tree, i suppose?
313 2012-06-02 15:33:31 <etotheipi_> then you acquire the tree once... and incrementally update it with new blocks/txs
314 2012-06-02 15:33:31 <sipa> merkle root
315 2012-06-02 15:33:43 <sipa> yes, of course
316 2012-06-02 15:33:52 <etotheipi_> right, the second chain, block headers contain the merkle root of the tree
317 2012-06-02 15:34:05 <sipa> much in the same way as an open-tx-set could be maintained
318 2012-06-02 15:34:13 <etotheipi_> and someone can download from peers the latest "snapshot"
319 2012-06-02 15:34:20 <etotheipi_> and verify it against the headers
320 2012-06-02 15:34:30 <sipa> yes, yes, i understand :)
321 2012-06-02 15:34:46 <etotheipi_> okay... as usual, I'm slow at figuring out where the discussion is diverging...
322 2012-06-02 15:34:50 <sipa> no need to convince me what it does or why it is useful
323 2012-06-02 15:35:16 <sipa> i'm just arguing that there may be a viable future where it isn't needed at all
324 2012-06-02 15:35:27 <Eliel> this sounds similar somehow to where I heard realsolid is heading with his next solidcoin (or microcash as he's renaming it)
325 2012-06-02 15:36:01 <etotheipi_> well honestly, I don't think it's worth it just for the compression aspect
326 2012-06-02 15:36:13 <gmaxwell> sure, where do you think he got the ideas for it? (not etotheipi_'s comments as much as bytecoins older discussion)
327 2012-06-02 15:36:23 <etotheipi_> but for lightweight nodes to be able to quickly verify the balance of any address...
328 2012-06-02 15:36:36 <Eliel> gmaxwell: ah, of course :)
329 2012-06-02 15:37:27 <etotheipi_> for instance, light nodes can verify that a particular incoming zero-conf transaction is actually valid and unspent
330 2012-06-02 15:38:25 <gmaxwell> etotheipi_: you need to be indexed by txns not by 'address' for that.
331 2012-06-02 15:39:05 <gmaxwell> Or at least that what I'd been thinking Because there can be multiple addresses to satisify an output, but perhaps not.
332 2012-06-02 15:40:05 <Eliel> etotheipi_: how would the tree-update algorithm react to an attacker that kept doing transactions that try to overload one small area of the hash space?
333 2012-06-02 15:40:47 <gmaxwell> Eliel: thats my comment about balancing.
334 2012-06-02 15:41:24 <gmaxwell> amiller was commenting some on some magic for this, but I haven't had a chance to read all the papers he directed me to you.
335 2012-06-02 15:41:31 <gmaxwell> gah. s/you/yet/
336 2012-06-02 15:42:16 <Eliel> although, the maximum depth would still be limited since the processing power needed for increasing it would double for each increment of one.
337 2012-06-02 15:43:40 <etotheipi_> gmaxwell: excellent point... I was still assuming that the treenodes were accessible by OutPoint, but that's not the key anymore in my scheme
338 2012-06-02 15:44:23 <etotheipi_> though, it wouldn't be hard to maintain a second tree indexed by OutPoint that has iterators to the address-indexed tree
339 2012-06-02 15:44:30 <etotheipi_> but that is getting contrived
340 2012-06-02 15:44:51 <sipa> well, we need the outpoint trew no matter what
341 2012-06-02 15:45:26 <sipa> tree
342 2012-06-02 15:46:23 <etotheipi_> speaking of patricia trees... anyone familiar with a "de la brandia tree"?
343 2012-06-02 15:46:23 <sipa> and a bitvector of spentness per tx is probably smaller than your iterator
344 2012-06-02 15:46:29 <gmaxwell> etotheipi_: right, I think thats the more interesting thing allowing for light fullnodes is important to me keeps us decentralized.  Though I also think your fast-secure-address indexes are somewhat interesting too.
345 2012-06-02 15:46:54 <gmaxwell> Never heard of that.
346 2012-06-02 15:47:02 <sipa> neither have i
347 2012-06-02 15:47:12 <etotheipi_> I learned about de la brandia trees in my data-structures class along with patricia trees
348 2012-06-02 15:47:19 <etotheipi_> but now I see no evidence they actually exist
349 2012-06-02 15:47:27 <etotheipi_> or rather, maybe there were called something else
350 2012-06-02 15:47:54 <etotheipi_> they were essentially patricia tress but using linked lists instead of pointer arrays
351 2012-06-02 15:48:04 <etotheipi_> reducing the memory cost of each node
352 2012-06-02 15:50:26 <etotheipi_> well back to the original point:  yes, I like the way we can maintain some extra degree of decentralization in this manner
353 2012-06-02 15:50:57 <etotheipi_> but there's so many details I can't even encapsulate what it would take and what the potential problems are with this kind of implementation
354 2012-06-02 15:51:16 <etotheipi_> I understand the concept of merged mining, but don't know what the risks are...
355 2012-06-02 15:52:48 <Nesetalis> ... something is weird... i've had a transaction pending since 11:40pm last night... (its 2pm today) still 0 confirmations.
356 2012-06-02 15:53:37 <TD> hmm
357 2012-06-02 15:53:46 <TD> is there any actually complete implementation of lightweight nodes, yet?
358 2012-06-02 15:53:53 <TD> i mean of the type that connects directly to the p2p network
359 2012-06-02 15:53:57 <TD> bitcoinj isn't .....
360 2012-06-02 15:55:33 <etotheipi_> well there will have to be...
361 2012-06-02 15:55:43 <Nesetalis> oh and NOW it hits the network, as soon as I mention it in here
362 2012-06-02 15:55:53 <etotheipi_> in fact, I'd like to make Armory lightweight in standard-user mode
363 2012-06-02 15:55:58 <etotheipi_> though I"m a ways off from that goal
364 2012-06-02 15:59:12 <sipa> TD: what do you mean by a lightweight node that connects to the P2P network?
365 2012-06-02 15:59:17 <TD> spv
366 2012-06-02 16:02:02 <sipa> ah, ok; i interpreted lightweight as something weaker, more like electrum
367 2012-06-02 16:05:50 <BlueMatt> TD: Im pretty sure there isnt
368 2012-06-02 16:06:00 <TD> well
369 2012-06-02 16:06:08 <BlueMatt> TD: but thats like #2 on my list todo in the next ~month
370 2012-06-02 16:06:20 <TD> to me there's little difference between something like electrum and SPV+a protocol extension to allow server side filtering of blocks
371 2012-06-02 16:06:52 <TD> once you have that, there's little benefit to putting all the logic on the server beyond "clients are simple", but you can't avoid dealing with the block chain
372 2012-06-02 16:07:02 <TD> you can only move the complexity around. so it may as well be on the client, at that point
373 2012-06-02 16:07:06 <BlueMatt> the way I always saw spv is you download full blocks, just dont store them
374 2012-06-02 16:07:20 <TD> well that's how it works now
375 2012-06-02 16:07:31 <TD> but in future it should be "you download blocks plus transactions that match a filter"
376 2012-06-02 16:08:57 <BlueMatt> +optionally download full blocks, Imho thats an important options for those who want to be able to run trust-less-ish
377 2012-06-02 16:11:50 <BlueMatt> but, yea, not downloading full blocks on mobile clients would be very important
378 2012-06-02 16:28:14 <Eliel> gmaxwell: I don't think the balance problem in the address indexed tree is a big issue. As long as the addresses are all the output of a cryptographic hash-algorithm, it'll take very high amounts of processing power to even be able to make it noticeable.
379 2012-06-02 16:28:43 <Eliel> it would be much more effective and economical to just spam the network with transactions.
380 2012-06-02 16:30:17 <luke-jr> sipa: is it possible to construct a signature with a specific fixed prefix/middle/suffix?
381 2012-06-02 16:34:01 <sipa> luke-jr: i doubt that
382 2012-06-02 16:34:23 <luke-jr> hmm
383 2012-06-02 16:34:36 <luke-jr> any way to make signatures cheaper?
384 2012-06-02 16:35:10 <sipa> 64 bytes suffices, instead of the DER encoding bitcoin uses...
385 2012-06-02 16:36:30 <luke-jr> coinbase& 100 - 5 for height - 36 for MM - up to 8 for extranonce = 51 left
386 2012-06-02 16:36:56 <luke-jr> hmm
387 2012-06-02 16:37:12 <luke-jr> I suppose I could stuff it in the pool output <.,
388 2012-06-02 16:37:14 <luke-jr> <.<*
389 2012-06-02 16:40:07 <sipa> you want to put a signature in the block chain?
390 2012-06-02 16:40:18 <sipa> in the coinbase, i mean
391 2012-06-02 16:40:28 <luke-jr> that'd be useful, yes
392 2012-06-02 16:40:41 <sipa> for? identifying pools?
393 2012-06-02 16:48:03 <TD> BlueMatt: even with filtering the trust level is still pretty low. a remote peer can hide transactions from you, but that's about it. if you select a bunch of peers, hopefully you can spot the discrepancy. i think over time the use of things like Tor will become routine, exactly to make such attacks difficult - if you don't know who is connecting to you, you can't easily hide transactions
394 2012-06-02 16:48:09 <TD> (without being trivially caught)
395 2012-06-02 16:48:49 <BlueMatt> TD: absolutely, but, in the end, it is still trust
396 2012-06-02 16:49:04 <BlueMatt> TD: and I know there are quite a few people who like the no-trust aspect of bitcoin
397 2012-06-02 16:49:10 <TD> sure
398 2012-06-02 16:49:14 <TD> it's just a bandwidth tradeoff
399 2012-06-02 16:49:17 <BlueMatt> yea
400 2012-06-02 16:49:36 <TD> it'd be nice to find some way of allowing you to do server-side filtering without the remote node being able to lie to you
401 2012-06-02 16:49:55 <TD> i'm thinking if you use loose enough filters and random spot checks of blocks, or something ..... not sure
402 2012-06-02 16:50:42 <BlueMatt> yea...that plus connect to 5 unrelated servers and Id think your % chance of missing a tx would be low
403 2012-06-02 16:50:55 <gmaxwell> TD: the committed tree setups allow provable naks.
404 2012-06-02 16:50:59 <BlueMatt> and a "do I have txes?" "no" exchange is very low bandwidth
405 2012-06-02 16:51:21 <gmaxwell> Which make it impossible to tell an undetectable lie.
406 2012-06-02 16:52:02 <TD> committed tree setups? this is where you include a  merkle tree of unspent outputs in the coinbase or something else?
407 2012-06-02 16:52:24 <sipa> that
408 2012-06-02 16:53:57 <gmaxwell> Right. The server gives you the headers and the connecting branches.  So the only way they can lie is by mining a doomed fork, and even that is detectable if you know the current time, and some of the correct chain from the not too distant past, unless they have an awful lot of hash power.
409 2012-06-02 16:56:18 <BlueMatt> sending a full merkle tree of unspent outputs could get expensive, though you could enforce ordering the leaves and send the branches surrounding your addresses...still could get bw expensive either way
410 2012-06-02 16:56:50 <gmaxwell> BlueMatt: you've missed etotheipi's proposal.
411 2012-06-02 16:57:00 <gmaxwell> BlueMatt: which is basically to use the output as the tree key.
412 2012-06-02 16:57:56 <gmaxwell> so I can just give you a list of keys I'm interested in updates on. (plus information about the trees I currently have for them) and you can send me the difference and the connecting branch to show that it's POW-confirmed.
413 2012-06-02 16:59:28 <BlueMatt> ah...yea ok
414 2012-06-02 17:00:03 <BlueMatt> still, compared to a raw list of txes or "no" its a significant order of magnitude greater bw (but Id think even mobile clients could handle that)
415 2012-06-02 17:03:01 <gmaxwell> well, you can still combine it with things like spotchecks. e.g. "Do I have new txns?, give me a signed response." "No." "Okay, prove it." "Ummm errr You see."  "Hey everybody, node XYZ lied to me and here is the proof"    making the spotchecks stronger has value.
416 2012-06-02 17:03:28 <gmaxwell> Withthout the tree proof you can only spotcheck by checking several nodes and they could be sybils.
417 2012-06-02 17:03:43 <gmaxwell> Plus you then have to load up several nodes doing the check.
418 2012-06-02 17:04:14 <BlueMatt> yea, no that would work pretty well...
419 2012-06-02 19:14:40 <jrmithdobbs> sipa: so how do you specify which address to advertise when using ipv6 on a dualt stack host and you want both types of connections?
420 2012-06-02 19:15:17 <jrmithdobbs> because i have a several ip6 assignments on the host, most of which really shouldn't accept bitcoin connections
421 2012-06-02 19:17:51 <sipa> jrmithdobbs: you select where to bind with -bind, you specify the address to advertize with using -externalip
422 2012-06-02 19:17:56 <jrmithdobbs> sipa: can you specify multiple -bind= and -externalip= ?
423 2012-06-02 19:17:58 <sipa> both can be specified multiple times
424 2012-06-02 19:18:07 <jrmithdobbs> ok cool
425 2012-06-02 19:18:26 <sipa> and externalip works +- smart, so it will favor giving an ipv6 address to ipv6 peers
426 2012-06-02 19:18:46 <jrmithdobbs> so -bind=0.0.0.0 -bind=[::] -externalip=ip4pub -externalip=ip6pub should work?
427 2012-06-02 19:18:56 <sipa> yup
428 2012-06-02 19:19:08 <sipa> though those two binds are the default anyway
429 2012-06-02 19:19:15 <sipa> so no need to specify them
430 2012-06-02 19:19:16 <jrmithdobbs> right
431 2012-06-02 19:19:27 <jrmithdobbs> just an example
432 2012-06-02 19:20:14 <sipa> there's a few more pull requests open that change some things though
433 2012-06-02 19:20:25 <sipa> i don't know exactly anymore what's already merged
434 2012-06-02 19:20:33 <jrmithdobbs> i already filter the connections on other ips at network level but would rather not pollute people's addr.dat if i am
435 2012-06-02 19:22:35 <jrmithdobbs> didn't several things get merged a while back to fix building on *bsd?
436 2012-06-02 19:22:53 <sipa> yes
437 2012-06-02 19:23:18 <jrmithdobbs> anyone actively testing?
438 2012-06-02 19:23:57 <jrmithdobbs> this is host is about to get migrated so i guess i'll find out
439 2012-06-02 19:28:31 <sipa> there's a few bsd users
440 2012-06-02 19:33:20 <etotheipi_> sipa: what is the status of the new wallet format?
441 2012-06-02 19:33:39 <etotheipi_> I'm not in any rush, I'm just curious
442 2012-06-02 19:35:43 <sipa> etotheipi_: you mean the deterministic wallets, or the append-only format?
443 2012-06-02 19:38:11 <etotheipi_> sorry, I meant deterministic wallets
444 2012-06-02 19:38:39 <etotheipi_> I didn't know you were working on an append-only format
445 2012-06-02 19:38:39 <sipa> regarding the seeding idea
446 2012-06-02 19:39:00 <sipa> ah
447 2012-06-02 19:39:27 <sipa> it's an append-only key=value store really, written in blocks to the file, each with a cryptographic checksum
448 2012-06-02 19:39:55 <sipa> so it can detect corrupted only half-written blocks and ignore them
449 2012-06-02 19:39:59 <etotheipi_> ahh
450 2012-06-02 19:40:08 <sipa> which should always result in a consistent state, with just a single file
451 2012-06-02 19:40:27 <etotheipi_> ahh, that's a different way to do it
452 2012-06-02 19:40:43 <etotheipi_> I just detect corruption and restore... you decided to write it in a way that it can just be skipped
453 2012-06-02 19:41:19 <sipa> but it will need rewrite-close-atomicmove cycles anyway, as for example last known best block changes too frequently to me negligable
454 2012-06-02 19:41:24 <etotheipi_> so how do you know that a partially-written block will have enough info in it to be meaningful?
455 2012-06-02 19:41:32 <etotheipi_> oh... constant block size?
456 2012-06-02 19:41:36 <sipa> no
457 2012-06-02 19:41:47 <sipa> if the checksum of the block doesn't compute, ignore the entire block
458 2012-06-02 19:42:06 <sipa> that means you lost durability, but the former block represented a former consistent state
459 2012-06-02 19:42:08 <etotheipi_> but how do you know that the blocksize field didn't get corrupted somehow?
460 2012-06-02 19:42:48 <sipa> if the blocksize field got corrupted to a too small value, you'll notice when trying to decode or certainly when checking the checksum
461 2012-06-02 19:43:04 <sipa> if the blocksize field is too large, it'll take you past the end of the file
462 2012-06-02 19:43:28 <sipa> which is also easily detectable
463 2012-06-02 19:43:34 <etotheipi_> that's the one thing that always concerned me... if that value gets corrupted, you don't even know where the next block starts
464 2012-06-02 19:44:08 <sipa> well, there are 4 marker bytes before each block, so you can search for those
465 2012-06-02 19:44:27 <etotheipi_> gotch
466 2012-06-02 19:44:29 <etotheipi_> a
467 2012-06-02 19:44:47 <etotheipi_> actually, I should consider something similar
468 2012-06-02 19:45:10 <etotheipi_> I don't use marker bytes
469 2012-06-02 19:45:31 <sipa> i'd like a way for doing in-place updates anyway, but it's probably not worth the extra complexity
470 2012-06-02 19:45:32 <etotheipi_> ehh... I don't think it matters... if anything goes wrong with my wallet, it's restored from the backup
471 2012-06-02 19:45:38 <sipa> indeed
472 2012-06-02 19:45:58 <etotheipi_> how so?  why can't you do in-place updates?
473 2012-06-02 19:46:10 <sipa> what if they get written only partially?
474 2012-06-02 19:46:15 <sipa> and they'll corrupt the checksums
475 2012-06-02 19:46:24 <sipa> sure that's all fixable, but it becomes hard
476 2012-06-02 19:47:32 <etotheipi_> I don't understand, why is overwriting different than adding?  in both cases a partial write leads to a messed up block of data
477 2012-06-02 19:47:58 <sipa> overwriting something means at least temporarily corrupting a block that was perfectly consistent before
478 2012-06-02 19:48:02 <sipa> i don't like that
479 2012-06-02 19:48:44 <sipa> and adding is different because it doesn't overwrite live data
480 2012-06-02 19:48:48 <etotheipi_> true, you risk losing stuff
481 2012-06-02 19:48:53 <sipa> indeed
482 2012-06-02 19:49:07 <etotheipi_> oh, that's a good point, too
483 2012-06-02 19:49:11 <etotheipi_> I've been avoiding multi-threading
484 2012-06-02 19:49:36 <sipa> plus, i'm sure that if i can show that the code never overwrites live data, it's much easier to gain trust that it will work
485 2012-06-02 19:50:11 <etotheipi_> I assume you'll be at least blanking the old data?
486 2012-06-02 19:50:21 <etotheipi_> in my file format, I have a code for "deleted"
487 2012-06-02 19:50:28 <sipa> no, that would corrupt the checksums
488 2012-06-02 19:50:36 <sipa> it's strictly append only
489 2012-06-02 19:50:49 <etotheipi_> what about previously-decrypted key data?
490 2012-06-02 19:50:55 <sipa> ?
491 2012-06-02 19:50:57 <gmaxwell> make a new file.
492 2012-06-02 19:51:14 <sipa> yes, that's why there are rewrite + move atomically cycles
493 2012-06-02 19:51:16 <gmaxwell> which we already have to do because bdb's deletion of data isn't trustworthy.
494 2012-06-02 19:51:27 <etotheipi_> oh, fair enough
495 2012-06-02 19:51:45 <sipa> that compacts the old space away, and rewrites all remainind old data in a single block
496 2012-06-02 19:52:48 <sipa> it could be rewrite to .bak file, close .bak file, close real file, rewrite real file too
497 2012-06-02 19:53:04 <sipa> so you always have a backup, like you do
498 2012-06-02 19:53:30 <sipa> and there are no problems with the wallet file being a symlink
499 2012-06-02 19:53:47 <etotheipi_> btw, stupid story
500 2012-06-02 19:54:08 <etotheipi_> some guy emailed me to tell me that he used armory for the first time, put 107 BTC into it, then restarted and all his coins were gone
501 2012-06-02 19:54:30 <etotheipi_> I must've spent 20 emails with him figuring out what happened to the wallet file, and how it could get corrupted so badly
502 2012-06-02 19:54:44 <etotheipi_> I would've thought it was a scam, except he never asked for me to replace them
503 2012-06-02 19:55:36 <sipa> and he used the wrong password?
504 2012-06-02 19:55:44 <etotheipi_> it turns out he had never actually opened Armory -- he had opened multibit, transferred 107 BTC, then closed it and reopened Armory
505 2012-06-02 19:55:52 <sipa> hahaha
506 2012-06-02 19:57:01 <sipa> anyway, gmaxwell, etotheipi_: about the master key seeding
507 2012-06-02 19:57:06 <etotheipi_> he donated 10 BTC for that
508 2012-06-02 19:57:28 <sipa> how acceptable is using a table of 16 iteration count / compare values, for the various strengths?
509 2012-06-02 19:58:10 <etotheipi_> you're going to have to elaborate on that ...
510 2012-06-02 19:58:24 <sipa> https://gist.github.com/2731997
511 2012-06-02 19:59:12 <jrmithdobbs> sipa: so uh, is there a way to force it to try and get at least one ip6 peer from seeds?
512 2012-06-02 19:59:20 <jrmithdobbs> without running ip6 only
513 2012-06-02 19:59:21 <sipa> jrmithdobbs: -addnode ?
514 2012-06-02 20:00:00 <etotheipi_> sipa: I'm going to need some time to absorb this
515 2012-06-02 20:00:13 <jrmithdobbs> sipa: so to bootstrap a dualstack host you really need to -addnode to ensure you get at least one of each?
516 2012-06-02 20:01:02 <sipa> jrmithdobbs: i hope that after a while there will be plenty of ipv6 addresses in the dns seeds
517 2012-06-02 20:01:36 <etotheipi_> sipa: what is A and B?
518 2012-06-02 20:01:43 <jrmithdobbs> sipa: well, i just started it up and it doesn't look like it even tried any ip6 hosts from the log
519 2012-06-02 20:02:11 <jrmithdobbs> sipa: normal 8 max outbound, 512 inbound, but since no v6 nodes it has broadcasted that address so your crawler wont pick it up ;p
520 2012-06-02 20:02:18 <jrmithdobbs> err hasn't
521 2012-06-02 20:02:30 <sipa> etotheipi_: basically, it's replacing the "check whether N bits are zero after 2^N steps" with "check whether the first 32-bit word of the hash is less than 2^32*f(N) after i(N) iterations
522 2012-06-02 20:02:41 <jrmithdobbs> i can addnode to force bootstrap
523 2012-06-02 20:03:14 <jrmithdobbs> but that might slow ip6 integration for a while if people need to do that and don't
524 2012-06-02 20:03:26 <sipa> etotheipi_: and an optimization process to find optimal values for f(N) and i(N) such that you get expected runtimes of A*B^N
525 2012-06-02 20:03:39 <sipa> etotheipi_: plus some other constraint
526 2012-06-02 20:04:00 <sipa> jrmithdobbs: but you're not running dual stack?
527 2012-06-02 20:04:01 <jrmithdobbs> sipa: will especially be an issue on hosts that have aaaa records deprioritized in /etc/gai.conf
528 2012-06-02 20:04:04 <sipa> no, you are
529 2012-06-02 20:04:15 <jrmithdobbs> ya i am
530 2012-06-02 20:04:44 <sipa> well, there is such an abundance for ipv4 nodes that there is just not much chance for picking an ipv6 one
531 2012-06-02 20:05:13 <etotheipi_> so what is A and B?
532 2012-06-02 20:05:32 <sipa> whatever we decide, but i propose 65536 and 2
533 2012-06-02 20:05:47 <jrmithdobbs> ya, maybe at least temporarily it might be worthwile to add some logic to try and hold one slot for an ip6 seed if a routable v6 address is found or explicitly -externalip'ed
534 2012-06-02 20:05:58 <jrmithdobbs> one outbound i mean
535 2012-06-02 20:06:05 <sipa> jrmithdobbs: probably we can add a v6-only dns seed
536 2012-06-02 20:06:25 <jrmithdobbs> sipa: or could add a v6-only outbound connect pool and give them both 8 outbound
537 2012-06-02 20:06:56 <jrmithdobbs> that might make sense really
538 2012-06-02 20:07:46 <sipa> etotheipi_: advantage: you can pretty well estimate the runtime for finding a key, and they are separately better than a factor 4 apart
539 2012-06-02 20:07:54 <sipa> s/key/seed/
540 2012-06-02 20:08:06 <sipa> s/separately/separated/
541 2012-06-02 20:09:28 <jrmithdobbs> sipa: tbqh, for bitcoind, i'm half tempted to add a #define to prevent building any of the seed logic and just pre-parse your seed stats csv and -addnode
542 2012-06-02 20:09:31 <jrmithdobbs> ha
543 2012-06-02 20:09:53 <jrmithdobbs> everyone providing a seed should provide same-format stats information ;p
544 2012-06-02 20:10:08 <sipa> jrmithdobbs: you know my crawler that creates that file also exposes it via my dns seed?
545 2012-06-02 20:12:19 <jrmithdobbs> sipa: txt records?
546 2012-06-02 20:12:21 <etotheipi_> sipa: I'm getting there... I'm just getting caught up in the equations and their meaning, with only single-letter abstract names
547 2012-06-02 20:12:52 <sipa> etotheipi_: it wasn't really intended as a real explanation, just notes of my own calculations so i can redo them :)
548 2012-06-02 20:13:04 <Eliel> sipa: I don't think the in-place update is necessary. You can always just append a block that "replaces" the old data.
549 2012-06-02 20:13:05 <sipa> jrmithdobbs: eh no, A and AAAA
550 2012-06-02 20:13:11 <jrmithdobbs> sipa: because that'd be much better than me pulling the whole thing (and sorry btw, i'll put that behind squid until i find out by what you mean, ha)
551 2012-06-02 20:14:40 <sipa> Eliel: sure, that's what happens now
552 2012-06-02 20:14:55 <sipa> Eliel: (there's a prototype in my logdb branch)
553 2012-06-02 20:15:02 <jrmithdobbs> sipa: should setup a reverse ip.arpa and in-addr.arpa style zone to pull the stats for specific nodes too
554 2012-06-02 20:15:09 <Eliel> sipa: the occasional rewrite will then clean things up nicely :)
555 2012-06-02 20:15:19 <sipa> Eliel: exactly, that's the plan
556 2012-06-02 20:15:37 <jrmithdobbs> sipa: like 255.255.255.10.in-addr.sipa.be IN TXT "<csv row>"
557 2012-06-02 20:15:51 <sipa> jrmithdobbs: i'd rather not
558 2012-06-02 20:15:54 <jrmithdobbs> then to query for yourself you don't have to pull the whole thing
559 2012-06-02 20:16:08 <jrmithdobbs> oh, ya, i forgot how you're doing the dns stuff ;p
560 2012-06-02 20:16:18 <sipa> ?
561 2012-06-02 20:16:44 <jrmithdobbs> it'd be a pretty simple change to the stats output code if it were just generating zone files
562 2012-06-02 20:16:55 <sipa> oh implementing it wouldn't be hard
563 2012-06-02 20:17:06 <sipa> but i prefer not to expose too much information
564 2012-06-02 20:17:20 <jrmithdobbs> well the information is already exposed in the stats data
565 2012-06-02 20:17:34 <sipa> i don't intend to keep having that public forever
566 2012-06-02 20:17:56 <jrmithdobbs> why? it's great for people running public nodes to use as a sort of bitcoin looking glass
567 2012-06-02 20:18:31 <Eliel> sipa: do you think it would make sense to store block headers (and perhaps merkle trees too) in one file and transactions in another?
568 2012-06-02 20:18:38 <jrmithdobbs> i've noticed problems on my node because of it, in fact
569 2012-06-02 20:18:41 <sipa> Eliel: yes
570 2012-06-02 20:19:08 <Eliel> is this being planned for 0.7?
571 2012-06-02 20:19:19 <etotheipi_> sipa: I'm almost there, but I don't understand what H(n) is computing
572 2012-06-02 20:20:43 <sipa> etotheipi_: how many iterations you've done on average *before* reaching strength N
573 2012-06-02 20:21:26 <sipa> I(n) extends that with the number of iterations at strength N itself
574 2012-06-02 20:21:49 <sipa> iirc
575 2012-06-02 20:21:59 <sipa> etotheipi_: it's implemented even by jeff, but there were some performance issues left
576 2012-06-02 20:22:02 <sipa> eh
577 2012-06-02 20:22:08 <sipa> Eliel: it's implemented even by jeff, but there were some performance issues left
578 2012-06-02 20:22:34 <sipa> Eliel: split blkindex.dat into blkhash.dat, txhash.dat and blkmeta.dat or something
579 2012-06-02 20:23:23 <etotheipi_> sipa: what is the conops?  I thought we pick a value N, and tons of seeds until we find one that meets the criteria
580 2012-06-02 20:23:27 <Eliel> sipa: oh, I actually meant splitting blk0001.dat into two :)
581 2012-06-02 20:23:39 <etotheipi_> but are you solving for lower-strength seeds, first?
582 2012-06-02 20:24:16 <Eliel> sipa: with one containing the block headers + merkle trees and the other only transactions.
583 2012-06-02 20:25:22 <sipa> etotheipi_: yes, you pick a value for N in advance, and then generate random seeds in a loop, and for all k<N, test whether the predicate doesn't hold, and for N it does
584 2012-06-02 20:25:59 <sipa> etotheipi_: sorry if the formulas are a bit dense - i found them by just working out a concrete example and then generalizing; it's probably a bit harder to verify its correctness now
585 2012-06-02 20:26:47 <sipa> etotheipi_: so you've picked N; let's say we want on average 2^20 iterations, so we pick N=4
586 2012-06-02 20:27:02 <sipa> that's the goal strength of the seed we are generating
587 2012-06-02 20:27:47 <sipa> you randomly generate a seed S
588 2012-06-02 20:27:57 <sipa> you do 257 HMAC-SHA512 iterations
589 2012-06-02 20:28:14 <sipa> and after 257 iterations, you check whether interpreting the first 4 bytes of the hash results in a value lower than 0x1010000
590 2012-06-02 20:28:32 <sipa> if it is, pick a new seed S and repeat
591 2012-06-02 20:28:54 <sipa> if it is not, continue iterating, and after 363 iterations, check that the hash is below 0xB60183
592 2012-06-02 20:29:00 <sipa> if it is, pick a new seed S and repeat
593 2012-06-02 20:29:18 <sipa> if it is not, continue iterating, and after 513 iterations, check that the hash is below 0x80C1A2
594 2012-06-02 20:29:25 <sipa> ...
595 2012-06-02 20:29:58 <sipa> eventually, when after 1028 iterations the hash is below 0x4080DF, you found a strength=4 seed
596 2012-06-02 20:30:37 <sipa> and if it is not, pick a new seed S and repeat
597 2012-06-02 20:32:45 <sipa> Eliel: the idea is having a short seed that generates an entire deterministic wallet
598 2012-06-02 20:33:23 <sipa> but the seed encodes its own "strength", which corresponds to a number of hashing iterations must be done on it
599 2012-06-02 20:34:57 <sipa> Eliel: https://gist.github.com/2731997
600 2012-06-02 20:35:36 <sipa> it's not a full explanation though, just the math for an idea that was known to the people who would read it :)
601 2012-06-02 20:35:44 <jrmithdobbs> sipa: i don't understand how that's useful if your prng is already giving you good seed data
602 2012-06-02 20:35:59 <sipa> jrmithdobbs: because people will use it with bad PRNG's
603 2012-06-02 20:36:02 <sipa> jrmithdobbs: aka their mind
604 2012-06-02 20:36:27 <jrmithdobbs> you're saying derive the seed from the passphrase?
605 2012-06-02 20:36:43 <Eliel> sipa: so, this is an attempt to measure the quality of the seed?
606 2012-06-02 20:36:44 <sipa> that's not the intented use case, but an assumed side use case :)
607 2012-06-02 20:37:06 <jrmithdobbs> why not randomly generate the seed and encrypt it, then you can use pbkdf2/scrypt/whateveryouwant for the key deriv
608 2012-06-02 20:37:07 <etotheipi_> jrmithdobbs: yeah.. people will pick stupid passwords... but if the client requires these special-format seeds, they have to add a X-bit nonce to get their stupid password
609 2012-06-02 20:37:30 <etotheipi_> *in order to make their stupid password work
610 2012-06-02 20:40:25 <sipa> Eliel: it's probably better to start with the original idea: start from a seed, and check whether after 256 iterations the first 8 bits are 0; if so, the 255th iteration is you key; otherwise, check whether after 512 iterations there are 9 zero bits, ... and so on