1 2012-09-26 00:20:11 <jgarzik> Tiggr: fix your software
2 2012-09-26 00:22:07 <MrTiggr> aye
3 2012-09-26 00:22:08 <MrTiggr> ikr
4 2012-09-26 00:22:10 <MrTiggr> in it
5 2012-09-26 00:22:12 <MrTiggr> *on
6 2012-09-26 03:20:17 <Varan> Is it possible for a government to spam the bitcoin network with transactions? Is this a doable DOS attack? If not ... why?
7 2012-09-26 03:21:04 <doublec> Varan: it would cost money via transaction fees
8 2012-09-26 03:21:48 <Varan> But if you have one output you send from address to address ... you dont have to pay a fee
9 2012-09-26 03:21:51 <Varan> right?
10 2012-09-26 03:22:47 <midnightmagic> Varan: assuming some miner is willing to include that one lonely tx in the next block without a fee for some reason..
11 2012-09-26 03:24:08 <Varan> yeah but there are some that do that right now
12 2012-09-26 04:32:57 <bonks> So 12 hours later I have just a few hundred more blocks to go...
13 2012-09-26 04:32:58 <bonks> I copied my backed up blk*.dat files to %appdata%/Bitcoin directory
14 2012-09-26 04:33:31 <bonks> And then loaded the qt client v0.7 and it appended started from block 0 and appended to my backups, what did I miss?
15 2012-09-26 04:34:25 <Luke-Jr> bonks: blk*.dat is only ever appended
16 2012-09-26 04:35:14 <bonks> Ok so how do people normally download those files elsewhere and continue from its last block?
17 2012-09-26 04:35:34 <bonks> I've always done it from scratch as recommended but this time it was my own taht I backed up while reinstalling
18 2012-09-26 04:36:04 <Luke-Jr> -loadblock=foo.dat
19 2012-09-26 04:36:20 <Luke-Jr> where foo.dat isn't blk*.dat
20 2012-09-26 04:38:30 <bonks> I see. Now I know, thank you :P
21 2012-09-26 04:39:28 <bonks> "Import blocks from external blk000?.dat file"
22 2012-09-26 04:39:40 <bonks> So since there are 2 files, do I use the loadblock option twice?
23 2012-09-26 04:39:49 <bonks> for each file
24 2012-09-26 04:39:50 <Luke-Jr> I think that would work.
25 2012-09-26 04:39:56 <Luke-Jr> if not, do one, then do the other
26 2012-09-26 04:40:30 <bonks> k
27 2012-09-26 04:43:33 <bonks> Something to note, the TML bitstream is available for the X6500 (beta) and I tested it earlier today and got 475MH/s on the one board I was testing
28 2012-09-26 04:43:43 <bonks> Wrong channel
29 2012-09-26 04:58:29 <Luke-Jr> lol
30 2012-09-26 04:58:36 <Varan> Have there been any developments with storing unconfirmed tx outputs in the block chain? I could only find: https://bitcointalk.org/index.php?topic=101734.0
31 2012-09-26 04:58:47 <Varan> has there been any more progress? A BIP?
32 2012-09-26 05:08:47 <weex> Varan: if it's in the block chain it's confirmed, perhaps you mean some of the m in m-of-n transasctions?
33 2012-09-26 05:09:51 <Varan> Ow sorry ... I mean unspend
34 2012-09-26 05:10:07 <Varan> like discussed in the forum topic
35 2012-09-26 05:10:27 <weex> taking a look...
36 2012-09-26 05:13:28 <weex> ahh, perhaps you mean to know if amiller has implemented much of that
37 2012-09-26 05:15:34 <weex> Varan: is this just out of curiosity or is there an application that you're trying to serve?
38 2012-09-26 05:16:22 <Varan> No ... this is juts out of general curiosity
39 2012-09-26 05:17:29 <Varan> and amiller has a github page were it is implemented ... I would like to know if there has been any further work done ... to get this in the main client
40 2012-09-26 05:17:41 <Varan> Or if it has been rejected or something
41 2012-09-26 05:17:53 <weex> from what i can gather that proposal is about being able to reduce the need for storage of the blockchain to send valid transactions
42 2012-09-26 05:18:12 <Varan> yes
43 2012-09-26 05:18:17 <weex> so i think the most likely line of progress is ultraprune
44 2012-09-26 05:18:19 <Varan> to verify transactions
45 2012-09-26 05:18:38 <Varan> do you have a link for this?
46 2012-09-26 05:18:45 <weex> just from what i've been watching, lemme see
47 2012-09-26 05:19:30 <weex> https://github.com/bitcoin/bitcoin/pull/1677
48 2012-09-26 05:19:53 <Varan> ah oke
49 2012-09-26 05:21:24 <Varan> I would have expected that there would be an BIP for this ... because I guess he has to add a extra field to the block header right?
50 2012-09-26 05:21:58 <weex> i don't think so
51 2012-09-26 05:22:38 <weex> the merkle tree means spent txouts can be dropped
52 2012-09-26 05:22:42 <Varan> Hmm oke
53 2012-09-26 05:22:52 <Varan> what do you mean dropped?
54 2012-09-26 05:24:15 <weex> well the full details of those transactions need not be stored because their inclusion in the block chain means they had to be valid when they were included
55 2012-09-26 05:27:42 <_dr> the original paper explains it pretty well imho
56 2012-09-26 05:28:23 <weex> ^--- better than me trying to re-explain what i think is going on
57 2012-09-26 05:29:40 <weex> amiller went a bit further than anything in the paper so i too would like to see what come out of that
58 2012-09-26 05:31:51 <_dr> spent txns can be dropped because they won't be refereced by future tnxs, since all coins will come from unspent txns
59 2012-09-26 05:32:05 <_dr> the merkle root/branch part is only for spv i think
60 2012-09-26 05:33:18 <_dr> also, only spv clients drop spent txns, the reference client keeps them for various reasons (such as bootstrapping other nodes)
61 2012-09-26 05:34:14 <_dr> (I think :)
62 2012-09-26 05:36:41 <Varan> spv?
63 2012-09-26 05:37:17 <_dr> simple payment verification. and i think that none of that has anything to do with ultraprune (except the 'prune')
64 2012-09-26 05:37:38 <Varan> Hmm oke
65 2012-09-26 05:38:36 <Varan> But if you bootstrap the client you still have to download and verify all the transactions right?
66 2012-09-26 05:38:54 <Varan> because you dont know the unspend tx outputs yet
67 2012-09-26 05:39:03 <Varan> (using ultraprune)
68 2012-09-26 05:41:39 <_dr> https://bitcointalk.org/index.php?topic=91954
69 2012-09-26 05:42:02 <_dr> Varan: this one is a little bit more verbose than the text on github
70 2012-09-26 05:42:16 <Varan> yeah I read that
71 2012-09-26 05:43:30 <_dr> it says it keeps the entire blocks, so it will still download the blockchain to bootstrap
72 2012-09-26 05:44:40 <weex> Varan: if you start downloading history from the beginning, you might want all transactions to get a clear snapshot of the blockchain up to the point you've downloaded
73 2012-09-26 05:45:00 <weex> newer ideas are to start with the latest n blocks and backfill
74 2012-09-26 05:46:00 <weex> but hopefully be able to validate unconfirmed transactions asap
75 2012-09-26 05:46:34 <weex> i'm not sure if any code does that at this point though
76 2012-09-26 05:51:35 <_dr> what i don't understand about the ultraprune post is: 'It simply means you cannot rescan/reorg/server those old blocks, but once those are deep enough (say a few thousand blocks), we can tolerate that.'
77 2012-09-26 05:52:20 <_dr> it's no longer a full node, right? or is there a way to keep the integrity of the network while discarding the older parts of the blockchain forever
78 2012-09-26 05:52:51 <Varan> Not using this patch I think
79 2012-09-26 05:53:08 <Varan> If all nodes delete the old blocks noone can bootstrap anymore
80 2012-09-26 05:53:50 <_dr> that's what i'm not 100% sure about
81 2012-09-26 05:54:28 <Varan> seems logical ... but then I dont understand how they can put this in the official bitcoin client
82 2012-09-26 05:54:36 <Varan> who is going to serve the old blocks then?
83 2012-09-26 05:55:21 <weex> it would be a switch probably so default would be to keep all blocks
84 2012-09-26 05:55:31 <Varan> hmm yeah oke
85 2012-09-26 05:55:49 <weex> i for one would like to run a full node or nodes until it's really not making sense
86 2012-09-26 05:56:09 <Varan> But if you would have a merkle tree of all unspend tx outputs in each block noone would have to store the old blocks right?
87 2012-09-26 05:56:41 <weex> sounds wasteful of bandwidth
88 2012-09-26 05:56:55 <weex> if mostly the same info is downloaded in each block
89 2012-09-26 05:57:05 <_dr> yeah, as i understood it the storage for all unspent tx is 70mb atm
90 2012-09-26 05:57:08 <_dr> and it'll grow!
91 2012-09-26 05:57:18 <Varan> weex, Hmm I think they had some clever sugestion for that
92 2012-09-26 05:58:05 <Varan> https://bitcointalk.org/index.php?topic=101734.0
93 2012-09-26 05:59:28 <weex> right so that needs another hash to be put in the block header
94 2012-09-26 05:59:46 <weex> it would be what's called a hard forking change
95 2012-09-26 06:00:21 <weex> https://en.bitcoin.it/wiki/Fork_Wishlist
96 2012-09-26 06:00:29 <Varan> yeah I know
97 2012-09-26 06:02:14 <amiller> i'm looking into practical ways of storing/indexing the merkle tree utxo now
98 2012-09-26 06:02:31 <amiller> i'm starting just by trying to make normal block handling more efficient
99 2012-09-26 06:03:23 <amiller> for example pynode goes much faster now that there's a leveldb (a balanced tree) to just to mostly hold utxo indexes, and a disk that's fast to append to
100 2012-09-26 06:03:31 <amiller> by disk i mean flat file
101 2012-09-26 06:04:36 <weex> i haven't run pynode for a bit but will have to check it out again
102 2012-09-26 06:07:39 <amiller> the disk and the leveldb are two fairly different datastructures, but you can basically plug'n'play different instances
103 2012-09-26 06:09:14 <weex> pynode still needs a node to connect to right?
104 2012-09-26 06:09:42 <amiller> it does but i think jgarzik is working rapidly towards having it ready to work for itself
105 2012-09-26 06:13:21 <sipa> weex, Varan: what do you need to know about ultraprune?
106 2012-09-26 06:14:42 <weex> i'm good but if i think of anything, i'll be back
107 2012-09-26 06:14:52 <weex> Varan brought it up :)
108 2012-09-26 06:16:16 <sipa> once actual pruning is implemented, someone who runs it will not be a full node anymore
109 2012-09-26 06:16:33 <sipa> but still a fully validating node
110 2012-09-26 06:17:01 <sipa> those who do store everything will become archive nodes
111 2012-09-26 06:17:02 <_dr> is there a way to run the network without full nodes?
112 2012-09-26 06:17:09 <amiller> sipa, appending to coins.dat is very efficient
113 2012-09-26 06:17:16 <amiller> even if you're not a full node, i wonder if it's useful to log some of it
114 2012-09-26 06:17:59 <amiller> could you do something like save a snapshot of the utxo, and then blockchain data on top of it, so you could recover from those
115 2012-09-26 06:18:52 <sipa> leveldb does support snapshots
116 2012-09-26 06:19:04 <amiller> i assume it takes a while to write one or it requires duplicating memory or something
117 2012-09-26 06:19:04 <sipa> should look into that
118 2012-09-26 06:19:11 <amiller> how long does it take to backup a utxo
119 2012-09-26 06:19:37 <sipa> backup where?
120 2012-09-26 06:20:22 <amiller> either to a different disk or to send it across a local network i suppose
121 2012-09-26 06:20:41 <weex> would a pruned ultraprune node still need to download all the data to get fully sync'd?
122 2012-09-26 06:21:09 <sipa> weex: to be zero-trust, yes
123 2012-09-26 06:21:54 <Varan> sipa, If it is implemented in the bitcoin client will there be like a switch between arcive mode and pruned mode?
124 2012-09-26 06:21:57 <sipa> if it trustsanother node, it can bo
125 2012-09-26 06:22:08 <sipa> Varan: yes
126 2012-09-26 06:22:16 <Varan> And the default will be archive?
127 2012-09-26 06:22:22 <sipa> yes
128 2012-09-26 06:22:25 <Varan> oke nice
129 2012-09-26 06:22:55 <sipa> but imho, far from everyone needs to be an archive
130 2012-09-26 06:23:13 <_dr> sipa: couldn't it just download the block header data and utxo and verify those like a spv client?
131 2012-09-26 06:23:52 <Varan> _dr, utxo cannot be trusted currectly unless you compute it yourself
132 2012-09-26 06:24:15 <sipa> _dr: if the merkle root of the utxo set is committed to by the coinbase, yes
133 2012-09-26 06:24:38 <sipa> but that still requires spv trust and not zero trust
134 2012-09-26 06:25:00 <amiller> backups become very efficient though
135 2012-09-26 06:25:19 <Varan> So if it is in the blockheader it would be zero trust right?
136 2012-09-26 06:25:39 <sipa> no, then it is spv trust
137 2012-09-26 06:26:13 <sipa> (spv trust = believing that the chain with the most work is valid)
138 2012-09-26 06:26:38 <Varan> ah oke
139 2012-09-26 06:26:49 <weex> then what's zero trust?
140 2012-09-26 06:26:54 <sipa> the only way to bootstrap zero trust is by validating the entire history
141 2012-09-26 06:27:03 <amiller> spv trust should include a parameter, which is the last block you know to be completely valid, like spv(work)
142 2012-09-26 06:27:17 <amiller> spv(0) means you're bootstrapping from the very beginning and have to validate the entire history in order to reach zero trust
143 2012-09-26 06:27:46 <sipa> weex: not making any assumption about the data you receive from others
144 2012-09-26 06:28:10 <sipa> bitcoind right now is *almost* zero trust
145 2012-09-26 06:28:30 <amiller> svp(-100) would mean you're 100 blocks behind, you don't need to revalidate blocks past that point, and if you can restore a utxo set at that point, you can just process the remaining 100 blocks
146 2012-09-26 06:28:33 <_dr> almost because you can never know if the blockchain you download is the real thing?
147 2012-09-26 06:29:23 <Varan> No because you need to know the genesis block and some bootstrap ip addresses
148 2012-09-26 06:29:30 <sipa> no, because of checkpoints hardcoded in the client
149 2012-09-26 06:29:41 <weex> so amiller your proposal was to include a hash of the utxo in each block?
150 2012-09-26 06:30:14 <amiller> weex, it's an old idea, i looked up a way of doing it using redblack trees
151 2012-09-26 06:30:24 <_dr> hm, there should really be much more documentation on stuff like that. i mean, the wiki is great for getting started, but there's a lot missing there
152 2012-09-26 06:32:17 <weex> _dr: agree, can't find much about this trust idea there
153 2012-09-26 06:32:34 <weex> but one can always come here and ask too
154 2012-09-26 06:57:02 <TD> sipa: leveldb snapshots are fast
155 2012-09-26 06:57:10 <TD> sipa: it basically just controls the sequence number used for compaction
156 2012-09-26 06:57:11 <TD> (iirc)
157 2012-09-26 07:13:17 <sipa> TD: but they're not persistent, afaics
158 2012-09-26 08:30:22 <TD> sipa: that could be the case, yes
159 2012-09-26 08:32:51 <gmaxwell> Doesn't look like anyone really responded to this:
160 2012-09-26 08:32:53 <gmaxwell> 22:20 < Varan> Is it possible for a government to spam the bitcoin network with transactions? Is this a doable DOS attack? If not ... why?
161 2012-09-26 08:33:31 <gmaxwell> **sigh** must everyone that shows up to bitcoin pick 'goverment' as their boogeyman? Bored 12 year olds are usually a far more likely to DOS attack. :P
162 2012-09-26 08:34:45 <epscy> bored 12 year olds don't usually have the same resources as a government
163 2012-09-26 08:34:47 <gmaxwell> The system is resistant to dos attacks because you have to burn a lot of coin days to get relayed for free; otherwise you have to include fees. If someone really wealthy started saturating the network by moving enormous coindays people could always turn up that requirement.
164 2012-09-26 08:35:35 <gmaxwell> 22:21 < Varan> But if you have one output you send from address to address ... you dont have to pay a fee
165 2012-09-26 08:35:56 <gmaxwell> And no, the anti-dos fee rules are specficially setup to catch rapid recycling of coins.
166 2012-09-26 08:47:06 <Varan> gmaxwell, Oke nice ...
167 2012-09-26 08:49:08 <Varan> But you dont need pay alot of transactions fees currently to double the amount of transactions right?
168 2012-09-26 08:52:04 <Varan> I mean the max number of transactions per day is about 50k currently you would only need 19 btc ... oke so still not trivial ... but for governments (or banks) this should not be a problem
169 2012-09-26 08:52:40 <Varan> At what amount of transactions per day do you think the bitcoin network would have a problem?
170 2012-09-26 08:55:20 <gmaxwell> Varan: I don't know that there ever would be a problem directly from that??? as then fee competition would take effect whenever the load became too high.. though we'd need to make client software improvements as the software doesn't currently help users set fees competitively... and other than mtgox codes (which are centeralized) there are any other mature secondary exchange methods for bitcoin.
171 2012-09-26 09:01:48 <Varan> Yeah oke ... but if a lot of load suddenly hits the network it could be out for an hour or a day ... before miners or developers respond
172 2012-09-26 09:02:12 <Varan> but I guess that is indeed not so bad ... at least it can survive that way
173 2012-09-26 09:03:29 <gmaxwell> Varan: fee competition is automatic. It wouldn't really be out, but transactions wouldn't go through for users who don't set their fees higher than the compeating load generator.
174 2012-09-26 09:05:24 <Varan> automatic how? I thought the client just ignores transactions that dont have 0.0001 fee and dont include them in blocks if the dont have 0.0005 fee ... how does it adjust this?
175 2012-09-26 09:06:26 <gmaxwell> Varan: no, other than a small amount of space reserved for free transactions miners preferentially include transactions based on fee/kb.
176 2012-09-26 09:07:02 <Varan> hmm oke
177 2012-09-26 09:07:18 <sipa> and when blocks get filled up more, they require an increasing BTC/kb as fee
178 2012-09-26 09:07:25 <Varan> yes that makes sense
179 2012-09-26 09:08:04 <gmaxwell> Indeed, I'd forgotten about the ramping threshold.
180 2012-09-26 09:08:09 <Varan> so all the txs with lower fees would go to the back of the queue if other people are paying more btc/kb
181 2012-09-26 09:12:15 <Varan> Thanks for explaining :)
182 2012-09-26 09:19:15 <sipa> jgarzik: currently my leveldb coins db is 117Mbyte, which is 40Mbyte keys, 60Mbyte values, and 17Mbyte leveldb overhead
183 2012-09-26 09:21:25 <sipa> so around 25 bytes per unspent txout
184 2012-09-26 11:28:13 <grondilu> Addresses on the testnet can begin by any letter, right?
185 2012-09-26 11:29:21 <sipa> certainly not any letter
186 2012-09-26 11:29:28 <sipa> but there are several, iirc
187 2012-09-26 11:32:48 <grondilu> Can someone give me the address for 5MdJ76GWf1RC6WppKeN8VfJ5fM4Z3LuZ82FjiKZtjJ22RrXHm1i for instance?
188 2012-09-26 11:39:26 <gavinandresen> testnet addresses begin with m or n, if I recall. 5 is a private key
189 2012-09-26 11:41:55 <grondilu> do you know who made https://bitcointools.appspot.com/? This stuff seems cool but does not seem to work with testnet addresses.
190 2012-09-26 11:42:45 <kjj_> gavinandresen: I tried to import it, it doesn't seem to be a valid WIF
191 2012-09-26 11:43:08 <sipa> grondilu: where did you get that?
192 2012-09-26 11:43:17 <sipa> i can't import it either
193 2012-09-26 11:43:55 <grondilu> sipa: you can't? I guess my code is buggy then.
194 2012-09-26 11:44:52 <sipa> wait... testnet WIF has a different version byte too
195 2012-09-26 11:44:57 <sipa> the one you gave is for realnet
196 2012-09-26 11:45:27 <sipa> first byte for testnet WIF is 239 instead of 128
197 2012-09-26 11:46:33 <grondilu> wasn't it 129?
198 2012-09-26 11:46:57 <sipa> no
199 2012-09-26 11:47:01 <grondilu> anyway I need to review my code.
200 2012-09-26 11:47:21 <lianj> :address_version => "6f",
201 2012-09-26 11:47:22 <lianj> :p2sh_version => "c4"
202 2012-09-26 11:47:25 <lianj> for testnet
203 2012-09-26 11:47:58 <sipa> grondilu: 0/128 for realnet; 111/239 for testnet
204 2012-09-26 11:48:25 <sipa> 0/5/128 vs 128/196/239
205 2012-09-26 11:50:47 <grondilu> ok ok. I'll study base58.h more seriously.
206 2012-09-26 12:01:39 <grondilu> what about this one? 933sbex9mybwDj6fgBCQALTtPjRRW7UUQvQDJj6QT2fTa9D7EoG
207 2012-09-26 12:02:24 <sipa> good one
208 2012-09-26 12:04:04 <grondilu> ok. I get eMX6dT7v7gPSqVAfc74odb4MYVuNfqBiA as the address. Doesn't begin by m or n.
209 2012-09-26 12:05:53 <grondilu> I know. I used 1 instead of 111 for the first byte.
210 2012-09-26 12:06:55 <grondilu> I now get muXsQaEp1xemQWphMkk89RXbb2qfYgz6K7
211 2012-09-26 12:23:31 <Varan> When is the official client supposed to say it's up to date?
212 2012-09-26 12:23:57 <Varan> Because it says this 6 blocks before it got the latest block for me...
213 2012-09-26 12:24:17 <Varan> doesn't seem correct
214 2012-09-26 12:24:27 <Varan> but maybe it doesn't know better
215 2012-09-26 12:25:15 <Varan> I started syncing 3 hours ago so this could be part of the problem ... maybe it doesn't try to find out what the block height is while syncing
216 2012-09-26 14:50:42 <jgarzik> Another alt-coin, with vastly different metrics and incentives: https://bitcointalk.org/index.php?topic=112676.0
217 2012-09-26 14:51:10 <jgarzik> I'm not convinced it will actually work, but it is nice to see alternatives beyond "change genesis block, call it jgarzik-coin"
218 2012-09-26 14:51:23 <jgarzik> amiller might like some of the features of the system
219 2012-09-26 14:52:12 <gmaxwell> jgarzik: it sort of lost my interest at the top with "have advantages of Bitcoin but would not have its disadvantages" which is something of a red flag. :P
220 2012-09-26 14:52:19 <kjj_> heh
221 2012-09-26 14:53:48 <jgarzik> gmaxwell: he clearly doesn't have a clue how bitcoin really works, or what problems it solves
222 2012-09-26 14:53:59 <jgarzik> doesn't detract from being an interesting experiment on its own
223 2012-09-26 14:54:37 <gmaxwell> Right, thats usually what its a redflag for; not understanding the problem space. But indeed, it is really nice to see something that isn't "copy bitcoin, change genesis block, call it jgarzik-coin".
224 2012-09-26 14:54:49 <kjj_> I'm having a really hard time following his train of thought. my first impression is Open Transactions, without the experience that FT has learned over the years
225 2012-09-26 14:56:05 <jgarzik> with a dash of ripple-for-computers
226 2012-09-26 14:56:15 <jgarzik> and the points being made about Sybil and botnets seem quite valid
227 2012-09-26 14:56:20 <kjj_> yeah, still reading. might be closer to ripple
228 2012-09-26 14:56:36 <gmaxwell> kjj_: I only read the initial post; but it simply doesn't explain it. The words 'quorum-based' fail to describe how the quorum would be determined.
229 2012-09-26 14:57:30 <gmaxwell> (and if it's handled naively then it's hopelessly insecure; if not then it ends up with bitcoin like problems)
230 2012-09-26 14:57:40 <jgarzik> but (even if the author doesn't realize it) there is a core problem I run into often -- sustaining a distributed service via member work contributions -- so I try to keep my mind open to new problems
231 2012-09-26 14:57:52 <gmaxwell> The author of that message has made a number of other clueless posts IIRC.
232 2012-09-26 14:57:57 <jgarzik> yes
233 2012-09-26 14:58:41 <gmaxwell> oh boy. the following posts are... a laugh.
234 2012-09-26 15:00:09 <gmaxwell> I'm happy to see Etlase2 manning the cluebat. (Etlase2 has been the author of some impressive hocus pocus proposals before; so seeing the table turned is enjoyable)
235 2012-09-26 15:14:13 <bonks> Luke-Jr: I tried using -loadblock for each blk file and it only loaded from the first one
236 2012-09-26 15:14:25 <bonks> How do I load from multiple files?
237 2012-09-26 15:14:47 <kjj_> use it twice, I think
238 2012-09-26 15:15:09 <bonks> That's what I did
239 2012-09-26 15:15:12 <kjj_> or, if it finished indexing the first file's stuff, just start it again pointing to the second file
240 2012-09-26 15:15:41 <bonks> But it automatically starts to get new data and appends it to the current file
241 2012-09-26 15:15:51 <kjj_> I'm sorta guessing here, I haven't read the code behind that option
242 2012-09-26 15:18:37 <gmaxwell> bonks: you should certantly not be pointing loadblocks at files bitcoin is going to write to.
243 2012-09-26 15:18:50 <gmaxwell> I normally just cat togeater the blockfiles I'm going to feed to loadblocks.
244 2012-09-26 15:19:20 <kjj_> gmaxwell: it treats the blocks read from the file just like it got them from the network, right?
245 2012-09-26 15:22:32 <bonks> gmaxwell: Oh I see, how do I do that?
246 2012-09-26 15:23:18 <bonks> Btw where can I find the latest documentation on the commands? https://en.bitcoin.it/wiki/Original_Bitcoin_client/API_Calls_list doesn't have the latest
247 2012-09-26 15:23:26 <kjj_> linux? cat file1 file2 > combined_file
248 2012-09-26 15:24:01 <bonks> oh right, then bitcoind -daemon -loadblock combined_file
249 2012-09-26 15:24:10 <bonks> Then let it create the blk0001.dat etc on its own?
250 2012-09-26 15:24:18 <kjj_> yes
251 2012-09-26 15:24:30 <bonks> Ok let me try that
252 2012-09-26 15:24:42 <kjj_> that probably is the latest proper documentation. are you volunteering to update it? :)
253 2012-09-26 15:24:59 <gmaxwell> kjj_: right.
254 2012-09-26 15:26:00 <bonks> kjj_: Not really, I'm just starting to understand this :X
255 2012-09-26 15:28:35 <jgarzik> multiple -loadblocks on same command line does not work.
256 2012-09-26 15:28:42 <jgarzik> cat'ing together works.
257 2012-09-26 15:29:11 <kjj_> you sure? the LoadExternalBlockFile call is wrapped in a foreach block
258 2012-09-26 15:29:26 <kjj_> and it uses mapMultiArgs, just like connect
259 2012-09-26 15:31:37 <jgarzik> kjj_: whoops, looks like you're right.
260 2012-09-26 15:31:52 <jgarzik> BOOST_FOREACH(string strFile, mapMultiArgs["-loadblock"])
261 2012-09-26 15:31:55 <kjj_> heh. I have the advantage of not knowing anything, so I have to go look every time
262 2012-09-26 15:34:17 <bonks> So I could've used it multiple times on files bitcoind isn't writing to?
263 2012-09-26 15:34:27 <bonks> Btw updated the wiki with loadblock command
264 2012-09-26 15:38:23 <kjj_> sweet!
265 2012-09-26 15:38:40 <jgarzik> we should guard that code, to make sure it does not import from $DataDir/blk????.dat
266 2012-09-26 15:38:47 <jgarzik> that is a common mistake
267 2012-09-26 15:41:18 <bonks> oh geez, all this does is allow me to use a specific blockchain file and require no network?
268 2012-09-26 15:41:28 <bonks> I wanted to avoid all the processing but it looks like it's still doing that
269 2012-09-26 15:42:24 <bonks> Which makes sense for like offline wallets, but I just wanted to move everything over
270 2012-09-26 15:42:55 <kjj_> oh, that's all you wanted to do?
271 2012-09-26 15:43:21 <bonks> Yep I moved servers
272 2012-09-26 15:43:25 <kjj_> shut down, start again with -detachdb=1, let it get going, then shut down again
273 2012-09-26 15:43:37 <kjj_> then copy the files over with scp or rsync.
274 2012-09-26 15:43:55 <bonks> I copied the blk files over already
275 2012-09-26 15:44:11 <kjj_> yes, but they don't work if you didn't detach on the last shutdown
276 2012-09-26 15:44:26 <kjj_> and you need blkindex.dat to go with them, copied at the same time
277 2012-09-26 15:45:12 <bonks> Ok I'll try that
278 2012-09-26 15:47:52 <bonks> How do you normally check if it's "going"? getinfo can't connect to server yet so I don't know what it's doing
279 2012-09-26 15:48:03 <kjj_> tail debug.log
280 2012-09-26 15:48:57 <kjj_> when you see it accepting transactions, CTxMemPool:accept ... blah blah ... (poolsz X) where X is increasing, it is running
281 2012-09-26 15:49:19 <gavinandresen> tail -f will keep showing you forever
282 2012-09-26 15:50:15 <kjj_> heh. I was using tail as a verb
283 2012-09-26 15:51:18 <bonks> Ok txns accepted. Is kill -9 pid a proper way to shutdown?
284 2012-09-26 15:52:29 <kjj_> use SIGTERM, or the stop RPC command
285 2012-09-26 15:53:53 <jgarzik> or pull the power plug
286 2012-09-26 15:54:22 <kjj_> heh. he's trying to get a clean detach. the 120 reset won't give him that
287 2012-09-26 15:55:15 <bonks> Ok checking md5sum on first block so I don't need to redownload
288 2012-09-26 15:56:25 <kjj_> shouldn't need to. it probably hasn't been changed since July 9th
289 2012-09-26 15:56:31 <kjj_> gotta run, back later
290 2012-09-26 15:57:16 <bonks> kjj_: Yeah just checking though, makes me feel better :P Downloading blk0002 and blkindex now. Thanks!
291 2012-09-26 15:58:33 <bonks> Once I have these 3 files in my new .bitcoind dir I can just start the daemon and this should work?
292 2012-09-26 15:59:45 <maaku> are there invalid blocks in the testnet3-in-a-box blk0001.dat?
293 2012-09-26 16:07:08 <jgarzik> maaku: don't think so, but it is possible
294 2012-09-26 16:07:32 <maaku> k thx
295 2012-09-26 16:07:47 <maaku> i think it's a problem with my code anyway, but I just wanted to be sure
296 2012-09-26 16:13:22 <bonks> Sweet! My move was successful. Thanks all. So next time I need to remember to shutdown with detachdb=1. Copy blk* files, then restart on new server
297 2012-09-26 16:13:49 <maaku> yup, my code :\\
298 2012-09-26 16:37:32 <amiller> bonks, that's neat, you transported your own blkindex and blk00* to a different machine and started it up again?
299 2012-09-26 16:38:26 <bonks> amiller: Yep
300 2012-09-26 16:39:48 <bonks> amiller: Obviously I need the blkindex otherwise bitcoind needs to reprocess the blk000? files to build the index (which is the most time consuming)
301 2012-09-26 16:40:44 <kjj_> doing that isn't documented anywhere, and I think the official line is "don't". but it works pretty well
302 2012-09-26 16:41:19 <bonks> kjj_: Well don't if it's from a public source
303 2012-09-26 16:41:25 <kjj_> I use it to feed my p2pool nodes that boot from PXE
304 2012-09-26 16:41:30 <kjj_> right
305 2012-09-26 16:41:32 <bonks> I copied my own which started from scratch
306 2012-09-26 16:41:54 <jgarzik> you're trading time cost for security cost -- because copying the index means zero validation is done on the copied blocks
307 2012-09-26 16:42:11 <jgarzik> if it's my machine A -> my machine B, that's just fine
308 2012-09-26 16:42:25 <gmaxwell> 11:40 < kjj_> doing that isn't documented anywhere, and I think the official line is "don't". but it works pretty well
309 2012-09-26 16:42:33 <gmaxwell> huh? it's documented all over the forums.
310 2012-09-26 16:42:38 <kjj_> jgarzik: I've been trying to think of a way that someone could actually gain by making a fake chain
311 2012-09-26 16:42:48 <gmaxwell> And the only reason I'd say don't now is that the whole detach thing is pretty confusing.
312 2012-09-26 16:43:00 <jgarzik> a safer option is copying the blk*.dat... then -checkblocks=0 -checklevel=6
313 2012-09-26 16:43:06 <jgarzik> probably faster too
314 2012-09-26 16:43:29 <jgarzik> that way you don't copy _again_ or reindex, just verify
315 2012-09-26 16:44:00 <kjj_> gmaxwell: only if you know what to search for.
316 2012-09-26 16:44:25 <jgarzik> kjj_: the odd isolated case is probably not much gain... but if the "copy index + data, don't verify" practice becomes widespread, it becomes easier to compromise the group
317 2012-09-26 16:44:26 <amiller> jgarzik, another interesting case is my machine A -> dropbox -> my machine B
318 2012-09-26 16:44:49 <gmaxwell> amiller: do you often out 3GB data on dropbox? 0_o?
319 2012-09-26 16:45:07 <kjj_> jgarzik: still, I don't see how to gain from it without a bunch of other unlikely circumstances
320 2012-09-26 16:45:08 <amiller> s/dropbox/s3
321 2012-09-26 16:45:41 <amiller> at least he md5'd each of his files
322 2012-09-26 16:49:04 <jgarzik> ACTION was thinking about S3 distribution
323 2012-09-26 16:49:25 <jgarzik> if there was some sort of communal pool for paying the Amazon bills, that was be a nice method of distribution
324 2012-09-26 16:49:31 <jgarzik> *would be
325 2012-09-26 16:49:52 <jgarzik> Amazon limits objects to 5GB IIRC, so we'd need to store at blk????.dat rather than One Big File
326 2012-09-26 16:49:55 <jgarzik> *as
327 2012-09-26 16:50:05 <amiller> it's weird beecause the blk*0.dat and the blkindex are slightly different for everyone
328 2012-09-26 16:50:19 <amiller> everyone builds blk00*.dat with the same blocks but in potentially different order
329 2012-09-26 16:50:55 <jgarzik> amiller: If I was building for distribution, I'd start fresh
330 2012-09-26 16:51:12 <jgarzik> ACTION has also been thinking about a torrent, updated either (a) once per release or (b) once a month or so
331 2012-09-26 16:52:03 <amiller> that's interesting, if people periodically went through their blk*.dat, and created new files with unused blocks removed
332 2012-09-26 16:52:14 <amiller> then they'd eventually all converge and we'd actually have the same blk*.dats
333 2012-09-26 16:52:51 <kjj_> if download was the problem, and we multithreaded the block fetcher, we'd be awfully close to being a torrent already
334 2012-09-26 16:53:41 <bonks> kjj_: I thought processing was the problem. network was never as slow as cpu/io
335 2012-09-26 16:54:20 <bonks> Which is why I didn't like -loadblock for moving as it reprocessed at essentially the same speed
336 2012-09-26 16:55:51 <kjj_> grabbing blocks from an untrusted source and running -loadblock on it is somewhat safe
337 2012-09-26 16:57:08 <amiller> what happens if you securely transport your own blkindex.dat but download blk0*.dat from an untrusted source, but just turn it on without -loadblock
338 2012-09-26 16:57:30 <kjj_> most likely a segfault
339 2012-09-26 16:57:54 <gavinandresen> jgarzik: S3 can server files automagically as a torrent
340 2012-09-26 16:58:10 <jgarzik> gavinandresen: quite true, I'd forgotten about that
341 2012-09-26 16:59:24 <jgarzik> gavinandresen: I've bandied about the idea in the past of patching the client to _not_ download any block below the last checkpoint
342 2012-09-26 16:59:44 <jgarzik> thereby forcing "the community" (perhaps me, on S3 and torrent) to come up with a solution for initial bootstrap
343 2012-09-26 17:00:01 <jgarzik> maybe create an easy bitcoin-firsttime.exe which executes bitcoin-qt.exe, for Windows users
344 2012-09-26 17:00:38 <jgarzik> not saying it's a great or thought out idea, but it's something to explore
345 2012-09-26 17:00:59 <amiller> jgarzik, i think it's a cool idea, suppose everyone's at the most recent checkpoint and their blkindex's are all different, but they represent the same set of txes
346 2012-09-26 17:01:06 <jgarzik> if there was a regular torrent, plus bitcoin-firsttime.exe to download it, the experience could be made pretty seamless
347 2012-09-26 17:01:35 <amiller> you can rearrange your blkindex, one non-changing transformation at a time, until it matches the 'community bootstrap' version
348 2012-09-26 17:02:03 <amiller> now you're safe to use the community checkpoint in the future, still zero trust
349 2012-09-26 17:02:08 <jgarzik> perhaps... I was mainly thinking about shifting the current burden away from public node, for serving ancient data
350 2012-09-26 17:02:14 <jgarzik> disk seeks++ right now
351 2012-09-26 17:02:27 <jgarzik> (unless you have enough RAM to cache 100% of the block chain)
352 2012-09-26 17:02:52 <bonks> I don't know the inner workings of bitcoind but could we someday have a a bunch of public nodes that host the blockchain and the local client validates remotely?
353 2012-09-26 17:03:12 <amiller> like there must be a canonical blkindex.dat associated with each checkpoint
354 2012-09-26 17:03:13 <jgarzik> bonks: that is what we do right now. public nodes all hold a copy of the entire blockchain.
355 2012-09-26 17:03:29 <gmaxwell> amiller: no.
356 2012-09-26 17:03:51 <bonks> jgarzik: Oh true. I guess I mean offloading the processing
357 2012-09-26 17:04:06 <gmaxwell> amiller: the blk files contain orphans for good and reasonsable reasons. So the index points to different locations.
358 2012-09-26 17:04:25 <amiller> a canonical blkindex.dat would be what you'd get if you consumed a blk*.dat that has exactly the blocks in order
359 2012-09-26 17:05:12 <kjj_> when downloading now, does the client do full validation on every transaction in every block?
360 2012-09-26 17:05:13 <jgarzik> shipping an index would be fine, and enable quick startup
361 2012-09-26 17:05:21 <jgarzik> just need a magic trigger for "verify index"
362 2012-09-26 17:05:30 <jgarzik> a first-time-run flag
363 2012-09-26 17:05:50 <jgarzik> actually... hmmm... I like that solution a lot
364 2012-09-26 17:06:32 <kjj_> if we believe in checkpoints, then there is no point in checking anything except the hash and the prevHash until we hit the most recent one
365 2012-09-26 17:06:41 <gmaxwell> jgarzik: I'm busily crossing my eyes at you.
366 2012-09-26 17:07:26 <gmaxwell> kjj_: That isn't what checkpoints are for.
367 2012-09-26 17:07:32 <kjj_> agreed
368 2012-09-26 17:07:35 <kjj_> still...
369 2012-09-26 17:08:03 <gmaxwell> And we do not "believe in checkpoints"; if we do we might as well just ditch this burdensome blockchain thing and have gavin sign all the transactions.
370 2012-09-26 17:08:22 <gavinandresen> my arm would get too tired
371 2012-09-26 17:08:37 <gmaxwell> I'm now imagining gavin with a stamp "[CONFIRMED]"
372 2012-09-26 17:09:16 <kjj_> hmm. gavin, are you likely to live longer than me? I'm trying to decide if this idea is good enough not to bite me in the ass decades down the road...
373 2012-09-26 17:09:26 <gmaxwell> The leveldb + ultraprune stuff makes the indexing part superfast. Next bottleneck is bad peer selection for downloading and getting stuck, I think.
374 2012-09-26 17:10:21 <jgarzik> gmaxwell: that solves nothing related to initial bootstrap. public nodes remain required to cache the entire blockchain in RAM, or face punishing disk seeks as ancient blocks are randomly requested.
375 2012-09-26 17:10:30 <jgarzik> face it, the network will never be tuned for IBD
376 2012-09-26 17:10:56 <jgarzik> torrent just flat out does a better job of "give me a big whollop of data, from other parties also interested specifically in same data"
377 2012-09-26 17:11:09 <jgarzik> that leaves the P2P network to do what it does best -- current TX's and blocks
378 2012-09-26 17:11:52 <gmaxwell> jgarzik: Have you ever run a big torrent seeder? random IO crushing them is an enormous problem for them.
379 2012-09-26 17:11:54 <kjj_> and torrent hashes the shit out of everything already, so it is secure-ish
380 2012-09-26 17:12:47 <gmaxwell> kjj_: what? no. irrelevant. geesh. Wrong security model.
381 2012-09-26 17:13:16 <kjj_> if you trust the torrent file to be correct, then the files you get from the network will be correct
382 2012-09-26 17:13:18 <jgarzik> mildly relevant, in that you are getting the hash publishing by the torrent maintainer
383 2012-09-26 17:13:23 <jgarzik> block chain is self-verifying
384 2012-09-26 17:13:28 <jgarzik> so it doesn't really matter
385 2012-09-26 17:13:34 <jgarzik> *published
386 2012-09-26 17:13:37 <gmaxwell> jgarzik: sure.
387 2012-09-26 17:14:08 <jgarzik> torrent nicely aligns with getting data from the people most interested in the data
388 2012-09-26 17:14:21 <jgarzik> once sync'd, P2P users are most interested in recent data
389 2012-09-26 17:14:44 <jgarzik> so P2P is essentially a nice real-time broadcast piece, with an ugly Big File Transfer piece hacked in
390 2012-09-26 17:15:06 <jgarzik> after a lot of work, we wind up... recreating the best logics of torrent ;p
391 2012-09-26 17:15:09 <gmaxwell> jgarzik: No. _bitcoin_ nicely aligns with getting data from the people interested in the data; Torrent nicely aligned with getting access blocked on networks "because p2p file sharing"??? similar to the problems we've had with IRC.
392 2012-09-26 17:16:15 <kjj_> if the initial download was multithreaded, we'd be like 90% of the way towards being torrent anyway
393 2012-09-26 17:16:17 <jgarzik> what major ISP blocks all torrents?
394 2012-09-26 17:16:21 <jgarzik> that sounds hyperbolic
395 2012-09-26 17:16:37 <gmaxwell> jgarzik: consider, a full block download even done most pessimally is only 200k random seeks in aggregate. It is not obvious to me that the seeking load is a burden.
396 2012-09-26 17:17:13 <jgarzik> gmaxwell: it is a highly fragmented file, randomly accessed at a high rate by >100 parallel streams at times
397 2012-09-26 17:17:23 <gmaxwell> jgarzik: it triggers secrutity warnings in some hosting facilities and on many commercial networks??? same story as IRC (no major ISP blocks that either)
398 2012-09-26 17:17:30 <jgarzik> my public nodes with < 8GB RAM see disk wait very frequently
399 2012-09-26 17:17:51 <jgarzik> it is clearly a problem, one that will only grow as public node counts decrease and block chain size increases
400 2012-09-26 17:18:11 <jgarzik> reinventing torrent inside P2P is a painful, slow way of reinventing the wheel
401 2012-09-26 17:18:21 <gmaxwell> jgarzik: the blocks are written at once. There are about 200k blocks currently. They should not be fragmented, and as the rate grows you're talking about one seek per 1MB data served.
402 2012-09-26 17:18:37 <jgarzik> gmaxwell: the blocks are written once every 10 minutes
403 2012-09-26 17:18:42 <Varan> Can you not generate a torrent file every 1000 blocks or some and seed that?
404 2012-09-26 17:18:45 <jgarzik> gmaxwell: highly fragmented, post-IBD
405 2012-09-26 17:18:54 <gmaxwell> Torrent has to deal with a lot of issues we don't.
406 2012-09-26 17:19:09 <jgarzik> it is the classic "slowly growing file" pattern that, just like mbox files, plays hell on your filesystem
407 2012-09-26 17:19:17 <gmaxwell> jgarzik: I'm already assuming a pessimal case there where each single block (which is written at once) is a seperate seek.
408 2012-09-26 17:19:34 <jgarzik> Varan: yes, that is possible
409 2012-09-26 17:19:46 <jgarzik> Varan: one could create a torrent for blk00001.dat, blk0002.dat, ...
410 2012-09-26 17:20:13 <Varan> But something tells me you dont think that a solution :P
411 2012-09-26 17:20:25 <maaku> Varan: the problem isn't download, but seeking on disk during verification
412 2012-09-26 17:20:36 <Varan> ah oke
413 2012-09-26 17:20:53 <amiller> maaku, it's not just verification, it's seeking on disk to server random client requests
414 2012-09-26 17:20:55 <gmaxwell> maaku: your comment is why this discussion should go away until after ultraprune+leveldb is merged.
415 2012-09-26 17:20:59 <jgarzik> gmaxwell: it is plainly in disk wait here, and it is self evident that seeks are unavoidable if the working set exceeds pagecache RAM
416 2012-09-26 17:21:25 <jgarzik> torrent offloads work that reduces the working set to manageable size
417 2012-09-26 17:21:26 <maaku> gmaxwell: agreed
418 2012-09-26 17:21:31 <amiller> being able to 'chunk' the blocks into various ways is useful, a block is already a chunk of individual transactions, and a blk00001.dat is a chunk of blocks
419 2012-09-26 17:21:39 <jgarzik> simple math of pagecache size
420 2012-09-26 17:21:53 <gmaxwell> jgarzik: What are you even measuring? Regular validation on a non-ultraprue/leveldb node spends tons of time in disk wait; but that has nothing to do with block download.
421 2012-09-26 17:22:14 <jgarzik> gmaxwell: public nodes, the server side
422 2012-09-26 17:22:43 <jgarzik> gmaxwell: you cannot only pay attention to the client side
423 2012-09-26 17:22:57 <Varan> Does bitcoin currently download blocks while it verifies others?
424 2012-09-26 17:22:58 <gavinandresen> yeah, we should revisit post ultraprune. The right thing is probably just a set of canonical blk000*.dat files available for download as a shortcut to asking your peers for blocks.
425 2012-09-26 17:23:03 <gmaxwell> I'm not only paying attention to the client side, but I think your figures are being distorted by it.
426 2012-09-26 17:23:33 <gmaxwell> jgarzik: and really, so relegate the blockpullers to another thread and let them wait on the disk.
427 2012-09-26 17:23:59 <jgarzik> gmaxwell: ...and that fixes precisely NOTHING in the problem described
428 2012-09-26 17:24:08 <gmaxwell> jgarzik: because you're describing a non-problem.
429 2012-09-26 17:24:32 <jgarzik> gmaxwell: no, I am describing a real world problem that is observed right now
430 2012-09-26 17:25:28 <jgarzik> Let's try again. Requests for blocks to public nodes are effectively random and parallel: my public nodes seem block #1000 simultaneously requested with block #130000 and #193000.
431 2012-09-26 17:25:35 <gmaxwell> You're insisting there is one??? not showing any evidence of it??? and invoking questionable bittorrent pixiedust to fix it. Which would basically double the amount of security risky code that goes into operating a full node (a full bittorrent implementation) and adds another p2p network with a bunch of liablities.
432 2012-09-26 17:25:46 <jgarzik> If a block is not in OS pagecache, it must seek to the block, read from disk, and serve.
433 2012-09-26 17:25:59 <jgarzik> The block will not be in OS pagecache, if block chain size > pagecache available size
434 2012-09-26 17:26:05 <jgarzik> this is the "working set"
435 2012-09-26 17:26:16 <jgarzik> the working set continues to grow as the block chain grows
436 2012-09-26 17:26:26 <jgarzik> thus, the cache requirement of public nodes continues to grow
437 2012-09-26 17:26:57 <jgarzik> moving requesting of old blocks off-network solves this
438 2012-09-26 17:27:09 <gmaxwell> jgarzik: it only grows if you're interested in serving out the chain at higher speeds that you can get from one seek per block.
439 2012-09-26 17:27:09 <kjj_> ok, I'll end this.
440 2012-09-26 17:27:19 <jgarzik> thereby making public node costs more sustainable over time
441 2012-09-26 17:27:29 <gmaxwell> And bittorrent has the same problem, it does not optimize for cache locality.
442 2012-09-26 17:27:40 <Varan> Is it not possible to have the client download a set of block for example block 0 to 500?
443 2012-09-26 17:27:46 <kjj_> torrent has LESS seek problems than bitcoin, because torrent operates on a blocks of bytes, not bitcoin blocks
444 2012-09-26 17:27:52 <jgarzik> gmaxwell: that does not matter, because _clients_ take the burden away from public nodes.
445 2012-09-26 17:27:58 <jgarzik> that is the part you keep skipping
446 2012-09-26 17:28:00 <amiller> it's possible to shard effectively by having a cluster of servers that are each responsible for a range of blocks
447 2012-09-26 17:28:00 <Varan> this would reduce the load on full clients right?
448 2012-09-26 17:28:06 <jgarzik> we need to encourage people to run public nodes
449 2012-09-26 17:28:14 <amiller> so if your public nodes were instead several nodes, then you could distribute
450 2012-09-26 17:28:35 <gmaxwell> kjj_: we work in batches of blocks in any case.. requesting 500 at a time.
451 2012-09-26 17:28:44 <amiller> that's not really any different than saying you can break it up into chunks and then torrent those
452 2012-09-26 17:28:44 <kjj_> right, but that's totally different
453 2012-09-26 17:28:51 <gmaxwell> (though we cap that to a couple mb because of sillyness with send buffer management)
454 2012-09-26 17:28:57 <amiller> it's not totally different it's just farther on one extreme
455 2012-09-26 17:29:11 <Varan> gmaxwell, but are those 500 blocks from one client?
456 2012-09-26 17:29:14 <amiller> you could have a blk000001.dat every 500 blocks
457 2012-09-26 17:29:17 <gmaxwell> Varan: yes.
458 2012-09-26 17:29:20 <Varan> ah oke
459 2012-09-26 17:29:23 <kjj_> gmaxwell: torrent divides the files up into chunks of equal size with no understanding of what the bytes mean
460 2012-09-26 17:29:39 <kjj_> bitcoin serves the blocks AS BLOCKS, not as bytes
461 2012-09-26 17:30:03 <gmaxwell> jgarzik: I'm not sure I follow you. The demand is X you can spread it out over 15,000 nodes... or 100. But the demand is X. It's a lot harder and more vulnerable to blocking and failure if it's on 100 rather than 15,000.
462 2012-09-26 17:30:04 <jgarzik> right now the IBD burden is foisted upon public nodes very heavily
463 2012-09-26 17:30:10 <gavinandresen> moving initial block download traffic off the p2p network seems like the right direction to go to me.
464 2012-09-26 17:30:39 <jgarzik> the torrent solution (or another off-network solution) shifts the burden to the set of folks actually interested in IBD data
465 2012-09-26 17:30:41 <gmaxwell> jgarzik: I several public nodes and simply don't see what you're talking about.
466 2012-09-26 17:31:22 <jgarzik> gmaxwell: I imagine your public nodes are not listed in the wiki as fallback nodes, nor always returned in a DNS seed query
467 2012-09-26 17:31:40 <gmaxwell> jgarzik: well perhaps you should stop DOSing yourself then.
468 2012-09-26 17:31:43 <gmaxwell> :P
469 2012-09-26 17:32:05 <gmaxwell> Your static DNS seed responses are apparently generating too much load on their victims.
470 2012-09-26 17:32:12 <jgarzik> It is rather nice to see problems before they hit the rest of the users.
471 2012-09-26 17:32:22 <jgarzik> which is apparent here
472 2012-09-26 17:32:48 <gmaxwell> jgarzik: I bevelieve the overwhelming majority of users could serve their network bottleneck with purely random IO.
473 2012-09-26 17:32:56 <gmaxwell> And our IO is far from purely random.
474 2012-09-26 17:33:24 <jgarzik> public node I/O is very random
475 2012-09-26 17:33:36 <jgarzik> and if it is uncached and random, that is not desirable
476 2012-09-26 17:33:42 <gmaxwell> IBD fetches are 500 blocks at a time.
477 2012-09-26 17:34:05 <jgarzik> times N clients
478 2012-09-26 17:34:17 <gmaxwell> Validation is much more random; which is why I was suggesting that perhaps your observations are polluted by that... leveldb + ultraprune helps a lot.
479 2012-09-26 17:34:21 <gavinandresen> our IBD code seems like a patched-together mess right now....
480 2012-09-26 17:34:35 <gavinandresen> ... which is why I think rethinking it is the right thing to do.
481 2012-09-26 17:34:55 <gavinandresen> And I'm not inclined to spend a ton of time on something that is done exactly once per user.
482 2012-09-26 17:34:58 <gmaxwell> jgarzik: doesn't matter. For later in the chain you're talking about serving up many megabytes per seek. Seek to bytes ratio is the efficiency metric that matters.
483 2012-09-26 17:35:10 <jgarzik> gmaxwell: _not hitting disk_ is what matters!
484 2012-09-26 17:35:24 <jgarzik> gmaxwell: how about a design that does not hit disk at alll
485 2012-09-26 17:35:46 <gmaxwell> gavinandresen: sure but what are you going to do??? leave a reference client that is unusable because it can't start itself; or merge in 40,000 loc of bittorrent network code, some of which has probably never been audited with a security critical eye?
486 2012-09-26 17:35:59 <Varan> jgarzik, for the time being it should be easy to have the whole blockchain in RAM
487 2012-09-26 17:36:01 <gmaxwell> jgarzik: Bittorrent does not get you that! At all!
488 2012-09-26 17:36:04 <D34TH> jgarzik, with pynode's branch mininode what is under tx besides hash and is_valid()
489 2012-09-26 17:36:12 <jgarzik> gmaxwell: for the public nodes, yes it does
490 2012-09-26 17:36:23 <jgarzik> gmaxwell: for the nodes currently hit the most, yes it does
491 2012-09-26 17:37:02 <gmaxwell> jgarzik: only because you're imaginging magical altruistic seeding nodes, which have enormous resources. And if we want that we could just run bitcoin on them.. and use a flag IBD_ME on them to prefer them.
492 2012-09-26 17:37:02 <jgarzik> bitcoin-firsttime.exe could download from Amazon S3. If that fails, use a hash to search for torrent peers.
493 2012-09-26 17:37:53 <jgarzik> <shrug> the solution I described does not involve any bitcoind changes at all, except perhaps a "don't IBD by default" flag
494 2012-09-26 17:38:00 <gmaxwell> Presumably someone is willing to spend the 40cts or so it would cost per user that installs from S3?
495 2012-09-26 17:38:01 <jgarzik> no 40 kLOC import
496 2012-09-26 17:38:17 <D34TH> jgarzik, for the firsttime.exe why not just incorperate it into the installer as an option "Download blockchain?"
497 2012-09-26 17:38:25 <jgarzik> gmaxwell: I think the user community absolutely is willing to pay for that, yes
498 2012-09-26 17:38:35 <jgarzik> D34TH: sure...
499 2012-09-26 17:38:44 <jgarzik> D34TH: RE mini-node, not sure what you are asking
500 2012-09-26 17:39:16 <D34TH> jgarzik, besides hash and is_valid() what else can i access under message.tx
501 2012-09-26 17:39:31 <jgarzik> D34TH: everything in CTransaction
502 2012-09-26 17:39:37 <D34TH> thanks
503 2012-09-26 17:41:36 <gmaxwell> jgarzik: as far as the import goes??? then you're still left with a handicapped node that doesn't work if there isn't a perpetual supply of altruists running S3 or bittorrent seeds, .. and the fact that the code isn't imported doesn't make it not a security concern... it just makes it attractive to attack bittorrent clients that are sharing the blockchain seed with the expectation that they have wallets.
504 2012-09-26 17:42:23 <jgarzik> <jgarzik> <shrug> the solution I described does not involve any bitcoind changes at all, except perhaps a "don't IBD by default" flag
505 2012-09-26 17:42:39 <jgarzik> obvious fallback -- The Old Way -- remains
506 2012-09-26 17:43:52 <gavinandresen> yes, we obviously don't have to get rid of The Old Way. I think people will choose The New Way if it is much faster/convenient.
507 2012-09-26 17:44:10 <gmaxwell> Okay, that indeed addresses most of that. I'm still highly skeptical that there is really anything to be concerned about now. Lets assume that under pessimal load nodes can only do 1mbit/sec of blockchain seeding. Thus the network aggregate rate (listening nodes) is 15,000 mbit/sec.
508 2012-09-26 17:44:27 <jgarzik> Yep. And I imagine people might even be more willing to run public nodes, if they knew the IBD traffic did not default to Them.
509 2012-09-26 17:44:43 <jgarzik> If the Default Way to IBD is not public nodes
510 2012-09-26 17:44:58 <gmaxwell> Assuming the download takes 1 hr, (as it does with ultraprune) that means you need about 24000mbit/hr for one user.
511 2012-09-26 17:45:41 <jgarzik> whatever it is, it becomes zero (0) mbit/hr for one user, from the viewpoint of a public node
512 2012-09-26 17:45:46 <gavinandresen> RE: altruists: if that is a problem I'm sure we could ask for companies willing to sponsor, and display "Blockchain History Download Sponsored By <foo.com>" ....
513 2012-09-26 17:46:16 <jgarzik> bitcoin-firsttime.exe could look at $URL/sponsor.html and render in a dialog
514 2012-09-26 17:46:17 <gmaxwell> So we should be able to support 2250 concurrent IBDers at full speed.
515 2012-09-26 17:46:39 <gmaxwell> and thats under the pretty pessimal assumption that seeking reduces nodes to just 1mbit/sec.
516 2012-09-26 17:48:26 <gmaxwell> I can't easily test this: I don't have a system with a spinning disk low enough ram that the whole chain isn't cached. But if it's one 10ms seek for 1MB (since we fetch upto 500 at once it should be much better than that) the real limit should be more like 800mbit/sec. for uncached spinning disk systems
517 2012-09-26 17:48:34 <jgarzik> In 12 months of growth at current rates, what will blockchain size be? That's the amount of cache RAM full nodes will need, to avoid hitting disk.
518 2012-09-26 17:48:50 <jgarzik> 4GB? 8GB? to avoid constantly disk hits, however quick
519 2012-09-26 17:48:58 <jgarzik> *constant
520 2012-09-26 17:49:11 <gmaxwell> Or around 1,800,000 IBDers. Of course it's not that high because most nodes will be network limited long before they become disk seek limited.
521 2012-09-26 17:49:21 <gavinandresen> The real issue is do we spend time re-creating BitTorrent or Amazon S3 to improve our existing IBD code (deal with slow peers, peers that disappear, etc etc etc) or not? I say not.
522 2012-09-26 17:49:29 <gmaxwell> jgarzik: I'm not assuming any ram at all. these are assuming a 10ms disk seek per 1MB transfered.
523 2012-09-26 17:49:46 <kjj_> gmaxwell: I have a shitty server you could connect to if you want
524 2012-09-26 17:49:51 <gmaxwell> gavinandresen: we must fix that code or we risk nodes getting stuck during normal operation, it's not just an IBD issue.
525 2012-09-26 17:51:23 <gmaxwell> jgarzik: so, yea. What about my figuring above is wrong? because it looks like our disk-seek-limit _can't_ be a bottleneck for the current network; even ignoring cachablity completely.
526 2012-09-26 17:51:24 <gavinandresen> Ok. So fix that for the non-IBD case, and if it fixes IBD too then we're done.
527 2012-09-26 17:52:48 <gmaxwell> jgarzik: one thing my nodes have few of, which I think you have lots of, are bitcoinj peers. I wonder if their non-processing of blocks they pull makes them a much worse pulling load?
528 2012-09-26 17:53:02 <jgarzik> gmaxwell: might be
529 2012-09-26 17:55:05 <gmaxwell> jgarzik: I suppose for migrations we need to teach ultraprune+leveldb to index in place, enh? If so.. then that would also be compatible with an installer download. (which I still think isn't needed; but it sounds like it's pure installer machinery to make that go)
530 2012-09-26 17:55:36 <jgarzik> gmaxwell: sipa has proposed reindexing as solving a couple problems; I agree
531 2012-09-26 17:56:35 <jgarzik> reindexing would save significant time
532 2012-09-26 17:56:55 <jgarzik> and avoid users doubling their blockfile size with -loadblock=$DataDir/blk0001.dat
533 2012-09-26 17:57:06 <gmaxwell> it would also address the annoyingly common issue with users that have corrupted databases.
534 2012-09-26 17:57:11 <jgarzik> yes
535 2012-09-26 18:01:43 <gavinandresen> I'm working on dealing better with corrupted databases right now, by the way
536 2012-09-26 18:12:53 <jgarzik> bleh
537 2012-09-26 18:13:00 <jgarzik> tree spews a metric ton of warnings now
538 2012-09-26 18:19:58 <jgarzik> ACTION re-branches from older, working HEAD
539 2012-09-26 19:30:16 <jgarzik> sigh, erik v. http://blog.bitinstant.com/blog/2012/9/26/bitcoin-the-first-five-questions.html
540 2012-09-26 19:30:28 <jgarzik> "1. Zero fee to transfer money"
541 2012-09-26 19:30:41 <jgarzik> "6. Any amount can be sent (perfect for microtransactions under a penny, for example)"
542 2012-09-26 19:32:30 <gmaxwell> jgarzik: why do you show me these things? You are a cruel person.
543 2012-09-26 19:33:01 <jgarzik> ACTION clicks "Merge" button for OP_DROP, just to taunt you
544 2012-09-26 19:45:31 <gmaxwell> :P
545 2012-09-26 19:56:22 <root2> why do people say things like that about bitcoin? Five minutes reading the FAQ would clear that all up
546 2012-09-26 19:56:37 <root2> why do peoples brains turn to mush when dealing with bitcoin?
547 2012-09-26 19:56:46 <root2> same thing with that pirate deal
548 2012-09-26 19:57:10 <root2> 3000% anual gain? sounds legit
549 2012-09-26 19:58:25 <jrmithdobbs> root2: why have people been falling for 409 scams for 40 years?
550 2012-09-26 19:58:30 <jrmithdobbs> people are stupid
551 2012-09-26 19:59:29 <jrmithdobbs> root2: s/409 scams/scientology/;s/40/60/; or s/409 scams/christianity/;s/40/2000/;
552 2012-09-26 19:59:31 <root2> I dont want to believe that. Lie to me jrmithdobbs!
553 2012-09-26 19:59:42 <jrmithdobbs> root2: people what non-existant easy answers to hard problems
554 2012-09-26 19:59:47 <jrmithdobbs> s/what/want/
555 2012-09-26 20:00:05 <jgarzik> Erik V is trying to sell bitcoin, and salesmen are prone to exaggeration / glossing over.
556 2012-09-26 20:00:21 <jgarzik> I wouldn't call him stupid
557 2012-09-26 20:04:21 <jrmithdobbs> jgarzik: i'd call him a con man, but meh
558 2012-09-26 20:05:13 <jgarzik> jrmithdobbs: BitInstant is a net-positive. SatoshiDICE, more of a meh but that's life.
559 2012-09-26 20:11:13 <maaku> sigh.. why is it the stupid one-line mistakes that take an afternoon to hunt down?
560 2012-09-26 20:13:42 <jrmithdobbs> because the bigger ones are too easy to spot ;p
561 2012-09-26 21:13:26 <maaku> Is there a script for generating blk0001.dat?
562 2012-09-26 21:13:47 <maaku> from an existing set of block chain files of course
563 2012-09-26 21:14:19 <gmaxwell> maaku: blk0001.dat is just a blockchain file... um.. cat? :P
564 2012-09-26 21:14:49 <maaku> I mean the reduced, final state version Luke-Jr posted
565 2012-09-26 21:15:24 <maaku> a script for pulling out blocks 0, 1, 2, ??? 188528 in order and concatenating them
566 2012-09-26 21:17:49 <gmaxwell> maaku: I believe luke used isolated bitcoin nodes for this. (it was at least what I suggested he do, since he needed the index too)
567 2012-09-26 21:18:32 <maaku> ok thanks
568 2012-09-26 21:20:23 <Gabit> Do you guys know if satoshidice is made with basic bitcoind-client?
569 2012-09-26 21:20:32 <Gabit> hi all, btw :)
570 2012-09-26 21:25:33 <sebicas> Gabit: I think they use Bitcoink
571 2012-09-26 21:25:40 <sebicas> Bitcoinj
572 2012-09-26 21:26:16 <gmaxwell> Gabit: if you're thinking of cloning it; don't bother... it's a network abusive service and none of the clones have done well.
573 2012-09-26 21:26:20 <sebicas> Gahit: before 7.0 is was no way to the oficial client to custom build transactions
574 2012-09-26 21:26:24 <Gabit> sebicas: Thanks :)
575 2012-09-26 21:26:57 <Gabit> gmaxwell: Not exactly, or even near. I'm just curious how they can see the 0-confirmations and play it out
576 2012-09-26 21:27:23 <sebicas> Gabit: you can do that with the new client 7.0
577 2012-09-26 21:27:32 <sebicas> rawtransaction method
578 2012-09-26 21:27:54 <Gabit> Oh thanks sebicas, saved me a ton of time :)
579 2012-09-26 21:29:05 <sebicas> Gabit: No problem??? you will need to use createrawtransaction
580 2012-09-26 21:29:07 <gmaxwell> sebicas: please don't supply tech support to likely DOS attackers. :P
581 2012-09-26 21:29:21 <maaku> or modify blkmond, dynode, or one of the other half-a-node libraries
582 2012-09-26 21:29:28 <sebicas> ok gmaxwell..
583 2012-09-26 21:29:31 <Gabit> gmaxwell: excuse me?
584 2012-09-26 21:29:47 <maaku> dynode=pynode
585 2012-09-26 21:29:50 <sebicas> There is a huge controversy..
586 2012-09-26 21:30:02 <gmaxwell> Gabit: Yes
587 2012-09-26 21:30:03 <gmaxwell> ?
588 2012-09-26 21:30:08 <sebicas> Since satoshi dice generates a lot of small transactions
589 2012-09-26 21:30:16 <sebicas> Wich is bad for the nextwork
590 2012-09-26 21:30:28 <sebicas> network..
591 2012-09-26 21:30:48 <gmaxwell> sebicas: _needlessly_ which is key. No other popular gambling site??? even cryptographically proven ones??? generate two transactions per play.
592 2012-09-26 21:31:29 <maaku> gmaxwell: it's essential to the psychology of gambling that underlies satoshidice's business model
593 2012-09-26 21:31:46 <maaku> gmaxwell: i'd like to see any detractor propose an alternative
594 2012-09-26 21:31:50 <sebicas> gmaxwell: I totally understand it now.. but I few weeks ago I had the same questions Gabit has
595 2012-09-26 21:32:16 <Gabit> Well, as I said I'm not interested in sending 0.01 coins over. I'm interested about the speed when a depositor can get to playing.
596 2012-09-26 21:32:25 <jgarzik> <shrug> satoshidice could wait 30 seconds, or even 15, before transferring results back (so as to gather results inside sendmany)
597 2012-09-26 21:32:30 <gmaxwell> maaku: there are hundreds of successful bitcoin gambling sites; many much more successful than dice in terms of income.
598 2012-09-26 21:32:54 <sebicas> Like?
599 2012-09-26 21:33:05 <gmaxwell> jgarzik: it could also not create tons of 1e-8 outputs which will never get spent. :-/
600 2012-09-26 21:33:53 <sebicas> I think since lots of people is cloning SD the should understand this problem to find a way to fix it.
601 2012-09-26 21:34:34 <maaku> jgarzik: no, even a few seconds delay would ruin the "can't stop now" martingale psychology that satoshidice relies upon
602 2012-09-26 21:34:53 <gmaxwell> sebicas: like pretty much every other one. Dice is net negative??? after fees and operating costs??? according to its owner. (in some post a couple weeks ago)
603 2012-09-26 21:34:57 <Gabit> the network could collaboratively decide minimum amount to send...
604 2012-09-26 21:35:17 <Gabit> they would decline small amounts... for example
605 2012-09-26 21:36:11 <gmaxwell> sebicas: it's also unclear how popular it actually is??? since you don't know how much of its traffic is pure promotional stunt.
606 2012-09-26 21:37:01 <sebicas> gmaxwell: you are right about it??? most players are robots, and can be easily be their own, since either way the money goes to him
607 2012-09-26 21:38:22 <gmaxwell> sebicas: originally most of the high value bets were losing a suspicious amount of the time... inspiring speculation that they were cooking it it make it look more rewarding to play. (the operating can pre-choose winning or losing bets)
608 2012-09-26 21:38:30 <Gabit> actually, shouldn't it be possible to make a feature where a miner could decline processing a payment if not a tx fee included?
609 2012-09-26 21:39:22 <sebicas> Gabit: Yes, miners can choose to that TXs include in blocks